CALICE DAQ/online software system instructions

• External packages installation
  • SBS PCI-VME driver (online use only)
    • The driver software needs to be installed into the Linux kernel. The driver straight out of the box is unfortunately not usable and so a version should be downloaded from the HAL web site (see below). The AHCAL and ECAL are currently (end 2005) using V2.3.A (aka v2p3pA). The driver can be installed in any directory. However, note that directly under this directory, it needs to have a directory structure "installdir"/1003/v2.3.a/ for the make files to work. This driver version also requires a modified version of the routine btpapi.h to compile in C++; the modifed version can be found here and this should be copied to the directory "installdir"/1003/v2.3.a/include/btpapi.h.
    • There are several jumpers which need to be set correctly on the VME card before use; check the SBS manual for these. In addition to the default settings, jumpers needed to be added to the ARB, SYS (positions 4, 5 and 6), and T-INT (position 6) arrays.
    • The software comes with various utilities. To use these, go to the directory "installdir"/1003/v2.3.a/src/. With the SBS cards installed and powered up, then run bt_revs. This should communicate with the VME card and report back the version numbers, etc. If everything is working, it should give something like this.
    • The default SBS settings result in a large number of informational messages being written to the kernel log file when doing VME block transfers. If this becomes an issue, then they can be turned off using the bt_info utility in the same directory as above. To do this, first check which bits are set in the relevant memory area using bt_info -p BT_INFO_TRACE which should give something like the first part of this. If bit 12 = 0x1000 = 4096 is set (as shown by the 0x7... in the example shown), then the informational messages are enabled. To turn them off, simply reset the value to the same thing with this bit set to zero, using bt_info -p BT_INFO_TRACE -v This should give something like the middle part of this. The last part is simply running the original command again to check the value is as expected.
    • If you want to use another VME controller card which is supported by HAL (see below) then this is possible but a little fiddly; contact Paul directly for information on this.
  • HAL VME interface (online use only)
    • The Hardware Access Library is a CERN interface to various VME controllers (including the SBS 620). The software is documented at the following web page. The exact version should not be critical; the AHCAL and ECAL are both currently (end 2005) using ver-03-06. Again, the software can be installed in any directory. The installation is best checked through use of the CALICE software after it is installed (see below).
  • USB_DAQ interface (onlilc use only)
    • Software to drive the USB_DAQ interface needs to be installed. The code can be downloaded as a tarball from here. This should be untarred and will create a directory called decal_sw. Go into this directory and source the decal_sw_setup.sh file to set some environmental variables. Then to build the programs, source the makes.sh file.
    • Communicating with the USB_DAQ requires a working version of udev to be installed on the PC; most recent versions of Linux will have this. Two files will need to be added to the udev rules lists, which requires root access. The two files are for the standard USB_DAQ firmware and the master USB_DAQ firmware. The files can be found here and here, respectively and these should be added to the directory /etc/udev/rules.d.
    • If all is well, then with a USB_DAQ plugged in, then the utilities decal_sw/TPAC1x/Tests/bin/Status and decal_sw/TPACMasterController/Tests/bin/Status, for the standard and master firmware respectively, should return sensible values; see here and here for example.
    • It is useful to add the required variables to LD_LIBRARY_LIB. This can be done in tcsh in a similar way to shown here.
  • CLHEP and GEANT4 (onlsim use only)
    • These can be downloaded from HERE!!!!! and they need to be installed according to their standard instructions. During installation of GEANT4, select the option to put all headers into one common directory.
    • It is useful to add the required variables to LD_LIBRARY_LIB. This can be done in tcsh in a similar way to shown here.
  • ROOT graphics (all uses)
    • ROOT is used for histogramming online and is documented at the following web page. A recent version needs to be downloaded from this site: go to the "Download Binaries" link and choose a recent version which is compatible with your Linux compiler. The exact version should not matter as none of the more recent features are used in CALICE; currently (end 2005) V4.02 and V5.02 are being used by the AHCAL and ECAL respectively. The installation can be put anywhere on the Linux PC.
    • Following installation, then define the environmental variable ROOTSYS to point to the installation top-level directory, add the ROOT libraries to the library path and add the ROOT binaries to the command path. For tcsh this can be done as shown here. To check the installation, try starting a root interactive session, which should give something like this.
• CALICE software installation
  • Source installation
    • For online/onlilc/onlsim use, in all the following, work within account caliceon with the properties: uid=12841(caliceon) gid=3145(calice) groups=3145(calice). For daquser use, any account can be used.
    • The central source code should be installed separately from the directories which will be used to actually build and run the code. Currently, the source should be downloaded as a tar'ed and gzip'ed file. (In future, this will migrate into CVS.)
    • The file should unpack to give a directory called code containing five subdirectories (packages), namely daquser, onlilc, online, onlsim and records. Within each of these, there are bin, inc and src subdirectories plus some others specific to each package.
  • Directory structures
    • There are four types of structure which are useful; three for acquiring data and one for analysing it. The former is called "online", "onlilc" or "onlsim", the latter "daquser". The instructions for all are similar. First, create a new directory in which to work and cd into the directory.
    • Soft-link (with the same names, except for onlilc and onlsim, see below) the packages from the central source installation into the new working directory as subdirectories. For a daquser structure, only the records and daquser packages are needed while the others need one of the online-like packages. For an online structure, link the online package. For onlilc and onlsim, link the relevant package into the working directoty but call the soft link "online" in either case. Hence, for all three acquiring data uses, there should be three soft links called daquser, online and records.
    • Create three directories within the new working directory called bin, inc and src.
    • If acquiring data (i.e. an online/onlilc/onlsim structure), create a directory "data" in the new working directory and within data, create further subdirectories data/log, data/run data/slw and data/sum. If onlsim, also create a data/sim directory. Also create a file data/runNumber.txt which contains the first run number you will want to use. The data directory can of course be a soft link to a different directory if more convenient. If analysing data (i.e. a daquser structure), then soft-link the data area containing the run files which are to be analysed into the new working area as a directory called "data", which should have the data/run subdirectory below it, which is the only one needed for analysis.
    • If an online structure, then link in the SBS and HAL software directories. For the SBS driver software, the link should be called "sbs" and should point to "sbsinstalldir"/1003/v2.3.a. The HAL link should be called "hal" and should point to the hal top level directory.
    • If an onlilc structure, then link in the USB_DAQ code directory The link should be called "usbdaq" and should point to the decal_sw directory.
    • If an onlsim structure, then link in the CLHEP and GEANT4 directories. The former link should be called "clhep" and should point to the CLHEP installation, while the latter should be called "geant4" and point to the GEANT4 installation.
    • When everything is completed, then the whole online working directory should look something like this and the onlilc should look something like this. The daquser working directory will be very similar but missing clhep, geant4, hal, online, sbs or usbdaq links.
  • Environmental variables and path:
    • No new environmental variables are needed for the CALICE software. However, it is convenient to add the executable directories to the PATH environmental variable. For tcsh, the way to do this is shown here.
  • Building executables
    • Within the working directory, then typing "build programname" should make the executable programname in the bin directory. This can be run by simply typing "programname". A simple example is build printHex, which makes an executable which writes out its integer argument as both a hex and binary number. An example of its output is shown here. A further example is printTime which converts an Unix epoch time integer given as an argument into a human-readable form.
    • For an online/onlilc/onlsim structure, the hardware locations used for a particular system need to be specified, as in the file daquser/src/runnerDefine.icc_COPY_THIS. Copy this to inc/runnerDefine.icc in working directory and edit it to correspond to your system.
    • For an online/onlilc/onlsim structure, typing buildAll should build the essential executables needed for the DAQ.
    • The utility alive should then be run (simply type alive). For online, this checks if any CRCs in the crate can be seen. A typical output of this is shown here. If you have only one SBS PCI card, then the other (PCI card 1) will correctly give an error stating it is not installed; this is expected. For onlilc, then alive should give something like this.
    • When editing code, copy it to the local working directory src or inc subdirectories first. These versions will be taken in preference to the versions in the central code directories and this allows code updates without losing your changes.
• Online and onlilc operations
  • Background tasks
    • The main program for the online DAQ executes in the background. To start this from the online working directory, type startUp.
    • The actual program called runner should appear in the process list (use ps to see it).
    • A pair of output (i.e. from cout) and error (i.e. from cerr) files will also be created in data/log/LogXXXXXXXXXX.out and .err respectively, where XXXXXXXXXX is the Unix epoch time of the startUp command. (Use printTime to see what date the XXXXXXXXXX value corresponds to.) Soft-links called ActiveLog.out and .err are also created to point to the current log file, so it is more convenient to use these rather than the explicit data/log file name. Note, some of the SBS information messages are sent to cerr so they appear in the .err file; this is normal.
    • Finally, a file data/slw/SlwYYYYYYYYYY.bin will be created; this contains the slow controls and readout data and the part of the program collecting this is always running. The value of YYYYYYYYYY is again the Unix epoch time of the start of the slow monitoring run which may differ by a second or so from the log file time.
  • Run control
    • To start a data-taking run, then the runStart program is used. This has several arguments; type runStart -h to see them all. You need to choose a run type; the list of available run types is given in a the file RunControlTypes.txt created at startUp. The current list for onlilc is here. A brief explanation of what they do and the meaning of the version numbers is given here or by running the program runTypeComment, although this has few fully commented run types. For example, a basic online run is crcNoise, which simply reads the boards as if they have no cables connected. To run this, type runStart -w -t crcNoise -v 5 for version 5.
    • If the -w option for runStart was not selected, then the run data records are written to disk as data/run/RunNNNNNN.MMM.bin. Here, NNNNNN is the run number (as originally specified in the data/runNumber.txt file, which is incremented for each run written to disk). MMM is the file number within the run starting from 000. Files are kept below 2GBytes to prevent large filesize issues in Linux so a single run, which can have more than 2GBytes of data, is split between several files which number incrementally.
    • The progress of a run can be checked using the runMonitor program. This can be run at any time but it is convenient to run it continuously in a small window in the corner of the terminal being used. It prints out numbers of records seen in the whole job, the current run and the current configuration period within the run. To stop the program, Ctrl-C in the window it is running in.
    • Some runs have a finite series of configuration periods and events within those periods and so will complete by themselves. Others will run (effectively) for ever. To stop any run currently in progress, type runEnd. The runner program will then default back to a slow monitoring run. To stop the current configuration period within the run, type configurationEnd. The next configuration period will then start, or if all have then been completed, the run will end.
    • To stop the whole runner background task, then type shutDown. This should always be done before changing any of the hardware, powering down the VME crate, etc. If the run crashes completely, then type panic to clean up lock files, etc, before doning the next startUp. Usually startUp will not work unless this is done following a crash.
  • Run monitoring
    • Histograms of the current run can be seen using ROOT. They are filled by the runner program and stored in shared memory. For example, to see the basic mean and rms histograms per CRC, type "hstDisplay HstCrcNoise" and one canvas per CRC will appear. They continually refresh and take up non-negligible CPU so it is best not to leave them running continuously. To close them, go to the "File" menu on any of the canvases and select "Quit ROOT". For a list of available displays, type "hstDisplay -h" and for any given display for further information on options for that display, e.g. type "hstDisplay -h HstCrcNoise"
• Simulation (onlsim) use
  • Data files
    • The simulation itself first produces a file in the data/sim directory called SimXXXXXX.YYY.bin where XXXXXX is the run number and YYY is 000, 001, 002, etc, for the various files in a run. The digitisation then processes these to files in data/run called RunXXXXXX.YYY.bin. Note the number of files in a run could increase as digitisation adds more data to the records. There will be correspondingly fewer records in each file so there will not be a one-to-one correspondence of records between the Sim and Run files numbered with YYY.
• Offline (daquser) use
  • Data files
    • To run over data, then the raw data "bin" files will need to be copied to the machine being used. These are all accessible from the DESY dCache. Contact Paul or Roman for further information.
    • For runs in 2004/5, each run corresponded to one bin file, with a name RunXXXXXX.bin, where XXXXXX was the run number. Since 2006, a run can continue for more than one 2GByte file and the bin files are now named RunXXXXXX.YYY.bin, where YYY is 000, 001, 002, etc, for the various files in a run.
  • Example and utility programs
    • The "count" program does a simple count of the types of records in a run. The code is in daquser/src/count.cc and is made using "build count". To run it, type "count XXXXXX" for run number XXXXXX. Typical output for a run consisting of two files is shown here.
    • The "example" program shows a few basic steps for accessing data from the bin files. The code is in daquser/src/example.cc This will print out the DaqRunStart subrecord at the beginning of the run (which says what sort of run it is), the DaqRunEnd subrecord (which says how long the run was) and the same count table as above for all the different types of records in the run. Typical output for a short crcNoise run is shown here.
    • An example of code which accesses the ADC data is daquser/inc/hst/HstCrcNoiseShm.hh which is mainly for producing mean and rms plots for the online monitoring (see above under Run Monitoring). This can also be run offline using the main program "hstGeneric". This puts the data into shared memory, just as for the online, and the histograms can be displayed using the same hstDisplay utility as for the online case. Do not run hstGeneric while taking data or the online histogram shared memory will be corrupted. The hstDisplay program needs to be run simultaneously to the hstGeneric program. The code in HstCrcNoise.hh can be studied to see how the ADC data subrecords are obtained from the record.