DAQ Coordination Meeting

America/Chicago
    • 09:00 09:10
      News 10m
      Speakers: Alessandro Thea (STFC - RAL), Asher Kaboth (Royal Holloway University of London/Rutherford Appleton Laboratory), Giovanna Lehmann (CERN)
    • 09:10 09:20
      Specifications document status 10m
      Speaker: Giovanna Lehmann (CERN)
    • 09:20 10:25
      Round Table
      • 09:20
        FD Upstream DAQ 5m
        Speakers: Jim Brooke (University of Bristol), Mr Roland Sipos (CERN)

        Progress :

        • Added random trigger generation capability to HSI FW (software support still in progess).
        • v2.8.2 changes backported to develop, and readout consolidation release merged to develop
        • Readout breakout ready, branches for affected packages provided
        • TP-FLX FW integration complete in dummy setup (ie. fake TPs) and ready for tests.
        • Performance measurements (SNB readout + heavy cpu load, TP post-processing for different formats).
        • VD-CB hardware setup is done (config maybe not yet final).
        • Progress on SSP readout at NP02; kind of working.  (Last minute work without much help from PDS. This has put back other work).
        • Further tests of DTS-WR integration in WR lab successful (but not yet complete).  
        • Discussion of Ethernet readout work including UK & CERN people started.

         

        Plans for next 2 weeks

        • Add HSI random trigger support to software (should have a tag within 1 day ?).
        • Once TP-FLX integration tests in dummy setup are done, test with real TP FW block (single pipeline).
        • Finish MIB firmware
        • Finalise VD-CB config ?
        • Debug VST setup (no light on WIB fibres, reasons completely unclear).
        • Finish DTS-WR tests in WR lab (stability of 1PPS signals; tests of recovered clocks to happen at Bristol lab in future).
        • Reports in UD meetings on : TP output format performance, DTS-WR tests, TP integration

         

        • Maybe not next 2 weeks, but following 2 weeks - we will have FW for the uTCA Timing system (MIB+FIB).  After standalone tests, will want to test integration in VST ?

         

        Cross-WG issues

        • Support for hardware configuration (that persists between runs) from appfwk/CCM ?

         

         

      • 09:25
        ND Upstream DAQ 5m
        Speakers: Patrick Dunne (Imperial College London), Patrick Dunne

        Focus still on unit testing. ZMQ link unit tests now ready to merge into develop PRs will open soon. 

        Next steps:

        - Will PV will set up grafana and produce some pretty plots from it

        - Krzysztof to look into integration tests following FD US and DS examples (i.e. check data into LBRU gives expected data out at other end of full chain)

        - Krzysztof to build raw TCP (STREAM) python generator and receiver on par (feature-wise) with what is offered by current python script for PACMAN. This is necessary for detectors that do not use ZMQ PUB-SUB (pushing ND-LAr onto STREAM as well longer term)

        - Later on adding/testing support for many ZMQ links to test full detector readout

      • 09:30
        FD Dataselection/Physics Performance 5m
        Speakers: Joshua Klein, Joshua Klein (University of Pennsylvania), Philip Rodrigues (University of Oxford)
        • Horizontal trigger for cold boxes moving ahead (discussion tomorrow)
        • Discussing of interfaces and division of online "analyses" between Data Filter, DQM
        • Also discussing with FNAL computing about getting effort on Data Filter
        • New efforts on SN burst trigger, induction-wire TPs starting
      • 09:35
        ND Data Selection/Physics Performance 5m
        Speaker: Dr Jonathan Hays (Queen Mary University of London)

        Progress since previous meeting

        - All PRs reviewed by Kurt B and now merged into develop. So ND dataselection by timing trigger is now possible from develop.

        - Alex chatted with Phil about the easiest way to develop horizontal muon trigger. Set up all template classes, added plugin etc.

        Intended work until next meeting

        - Continue work on horizontal muon trigger - adapt the SumADC algorithm to keep track of the number of collection plane channels in a time window (instead of or as well as the SumADC)

        - ND DS planning discussions

        Points of discussion

        - No substantive points to raise.

      • 09:40
        Dataflow 5m
        Speaker: Dr Kurt Biery (Fermilab)

        Recently completed:

        • we've successfully demonstrated the initial basic NetworkManager demo (switched to using it for DataRequests and Fragment replies between Data Collection and Readout).
        • HDF5 write changes for release v2.8.1 (Coldbox, etc.)

        Upcoming work:

        • Additional NetworkManager demonstrations, for example pub/sub TimeSync messages between Readout and FakeHSI apps (Kurt) and TriggerDecision/TriggerComplete messages (Marco and Carlos).
        • Adam and Wes: HDF5Reader for DAQ users

        Cross-working group topics:

        • Discussion with Georgia, Josh, Mike Kirby about Data Filter (hopefully, Josh has mentioned this already).

         

         

      • 09:45
        CCM 5m
        Speakers: Alexander Tapper, Alexander Tapper (Imperial College London)

          Progress

        • Development of nanorc to support sub-systems and parallel commands etc.
        • Run number generation and run registry database
        • Dashboard development and management

        Intended work

        • Short term:
          • Complete above (maybe today, will know more after daily CCM meeting at 3pm)
        • Longer term:
          • Many Kubernetes related studies (Enrico, Gordon ...)

        Cross group

        • PR from Brett on minidaq config -- dangling
      • 09:50
        Data Quality Monitoring 5m
        Speaker: Pip Hamilton (Imperial College)

        Progress since previous meeting:

        • Web platform back online + under testing.
        • Fourier transform output fully validated.
        • DQM specs chapter finished.

        Work until next meeting:

        Top priority:

        • Debugging persistent strangeness in raw event display.
        • Implementation of VD channel map.
        • Completing integration of trigger record monitoring from files.
        • Configuring web display with all information necessary for reading.

        Other:

        • Switch to BSON transmission.
        • Optical data integration.
      • 09:55
        Facility 5m
        Speakers: Prof. Alec Habig (Univ. of Minnesota Duluth), Bonnie King (FNAL)

        Since last meeting (much from Friday's Integration meeting):

        • NP04 coldbox preparations (Alessandro)
          • Single mode fiber is run to NP02, with a 1G switch there to run the coldbox stuff. The 10G switch was an HP model that only takes HP cartridges that are hard to find and expensive, so we don't have 10G there like was planned.
          • a 25-pair fiber trunk will be laid soon. Then we can add an optical NIC card to the FELIX server and get all the bandwidth we need back to the rest of the DAQ
          • Plan is also for a 100GB switch to ride the end of that trunk. Lead time on this is 7-8 months(!) but that should still be before we have a whole VD detector to read out.
        • NP04 server upgrades and issues (Alec)
          • Need 1G temporary connection for np04-srv-004 to PXE boot from for upgrade.  People will be around next week who can plug that in (we will need to register the NIC too: first 1G max address port is a4:bf:01:38:c0:15, just to get it written down where I can find it).
        • DNS problems (Alec)
          • Using /etc/hosts will keep ups from exciting the CERN DNS server.  To make that easier to manage, Pengfei is cooking up an ansible script that pulls from DNS, writes, and distributes /etc/hosts for our machines so internal traffic never has to hit the DNS.
        • RAID issues (004 reinstall) (Pengfei)
          • Pengfei reconfigured the raid on this machine from raid5 to faster (but less storage) raid10.  Performance greatly improved.
          • Addressing the raid via UUID rather than /dev/md0 hopefully will make it come back automatically after reboot, something that has been a problem otherwise.
        • mainline kernel testing with FELIX: packaging issues (Bonnie)
          • We need kernel 5.x for FELIX drivers.  CS8 has only 3.x.  There is a "mainline" kernel 5.x available from the EPEL repo: but the devel headers rpm is missing bits FELIX needs.  Bonnie will get an rpm we can deploy manually, and look into fixing the EPEL version, so we can eventually just track that.
        • podman/pocket investigations, TRACE from containers (Ron)
          • Trace can write to a file in the shared data area outside the container with the right config (both container and server side).
          • file is owned by the owner of the container.
          • Multiple containers all tracing could be an issue.  Do they all use different files, or put process names/IDs in the trace messages to ID which line is theirs?  Either is an option.
        • CS8 (Alessandro)
          • Pengfei has a native CS8 build of the DAQ almost ready for primetime, which will then need to be tested both on the CS8 systems, and in CS8 containers.
        • HWDB (Alec)
          • There is a push for each consortium to generate unique Part IDs to serve as HWDB keys.  A DAQ scheme to do so is being put together for discussion at Wednesday's Installation meeting.  Initial draft is in the materials for this indico.
          • Timing system scheme will be drawn from the list of parts in EDMS (thanks David!)
          • Will expand DAQ computing from this materials list (thanks Alessandro!)
          • Discussion of cables: we don't want most cables in the HWDB. Long fibers, yes: because they have QA and loss information so are each distinct.  But no one cares which 1m cat6 cable is where.  Note that most long fibers are owned by  Installation or Facilities, not DAQ.
        • Run number app running on np04-srv-021 (Bonnie)
          • out of NFS /nfs/home/np04daq/np04rn-rest
        • spec document: clarified what is needed

        Intended work until next meeting:

        • Grafana in pocket: move configs to k8s and look at what dashboards/configs make sense for various environments
        • Update specification document

        Cross-WG discussions

        • run number db app in pocket: is there a use case other than NP04?
      • 10:00
        SW/FW coordination 5m
        Speaker: Alessandro Thea (STFC Rutherford Appleton Laboratory)
    • 10:25 10:35
      Topics for next general meeting 10m

      - slack developments for building DAQ, daq-build tools, ...

      - hdf5 reader