DAQ Coordination Meeting

Monday 17 Jun 2024, 08:00 → 09:30 America/Chicago

 https://fnal.zoom.us/j/369714575

08:00 → 08:05 General news

Speakers: Alessandro Thea (STFC Rutherford Appleton Laboratory), Asher Kaboth (RHUL), Roland Sipos (CERN)

08:05 → 08:20 Actions from previous meetings

Speakers: Alessandro Thea (STFC Rutherford Appleton Laboratory), Asher Kaboth (RHUL), Roland Sipos (CERN), Wesley Ketchum (Fermi National Accelerator Laboratory)

08:20 → 08:30 Run Coordination news

Speaker: Wesley Ketchum (Fermi National Accelerator Laboratory)

Beam should start Wednesday June 19 at 18:00 CEST.

 

Patch release:

• project page: https://github.com/orgs/DUNE-DAQ/projects/5
• Targetted for what we need for smoother operations for EHN1 + simple additions that can be added with very little risk (e.g. ICEBERG TC types, config fix for dual nic)
• Most PRs are ready for merge (Wes will push them to be merged today). Goal will be everything merged in by end of today CERN time, tested and tagged tomorrow, ready to hand off to software coordination by the morning FNAL time.

 

EHN1 activities (in some version of priority for readiness for beam next week)

• Configuration management
  • Creating SSH+K8s configs together: complete
  • Providing cleaner management of configs for shifters / experts to create: largely ready
    • CTB+trigger team working on preparing beam configs
• Performance at higher trigger rates
  • Kernel isolation on readout servers: complete.
    • Noticed improvement in burst sizes on ethernet readout, though still see some correlations in missed packets with slowdowns on data requests over 10G network. Hopefully ok for beam run, though offline DQM will check.
  • Use of new storage server / understanding network bandwidth and performance limitations: ready for beam
    • See clear limitations when pushing at high rates, but able to run rather stably with a 30 Hz instantaneous rate (3 ms readout windows) with a 'fake beam spill' configuration. (average 5-10 Hz overall rate)
  • Tuning of timeouts / operational parameters, trigger config: mostly complete, final testing in progress
    • Long data challenge runs last week and over weekend with TPC-only data and no TPG were success
    • Aiming to add PDS today (PDS calibration work ongoing) and TPG today or tomorrow
      • Spent lots of time with PDS group Friday debugging readout problems they had (in the end due to a bad threshold setting on the DAPHNE)
      • Artur preparing trigger configs, collection only for the beam run to start with.
• DQM for shifters
  • Script to transfer data to rscratch 'analysis' location: complete
    • First iteration from Wes complete, PDS group helping with some additional options
  • automated analysis scripts: in progress, but coming together
    • Examples in place, 'event display' script working, need to add PDS and WIB scripts (they largely exist)
  • automated plotting scripts
    • Examples in place for event display, http://np04-srv-015.cern.ch:8005/event_display/apa1_plane1
• Timing commands as triggers for PDS calibration on APA1: complete?
  • Triggers working / DAQ seems to be ok. PDS understanding what they see -- worried about some timing alignment/synchronization potentially, but not clear it's a problem from our side yet at all
• Operations preparation
  • Expert training and documentation: started
    • First training session last week; daily meeting started today; expert rotation started
  • Configurations for the beam trigger: todo, but in progress
    • See above
  • Shifter documentation review and DAQ checklist: todo
    • Wes to get ready this week
• CRT readout: minor progress
  • fddaq-v4.4.2 area prepared. Wes to pass along instructions...

 

08:30 → 08:45 Working groups round table

Speakers: Adam Barcock (UKRI STFC), Alec Habig (Univ. of Minnesota Duluth), Alexander Tapper (Imperial College London), Artur Sztuc (University College London), Bonnie King (FNAL), David Cussans (University of Bristol), Jonathan Hays (Queen Mary University of London), Joshua Klein, Kurt Biery (Fermilab), Pierre Lasorak (Imperial College London), Roland Sipos (CERN), Stoyan Trilov, Wesley Ketchum (Fermi National Accelerator Laboratory)

Timing System

Currently, no problems reported with timing endpoints that can't receive clock/timestamp (that haven't been worked around)

List of endpoints at NP04 at https://cernbox.cern.ch/s/OOIoDZuX3FvdiXA

Status for timing endpoints on NP04:

1. WIBs using out of date endpoint firmware which is believed to be the cause of problems accessing endpoint registers from the master. The good news though is that the "downlink" works and that "uplink" transmission is possible from most of the WIBs in a crate. The fact that the problem only affects certain WIBs and all traffic goes through the same PTC indicates that this is a WIB/WEIC/PTC issue, not a central timing system issue
2. The DAPHNEs timing endpoints work ( @Manuel Arroyave has reported issues with the timing endpoints in the DAPHNEs. However, if they don't reset the boards the link is stable. I suspect this might cause problems in the future but I don't think there is anything we can do at the moment ( suggestions welcome) )
3. The control units for lasers P1 and P2 can receive clock and timestamps. Return path has been tested.
4. Timing hardware in CRT rack has been updated. Delivering 62.5MHz and 1PPS.
5. CTB is receiving clock and timestamps. Issues with return path fixed earlier today ( will need a minor addition to a future patch release on CTB side)
6. Fanout-1 ( delivering timing signals to NP04 ) locked to 10MHz clock from beam instrumentation White Rabbit network. Will attempt to install an additional Charon (10MHz/1PPS) endpoint in NP04 barracks and check that the 1PPS from the DTS is aligned with the 1PPS from a White Rabbit endpoint.

We will run the endpoint monitoring service when the system is stable ( to avoid filling Grafana with timing system error messages about dead timing endpoints ).

Trigger

Various fixes and updates related to np04 running:

1. Online latency monitoring finalised, PR pending.
2. New TC types for CTB. Can run CTB successfully with MLT merging, logic works without issues.
3. New monitoring for the trigger: buffer occupancies, zipper occupancies. Implemented, Grafana update pending.
4. Bugfix in the TC & TA adapter / buffering that would prevent retrieving all TCs and TAs from the buffer if they have identical start time.
5. New ability to run TPG without software triggers.
6. Making naive TPG consistent with AVX.
7. Changed trigger's thread names to be all unique: will allow CPU pinning if needed.

08:45 → 09:00 Activity coordination round table

Release coordination
Iceberg
ND test setups
EHN1 integration
SW coordination

Speakers: Bonnie King (FNAL), John Freeman, Kurt Biery (Fermilab), Patrick Dunne (Imperial College London), Wesley Ketchum (Fermi National Accelerator Laboratory)

ICEBERG

• Running fddaq-4.4.1 in ICEBERG and reading out all 9 FEMBs thanks to Kurt, Ron, Roland
• Matthew M. created CPU pinning file
• justintime and Pocket being used

09:00 → 09:05 Actions