EHN1 Operations and fddaq-v4.3.0 Release Preparation

Tuesday 13 Feb 2024, 07:30 → 09:00 US/Central

Kurt Biery (Fermilab), Wesley Ketchum (Fermi National Accelerator Laboratory)

Zoom:  https://fnal.zoom.us/j/92847431247

 

07:30 → 08:00 Operations

Links:

• EHN1 Operations Issues

Speaker: Wesley Ketchum (Fermi National Accelerator Laboratory)

DAQ Hardware Activities:

• Installation/use of of NP02 readout switch: Wes, Roland, Alessandro
  • Still no connection of readout servers, investigating with network configuration experts at CERN
• Todo -- update the rack layout diagram and check system inventory: Wes

---

Standard DAQ monitoring runs:

In progress (Wes):

• New dunedaq-fddaq-v4.3.0-rc1-dev area setup with new configurations being validated
  • CTB as HSI working (needs ctbmodules update)
  • All WIBs working, TPC TPs in SimpleThreshold
  • 10Hz runs with 6 dataflow apps and tpstream writer working
  • Still todo
    • Inclusion of DAPHNE readout
    • Use of k8s configs
    • Setup of DQM tests / area
• Highest-priority issues for basic DAQ running (distracting warnings, elog posting):
  • DAPHNE configs not requirng TP buffers
    • Kurt to help followup (maybe include Roland in conversation as well)
  • Warnings of overlap with previous window
  • Warnings about late data packets when pinning threads
  • Elog automated posting

Working on updating documentation alongside that:

• Main page: https://twiki.cern.ch/twiki/bin/view/CENF/DUNEDAQProto , some reordering.
• New instructions for setting up an area for running: https://twiki.cern.ch/twiki/bin/view/CENF/SetupDAQAreaEHN1

---

Ongoing / to-be-done DAQ testing involving running with hardware ~this week:
These all inform a baseline configuration for readout of NP04.

• K8s-based testing: Tiago, Juraj, Pierre
  • Notes from last meeting:
    • APA4 running largely successful (though missing data from readout id 440), but when tried to raise to 10Hz hit inhibits ... need to followup on how many data writers are being used.
    • Need to do a comparison with the bare metal.
  • Status
    • Working? Ready for full integration / testing with the full release candidate
• Dataflow with new np04-srv-005 server: Wes, Alec and Bonnie, Kurt + Steve Timm to check metadata and transfer
  • Notes from last meeting
    • Issue on disk setup not being stable after reboot. Alec has detailed report of what was done in #np04-sysadmin channel. Additional documentation needed?
    • Disk areas now set: /data0 and /data1 for spinning disk, /ssddata2 and /ssddata3 for SSDs
      • Both Kurt and Steve have been following the discussion ... metadata and transfers in place for those final locations?
    • Bonnie to run a phoronix benchmark test
  • Status
    • Reconfigured, but all with RAID10.
    • Performance tests show 3000 MB/s ... not as good as expected. Plan to rerun tests with RAID0
    • Bonnie has a list of tests from Roland to run -- Bonnie starting thank
    • Outside of performance tests ... ready for use?
• NP04 readout servers with multi-NIC running, use of new readout server (np04-srv-031): Alessandro, Roland, Wes
  • Notes from last meeting
    • Alessandro still to do the update on the configuration, aiming for tomorrow morning
    • Default servers to use for now are the ones we have been using: 21, 22, 28, and 29. 31 is plugged into the NP04 readout switch
  • Status
    • Configs updated and NP04 readout working on servers 21, 22, 28, and 29
    • Has server 31 been testing? Do we want to include configs for it?
• CTB testing: Jon, Stoyan, Wes
  • Notes from last meeting
    • There is a change in the HSI word type that will need to be folded in ... will be needed for beam instrumentation. Stoyan knows details.
  • Status
    • Met last Friday, got things working from a v4.2.1-based area
      • Two issues pushed to later: faulty error on stop of run, error messages when running two low-level-trigger (LLT) emulators
    • Tested with release candidate, and need a minor update in ctbmodules (PR in)

---

Other testing for ~this week:
Goal to have preliminary implementations ready for this week/early next week.

• Automatic elog posting: Pierre, Roland, Enrico.
  • Notes from last time
    • We think this can work for this release candidate
    • PDSP elog has been moved by Enrico to OpenShift
    • elisa update that requires new dependency on python repo -- Pierre hasn't yet merged the PR on elisa for that to work, so needs to be done
    • trivial nanorc PR to update addresses of the services
  • Status
    • Still having issues when 2FA is always on (like it is for Wes :/ )
• DQM justintime updates and deployment: Vitaliy, Wes
  • Status
    • Vitaliy has
• Readout servers hugepages/NIC configuration after reboot: Pat, Roland, Alessandro
  • Notes from last time
    • Still todo. Not highest priority. Needs input from Roland and Alessandro
  • Status
    •  

---

Beyond this week

• CRT integration
  • Notes from last meeting
    • Matt Murphy reported preliminary software reading data from file and creating HDF5 daq files.
    • Needs further input on analysis of HDF5 files (Wes to help)
    • Next step for software? DAQ experts to review what's been done, help suggest/implement changes, and begin strategy of integration with real-time data
      • Would be good to consolidate that into a daq-deliverable and provide a lot of the background information to help with review
    • Need to followup on CRT integration with the CTB and trigger
  • Status
    • Matt shared code with Kurt and Roland
    • No other updates?
• Timing/trigger support for calibration modes
  • Notes from last meeting
    • Plan is for everything to go through the physical HSI rather than the CTB. Will require some updates to the HSIEvent most likely.
  • Status (if any to report)
• TP validation
  • Notes from last meeting
    • Adam made some issues for operations; finalizing absolute running sum and then would like to do some testing with the APAs (will need to decide on whether to test with noise or pulser data)
  • Status (if any to report)
    •  
• Other?

08:00 → 09:00 Release Preparation

Topics for fddaq-v4.3.0-rc1:

• how is testing going?
• are there known changes that are needed?
• which of the PRs that have been created in the last 24 hours are for fddaq-v4.3.0 and which are for v4.3.1?
• The Tag Collector now has a column for rc2 (if that will be needed)

Discussion of goals for subsequent fddaq-v4.3.x releases:

• Feature list for NP04
• Comments and questions from Kurt:
  • What are reasonable target dates for fddaq-v4.3.1 and fdaq-v4.3.2? February 23 - March 1, and March 28-April 5?
  • How will miscellaneous changes be handle for v4.3.1? Can they be merged to production/v4 branches, or should a different branch be used? Example
    • Here is a view of candidate fddaq-v4.3.1 miscellaneous changes. Some of these will likely be deferred even further

Speaker: Kurt Biery (Fermilab)

fddaq-v4.3.0 - contributing to prep-release branches now

fddaq-v4.4.0 - contributing to production/v4 now

fddaq-v4.4.0 - contributing to prep-release branches during the release candidate phase

 

 

prep-release/fddaq-v4.3.0