DAQ Coordination Meeting

Monday 4 Oct 2021, 09:00 → 10:45 America/Chicago

 https://fnal.zoom.us/j/369714575

09:00 → 09:10 News

Speakers: Alessandro Thea (STFC - RAL), Asher Kaboth (Royal Holloway University of London/Rutherford Appleton Laboratory), Giovanna Lehmann (CERN)

09:10 → 09:20 Specifications document status

Speaker: Giovanna Lehmann (CERN)

DUNE_DAQ_System_Specifications.pdf

09:20 → 10:25 Round Table

09:20 FD Upstream DAQ

Speakers: Jim Brooke (University of Bristol), Mr Roland Sipos (CERN)

Timing system

• Hardware configuration is main outstanding software functionality.  Plan to understand requirements from Timing Sys point of view, then discuss with CCM.
• Firmware (for test stands etc.) ported to new dev board (Nexsys) due to unavailability of Enclustra board.
• Optical calibration system have integrated Timing endpoint.  Will request report in an upcoming UD meeting.
• Firmware for MIB ongoing - will need to plan commissioning of new HW at NP04 carefully...

Readout/TP Firmware

• Main outstanding item is integration of TP firmware with FELIX phase 2 base firmware
  • FELIX AXI4S protocol more complex than previously understood.  Solution proposed by FELIX FW team, under investigation.
• Some patches need to be merged with FELIX base FW (4.8 Gbps links for DAPHNE, link config setup).
• Tests of new TP output format ongoing (goal is to reduce software overhead).
• Need to complete system level integration tests with fake data sources/sinks before proceeding with full integration of TP algorithms.
• WIB2 changes are in the shadow of integration with FLX, but ongoing.
• Plan for validating TPs in data in development.

Readout Software

• Hardware config needs to be implemented in appfwk, as for Timing sys. Discussion needed.
• Some package re-factoring expected (eg FE specialisations / templates.

Ethernet readout

• Some basic tests done, plans gathering momentum
• Meeting this week to plan some tests at RAL.  (Use of Xilinx Alveo card, integration of TP FW, ...)

Operations

• Finalising hardware configuration for HD & VD coldboxes, Vertical slice, NP02 PDS
• Small team at CERN to support.  WIll

09:25 ND Upstream DAQ

Speakers: Patrick Dunne, Patrick Dunne (Imperial College London)

Continuing to implement unit tests. Done so far:

• ZMQ Pub-Sub

• ZMQ Stream raw TCP

• PACMAN Frame

 

Seems to be challenging to make unit tests that take external input files rather than hardcoded

 

Next steps:

• Developing unit tests specifically calling ZMQLink:

  • -  Hard coded arguments/parameters

  • -  Using what was learnt in general ZMQ unit tests to send test messages to it

• Taking part in the readout repo reorganization

• Looking into the new Grafana dashboards from “Pocket” repo to get some plots

09:30 FD Dataselection/Physics Performance

Speakers: Joshua Klein (University of Pennsylvania), Joshua Klein, Philip Rodrigues (University of Oxford)

•  Implementing "horizontal muon" trigger for bottom-side electronics in VD coldbox.  This will be a small change to the trigger Alex Booth created, plus some of the logic we used in PD I.
• Working on list of physics performance goals for PD II (Peeters).
• Working through definitions for MLT coincidence logic.

Major items still pending:

• DAQ model in offline code
• PDS triggering
• High-Level Data Filter effort (some possible progress here)
  • How do we pass information on filtered (removed) events? To data stream, DQM?
  • Should SNB direction be done here or DQM?

09:35 ND Data Selection/Physics Performance

Speaker: Dr Jonathan Hays (Queen Mary University of London)

Current activity:

 - development of unit tests continues

Future activity:

 - data selection planning

09:40 Dataflow

Speaker: Dr Kurt Biery (Fermilab)

Notes from Kurt.

Progress:

• we've started meetings to plan and coordinate the work on the network layer changes (~ a NetMgr-Dev group, similar to the MiniDAQ-Dev group).  We've identified the steps needed for an initial demo, and we are working on those.  For example, Eric Flumerfelt has provided an initial implementation of the NetworkManager code and associated unit tests.
• We're somewhat behind in providing HDF5 "reading" tools (my fault), both for our own use and as an example for offline.  To help remedy this, Adam Abed Abud and Giovanna have put together an initial HDF5Reader.  And, Wes Ketchum is thinking about how to approach this topic in the medium term.

Next Steps:

• Make progress on the NetMgr Demo
• Formulate a plan for HDF5 "reading" (and discuss tweaks to "writing", too)
•  

Cross-WG topics:

• Ron Rechenmacher has made progress in thinking about the use of TRACE in Pocket-DAQ and mentioned that he may be ready to present something soon.
• I believe that I'm now in a position to write some 'Graphical Viewers using Pocket' instructions, but it would be great to confirm that I'm on the right track and not duplicating effort. 
• What's the right forum to ask about next steps and technical questions about the Pocket-DAQ investigations?
• What (possibly crude) tools should we use for communicating exclusive use of computers and electronics at NP04/2?  Google Sheet?
• Nightly builds: what are our targets for reliability?  For the infrastructure used in creating the build; for building 'user' code; for being able to run a minidaq system based on the code

 

09:45 CCM

Speakers: Alexander Tapper (Imperial College London), Alexander Tapper

Progress

• Specification and functionality document
  • No progress in the last week
• Training/new people
  • Provided some details of release testing processes to newcomers (Pierre L. and Alan et al. so far).

Intended work

• Short term:
  • Development of nanorc to support sub-systems, partitions etc.
  • Development of configuration generation to support above.
  • Run number generation and run registry database
  • Further dashboard development and management
• Longer term:
  • Many Kubernetes related studies

Cross group

• Configuration work e.g. recent work by Phil
• Kubernetes work with Facilities (?)

09:50 Data Quality Monitoring

Speaker: Pip Hamilton (Imperial College)

Progress since previous meeting:

• Channel mapping implemented.
• Fourier transforms re-enabled.
• 1st draft of DQM chapter for system specifications document.

Work until next meeting:

• Validating output of new code.
• Restarting work on integration of optical data.
• Integrating trigger record monitoring.
• Further fleshing out DQM specifications chapter.

Cross-WG points:

• Where should we look/ask for external constraints on DQM system specs (e.g. available storage)?

09:55 Facility

Speakers: Prof. Alec Habig (Univ. of Minnesota Duluth), Bonnie King (FNAL)

Since Last Meeting

VD Coldbox Cabling discussion

• Giovanna installing devices Monday/Tuesday next week
• Roland has prepared FELIX server for more (026?)
  • Can we connect IPMI to the 10G switch?
• Not sure how accessible rack will be, so asking Xavier about managed PDUs for non-server devices

Planned work

Stream upgrades

• If we can reserve a few machines on CC7, we should move forware with CS8 upgrades
  • ~5 machines to service coldbox
• FELIX has been validated on 8
• Need a native CS8 DAQ build - Pengfei
  • Most technical issues have been resolved, builds exist for previous releases
• Upgrade disk server 004 to CS8, RAID10
  • General space cleanup, announce to np04-daq-dev eGroup
  • Pengfei to benchmark with fio
• specification document contributions

For further discussion

Kubernetes

• Will be joint CCM/Facility effort
• vanilla docker images (easier config for Grafana etc)
• podman investigatons: contact Ron R

10:00 SW/FW coordination

Speaker: Alessandro Thea (STFC Rutherford Appleton Laboratory)

Since Last Meeting

SPACK 

• External packages suite made available in SPACK (Patricia);
  • Working on debug builds
• Prototype porting of daq-buildtools development environment prepare by John;
• John working through daq packages;
• Learning how to work in SPACK environment.

Continuous Integration

• Automatic deployment of nighly builds activated by Pengfei

Planned work

CVMFS-based Python environment

• Hoping to wrap it up soon (Pengfei)

SPACK 

• Continue to work on DAQ packages in SPACK and build an understanding of how best use SPACK;
  • Development environment;
  • Deployment options.

Centos8

• Discussed at the Facility meeting on Friday.
  • Agreed to aim at deploying Centos8Stream cluster-wide at high priority to validate whether it's feasible to migrate before ProotoDUNE-II
  • Will push for a DUNE-DAQ native Centos8Stream UPS build.

For further discussion

• None

10:25 → 10:35 Topics for next general meeting