DUNE Core Computing Meeting

Monday 18 Oct 2021, 09:30 → 11:05 US/Central

https://fnal.zoom.us/j/336647590

Andrew McNab (University of Manchester), Andrew Norman (Fermilab), Brett Viren (BNL), David Adams (BNL), Heidi Schellman (Oregon state), Michael Kirby (FNAL), Steven Timm (Fermilab), Thomas Junk (Fermilab)

We will have the DUNE General computing meeting:

https://fnal.zoom.us/j/336647590

Meeting ID: 336 647 590

standard computing password, will be sent by email for those who don't know it.

One tap mobile
+16699006833,,336647590# US (San Jose)
+16465588656,,336647590# US (New York)

Dial by your location
        +1 669 900 6833 US (San Jose)
        +1 646 558 8656 US (New York)
Meeting ID: 336 647 590
Find your local number: https://fnal.zoom.us/u/adJsct1aBR

News from Computing Management etc: News from Computing Management

1 News from Computing Consortium

Speakers: Dr Andrew McNab (University of Manchester), Heidi Schellman (Oregon state), Dr Michael Kirby (FNAL)

Slack channel #dealingwithlargedata

email list dune-large-data

Join these for discussions of how we actually deal with large data.

 

2 News from partner projects (OSG, WLCG etc)

Andrew McNab on the benchmarking project

Speaker: Andrew McNab (University of Manchester)

GDB - report from network data challenges from collider experiment

Rucio/condor workshop reports.  

GDB last week. Network DC, RUCIO and HTCondor workshops. See agenda for slides: https://indico.cern.ch/event/876794/

3 HDF5 discussion

HDF5 discussion

Speakers: Heidi Schellman (Oregon state), Thomas Junk (Fermilab)

Agenda for last week's HDF5 discussion

hdf5 equivalent of config_dumper

we need to have some kind of metadata in the hdf5 files. 

(normal tuples also don't have art metadata, so need it for them as well)

working on a header-only reader- has a specialized exception ERS package that may introduce more dependence than we want. 

Working on channel map. 

Compression out of the daq - do it on SSD?  Cost estimate?  LDRD

Operations

4 protoDUNE II Horizontal Drift (np04) operational status

Speaker: Dr Michael Kirby (FNAL)

5 protoDUNE II Vertical Drift (np02) operational status

Speakers: Elisabetta Pennacchio, Dr Elisabetta Pennacchio (IN2P3)

6 Operations Meeting report

Sites, networking, and central DUNE computing services (GlideinWMS, SAM, RUCIO, CRIC, ETF, Databases).

Speaker: Dr Steven Timm (Fermilab)

Progress on rucio -> RAL and Manchester

GPU's available at QM. A100's

7 Production status report

https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&from=now-30d&to=now-5m&var-experiment=dune

Speaker: Dr Kenneth Herner (Fermilab)

8 User analysis status report

Software Development

9 SP/DP/ND Software Development

(This item should probably be split into smaller units?)

Speakers: Dr Elisabetta Pennacchio (IN2P3), Elisabetta Pennacchio, Mathew Muether (Whichita State), Dr Mathew Muether (Fermilab)

10 Databases Development

Speakers: Prof. Norm Buchanan (Colorado State University), Norman Buchanan, Paul Laycock (Brookhaven National Laboratory)

DB report.  Centralized database for conditions. Next meeting will be metacat... (runs db?)

new postdocs.  Organization meeting once they arrive. 

11 Software Management

Speakers: David Adams, David Adams (BNL), Etienne Chardonnet, Jake Calcutt (Michigan State University), Dr Kurt Biery (Fermilab), Dr Thomas Junk (Fermilab)

12 Frameworks Task Force

Speakers: Dr Andrew Norman (Fermilab), Paul Laycock (Brookhaven National Laboratory)

Computing Services Development

13 Data Management development

SAM, RUCIO etc

Speakers: Dr Andrew Norman (Fermilab), Igor Mandrichenko (FNAL), Dr Steven Timm (Fermilab), Dr Stuart Fuess (Fermilab)

Andrew McNab has gotten rucio client to work within jobs. 

need python3 and then install pip -> rucio clients. Bit complicated. 

will we put the job environment in cvmfs? containers? 

how does this work on the grid? Do Robert and Brandon have a script that does this? 

in the end we need a ups/spack product for rucio client

 

14 Workload and Workflow development

HEPCloud, GlideinWMS, POMS, etc. https://github.com/DUNE/dist-comp/labels/WMSdev

Speakers: Dr Andrew McNab (University of Manchester), Dr Andrew Norman (Fermilab), Kenneth Herner (Fermilab), Dr Steven Timm (Fermilab)

15 Information Systems development

CRIC

Speaker: Andrew McNab (University of Manchester)

16 Monitoring Systems development

ETF site monitoring, perfSonar network monitoring

Speakers: Christopher Brew, Heidi Schellman (Oregon state), Peter Clarke, Raja Nandakumar, Steven Timm (Fermilab), Prof. peter clarke (University Edinburgh)

17 Documentation and Training

Speakers: Claire David (York University / FNAL), Heidi Schellman (Oregon state), Dr Michael Kirby (FNAL), Dr Thomas Junk (Fermilab)