DUNE Data Management Meeting

Tuesday 17 Dec 2024, 09:00 → 10:05 US/Central

203 (Feynman Computing Center)

This is the weekly DUNE Data Management Meeting. We discuss the Data Management Project for the upcoming DUNE experiment, and also data management operations issues. Attendees are people who are actually working on data management plus higher DUNE computing management as required.

1 Standing Item: Rucio Token Development Status

Speakers: James Perry, Steven Timm (Fermilab)

Rucio--

according to James at ops meeting yesterday

He has managed to make all the various test code token stubs work but had to make some rucio code changes

to make them work.. need to get this documented in our issue tracker and get the relevant issues in the rucio tracker documented in our tracker.

 

There was long discussion in the ops meeting yesterday about Rucio's plan to be a token issuer on own.

Seems the drivers are that they have too many tokens in flight, are concerned that IAM can't handle rate

and the SLA is not for 24x7 support on IAM either from CERN operators or the INFN developers.

(neither is CILogon)

Huge bureaucracy wouldybe needed to get an extra token issuer accepted at Fermilab or anywhere else

even alpha rucio upload is not envisioned anytime soon, based on last rucio meeting.

Can we start to get token subjects loaded into rucio for user auth?  Brandon thinks so..

There's another whole daemon that has to be configured rucio-oauth-manager  daemon 

Can get IAM tokens and test with that in the interim.

 

2 Standing item: Proposed upgrades / configuration changes

Speakers: Brandon White (Fermilab), Marc Mengel (Fermilab), Steven Timm (Fermilab)

Rucio v35_6 was deployed last Tuesday 12/10

Metacat v4_0_2 was deployed 2 weeks ago.

Steve working on installing latest ingestd and declad at CERN

Have made integration instances now protodune-ingestd-test.cern.ch and protodune-declad-test.cern.ch

Native python 3.11 available in Alma9 

 

Transfers of 2.5PB from tape-backed to persistent--one increment just after holidays and second one in month of March

 

 

3 Standing Item: Datasets pending DM action

4 declaration of virtual files in Metacat as part of merging workflow

Speakers: Aaron Higuera Pichardo (Rice University), Elisabetta Pennacchio, Marc Mengel (Fermilab)

The files from /pnfs/dune/scratch/users/epennacc/merging/fardet-hd_supernova_0_500_artcat.fcl_20241217T041349

have been merged but the individual constituent files of the merged files were never declared to rucio or metacat

Thus we can't add the parents into the merged file

 

It was agreed that these merged files could be declared without parent.s

 

5 Where to put production output

Speaker: Doug Benjamin (Brookhaven National Laboratory (US))

QMUL - has 485.3 TB free

PRAGUE - has 531.3 TB free

Please split the output between the two sites.