We will have the DUNE General computing meeting:
Heidi Schellman is inviting you to a scheduled Zoom meeting.
Topic: General DUNE Computing meeting
Time: This is a recurring meeting Meet anytime
Join Zoom Meeting
https://fnal.zoom.us/j/162652254
One tap mobile
+16699006833,,162652254# US (San Jose)
+16465588656,,162652254# US (New York)
Dial by your location
+1 669 900 6833 US (San Jose)
+1 646 558 8656 US (New York)
Meeting ID: 162 652 254
Find your local number: https://zoom.us/u/acoyhG3WuF
Present: Heidi S., A. McNab, A. Mazzacane, J. Perry, K. Herner, N. Buchanan,
Peter Clarke, Petr Vokac, Raja Nandakumar, Rob Currie, Robert Illingworth,
Stu Fuess, Teng Li, Tom Junk, Xavi Espinal, Doug Johnson, David Adams,
Notes on dune meeting file usage discussion.
Persistent dcache is full, jmalbos filled up with 10K 2G files
Can we have quotas on persistent? Not right now.
Should we push back and require quotas?
We put in a request as part of SC-PMT
Still waiting for SC-PMT to send back final report.
Kirby—should push for quotas
But who’s going to do the management?
CDF gave quotas to the physics groups,
Also need to push for expanding actual allocation.
But have to push back and tell the physics group that they have to manage it.
Mc.Nab—shouldn’t put much beyond the terabyte level.
Past
Kirby—no reason that a sample that everyone wants should reside in persistent
Herner—If everyone really wants it then it will stay in the RW Cache
Put a AAREADME in that directory
Tom—not sure it has been accessed recently.
Steve—will make the documentation of how we did the Sheffield data backup
Not ready to push user data with rucio
Robert—we can move it around—but do we understand how to keep track of it
Any large data set has to be in sam4users.
========
James Perry / Teng. /Peter Clarke—jumps in because have to be done with 10
Peter wondering what is the followup post-the JLAB discussion, should they start working on metadata for Rucio— Robert/Heidi—think we want a standalone file metadata server outside of rucio.
Steve—need to make requirements
Robert—want to streamline the data set definitions.
Heidi—people are making huge numbers of DSD’s in project.py, annoying.
—
Back to quota discussion--Heidi if we get more persistent quotas can we divide them up among the groups. Generic user space also useful.
Xavi—@ CERN they are managing the storage via physics groups and quotas.
Kirby has emailed dcache people to begin the discussion.
Have to figure out what is going with CERN user usage at once. STeve—right now it is small,
< 1% of all EOS content.
Tom==would like physics groups stuff be readable by the collaboration.
—
Stu—tape and small files don’t get together.. but other than that dcache doesn’t care.
Are small files worthy of being on tape?
Tom—high turnover, education issue
Heidi—first steps—quotas, group areas, sam4users document ions
Whatever in persistent goes to tape backed
Scratch is a good solutions
Xavi—main experiment of LHC are shielding the tape from the users
—
Roundtable
Steve—data management, all was covered in 9AM call
Ken Herner—production, have figured out how to split the DP MC into two phases and
Reduce amount of memory needed.
Photon libraries, Alex Himmel and John Losecco looking, may have a solution.
Need to ask for time on protodune-sp ops meeting on Friday and protodune-dra reconstruction.
Software mgmt (Tom) —nothing new this week. New release maybe next week.
One of the keepup logs run 5849 took longer than normal due to 7.5ms
James Perry—code for URL signing for S3 and Swift merged into Rucio Github
Teng Li—have done some data testing locally and now working with Kevin Retzke to set up the ingest points at Fermilab.
Robert has just started a bunch of rebalancing, should have something to see.
Anna Monitoring the NAS volumes--should talk more with the monitoring team to see what can be provided.