protoDUNE DM (Monday 10/9/2017 PM meeting)
Monday, October 9, 2017
3:02 PM
Old business:
- Updates from Maxim?
- Waiting for him to respond whether MCC9 files are good for his DQM
- Script that copies files into dropbox status:
- Igor:
- Haven't tested time-staggering function
- Script location:
- On protodune-eos-test6.cern.ch
- In /home/np04data/copy_dpdata/copy_data
- Is a python script
- Configurable with options on command line; has default
- One input is json file name with constants
- Generates metadata
- Run, subrun, file name, "constants"
- Uses 3rd party xrdcp
- Target dropbox for the script, input for FTS-light is /eos/experiment/neutplatform/protodune/scratchdisk/daq/data
- Does 2 functions
- WA105 data from "their EOS area", bring to Fermilab
- Copy from, for example, MCC9 areas to dump into a dropbox, renaming to something like "junk*"
- FTS-light testing
- Almost working, using ".data" instead of ".dat"
- Sequence
- Read metadata file
- xrdcp to /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
- Capture checksum from that copy
- Add checksum to metadata
- Copy metadata file to /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
- FTS
- Watch /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
- Configure FTS: done
- Sequence
- Read metadata file
- Declare to SAM with different name, clear that it is junk
- xrdcp to /eos/experiment/neutplatform/protodune/np04tier0/p3s/input
- globus-url-copy to scratch dCache
- /pnfs/fnal.gov/usr/dune/scratch/users/dunepro/…
- Build directories using metadata
- FTS for Brett
- Voms-admin
- DUNE jobs at CERN Tier-0
- Storage access is the hard part
- Works now, but anyone added to DUNE VO once voms-admin is gone won't have access
- FERRY needs to come up with alternative
- Discussing with CERN on replacement process
- Partly in Burt's hands for long term solution
New business:
- Create Indico meetings for this meeting (Steve will do)
- Metadata conversations
- Heidi looking at this
- Need to spend time working on design
- Will start with Nova information
- Andrew: is it good enough to put metadata in the file?
- Heidi: a "duplicate location"
- Andrew: "for data ingest, trust the file"
- Stu: "our solution doesn't use the metadata within the data file, only the auxiliary file"
- Heidi: please document
- Steve: would like to learn how to get metadata in/out of file
- When do we kick off the file movement part of "data challenge"
- Ruth's email list: dune-proto-sp-dc
- Docs are in github
- Benchmark tests as opposed to "data challenge"
- Needs:
- Find knob on FTS to avoid calculating the checksum
- Steve has email from Robert - send to Stu
- Learn how to delete an entire dataset in sam (Stu)
- Stu can incrementally do rate tests (next 2 weeks)
- Check that everything works as np04data
- Login as root, su to np04data
- Create a logbook
- Unassigned (Adam, Steve, or Anna), too busy now
- Anna has an ECL version for production
- Separate categories or new ECL?
- Igor suggests separate
- Service ticket (Steve White) to request new tags
There are minutes attached to this event.
Show them.