We continue to review all events currently planned for the next sixty days and organizers will be notified if their event must be canceled, postponed, or held remotely. Please, check back on Indico during this time for updates regarding your meeting specifics.
As DOE O 142.3A, Unclassified Foreign Visits and Assignments Program (FVA) applies not only to physical access to DOE sites, technologies, and equipment, but also information, all remote events hosted by Fermilab must comply with FVA requirements. This includes participant registration and agenda review. Please contact Melissa Ormond, FVA Manager, with any questions.

Indico search will be reestablished in the next version upgrade of the software: https://getindico.io/roadmap/


This search is only for public events. Restricted events are not available.


Data Management Meeting


protoDUNE DM (Monday 10/9/2017 PM meeting)

Monday, October 9, 2017

3:02 PM

Old business:

  • Updates from Maxim?
    • Waiting for him to respond whether MCC9 files are good for his DQM
  • Script that copies files into dropbox status:
    • Igor:
      • Haven't tested time-staggering function
      • Script location:
        • On protodune-eos-test6.cern.ch
        • In /home/np04data/copy_dpdata/copy_data
        • Is a python script
          • Configurable with options on command line; has default
            • One input is json file name with constants
        • Generates metadata
        • Run, subrun, file name, "constants"
        • Uses 3rd party xrdcp
        • Target dropbox for the script, input for FTS-light is /eos/experiment/neutplatform/protodune/scratchdisk/daq/data
    • Does 2 functions
      • WA105 data from "their EOS area", bring to Fermilab
      • Copy from, for example, MCC9 areas to dump into a dropbox, renaming to something like "junk*"
  • FTS-light testing
    • Almost working, using ".data" instead of ".dat"
    • Sequence
      • Read metadata file
      • xrdcp to /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
        • Capture checksum from that copy
      • Add checksum to metadata
      • Copy metadata file to /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
  • FTS
    • Watch /eos/experiment/neutplatform/protodune/scratchdisk/test_files/dropbox
    • Configure FTS: done
    • Sequence
      • Read metadata file
        • Declare to SAM with different name, clear that it is junk
      • xrdcp to /eos/experiment/neutplatform/protodune/np04tier0/p3s/input
        • No subdirectories
      • globus-url-copy to scratch dCache
        • /pnfs/fnal.gov/usr/dune/scratch/users/dunepro/…
        • Build directories using metadata
  • FTS for Brett
    • Works - DONE
  • Voms-admin
    • DUNE jobs at CERN Tier-0
    • Storage access is the hard part
      • Works now, but anyone added to DUNE VO once voms-admin is gone won't have access
      • FERRY needs to come up with alternative
    • Discussing with CERN on replacement process
      • Partly in Burt's hands for long term solution


New business:

  • Create Indico meetings for this meeting (Steve will do)
  • Metadata conversations
    • Heidi looking at this
      • Need to spend time working on design
    • Will start with Nova information
    • Andrew: is it good enough to put metadata in the file?
      • Heidi: a "duplicate location"
      • Andrew: "for data ingest, trust the file"
      • Stu: "our solution doesn't use the metadata within the data file, only the auxiliary file"
      • Heidi: please document
      • Steve: would like to learn how to get metadata in/out of file
  • When do we kick off the file movement part of "data challenge"
    • Ruth's email list: dune-proto-sp-dc
    • Docs are in github
    • Benchmark tests as opposed to "data challenge"
      • Needs:
        • Find knob on FTS to avoid calculating the checksum
          • Steve has email from Robert - send to Stu
        • Learn how to delete an entire dataset in sam (Stu)
        • Stu can incrementally do rate tests (next 2 weeks)
          • Check that everything works as np04data
            • Login as root, su to np04data
  • Create a logbook
There are minutes attached to this event. Show them.