Author: Heidi Schellman
Company: Northwestern

Summary of the Edinburgh Workshop, October 8-9, 2018

People contributing to data movement and processing for DUNE, and interested parties met in Edinburgh October 8-9, 2018.   There were around 35 attendees, mainly from the UK and Fermilab but with experts from CERN also present.  We thank Andrew Washbrook, Peter Clarke and the staff at the University of Edinburgh for an extremely productive meeting. 

The indico page is: https://indico.fnal.gov/event/18094/

There were major topics of discussion, the big picture, data management, authentication and workflow management. 

The big picture - DUNE has successfully taken, stored and reconstructed data from ProtoDUNE using existing facilities and tools.  UK (FZU, IN2P3, CERN) have the basic infrastructure needed to perform analysis of the reasonably sized protoDUNE data.  The data size from the current run is already greater than 1 PB and growing CPU needs for simultaneous keeup processing of raw data and simulation are of order 10,000 cores with 2-4 GB of memory and this has been available through the use of the broad global resource. 

Real DUNE will be more problematic with existing tools as the far detectors are at a minimum 25 times larger and the near detector is likely to generate at least as much data volume as the far detector.

Supernova events are 3-4 orders of magnitude larger.

The overall data rates from DUNE will not be that much larger than protoDUNE due to the lower event rate.  Reconstruction times/byte are also not that far out of current ranges for large experiments. The difference is that the single event size brings us to new territory in computing. Dealing with “events” which are 25-100,000 GB in size is one of the major R+D projects we will need to undertake.

Big picture continued. 

DUNE is one of a suite of “Intensity Frontier” experiments at Fermilab which share common software (Larsoft, GENIE) and computing infrastructure.  But DUNE is in addition (through ProtoDUNE)  a CERN experiment and  a large-scale international collaboration with significant numbers of collaborators in Europe and Asia who need and have access to computing resources.  DUNE needs to view itself as a global experiment and find global (or multiple) solutions to its needs for calibration, simulation, reconstruction and data analysis which serve all collaborators.

Extending our collaborations beyond the IF experiments and LHC by collaborating with astrophysics experiments on similar scales (SKA, LSST … )  would be desirable.

The WLCG provides a prior model of international collaboration.  We do not propose to join the WLCG but to work with the WLCG on common projects which build on the substantial experience and code-base from 20 years of    
LHC computing. 

DUNE has time to evaluate and propose common tools due to the start in the mid-2020’s and it is in fact unwise to make irrevocable commitments to technologies 8 years out from data taking.  The best solutions will take into account the considerable prior investment in both LHC and smaller experimental efforts with an emphasis on incorporating lessons learned and simplicity.

Data management - Prototype Rucia adaption for DUNE is underway and is going well.  We will need to determine how to integrate the existing data catalog with the Rucio framework and ensure that the DUNE implementation can scale to a worldwide experiment.  ProtoDUNE samples are now available and known to Rucio at CERN, FNAL and Manchester and Rucio interfaced to sam was able to successfully obsolete data files on EOS with correct  tracking in the sam data catalog.

Authentication - This will remain a challenge as the security landscape continues to change and national agencies will likely require different solutions.  This is a problem that must also be solved for the WLCG and DUNE will likely benefit from their work. 

Workflow management - several multi-experiment solutions exist - GlidenWMS and DIRAC were the most discussed - my takeaway is that we should do a serious comparison of the different systems and work for a common solution (or maximal interoperability) which preserves the best parts of the present solutions.  At the same time making the interfaces more general to support multiple present and future facilities (e.g hooks to batch systems rather than dependencies on those systems.) should be a goal.  One may point out that the current hybrid system using FNAL jobsub/condor/GlidenWMS and WLCG resources has worked quite well for ProtoDUNE thanks to years of previous effort in integration. 

Other - DOMA and other WLCG projects are of considerable interest to DUNE and we would like to collaborate on areas of mutual interest.

General summary - the workshop was extremely successful in completing the lines of communication which have already worked well in getting data store and grid jobs running on global grid sites.   It was very nice to meet our colleagues in person, especially after the success in establishing the protoDUNE data processing and simulation campaigns. 

Goals going forward

        • extend to other DUNE insitutions/countries through the formation of the DUNE Computing Consortium
        • Assess the near-term and long-term data model for DUNE which will drive  the resource specifications for          DUNE computing in the near 2020’s.  An internal task force is already being formed to inform the upcoming Conceptual Design Report. 
        • Evaluate workflow management requirements using existing systems as a model. 
        • In all of this, maximize synergies with other experiments facing similar issues including both the WLCG and smaller expts such as FNAL IF, CERN fixed target, SKA, HyperK and T2K. 
        • Define Consortium work packages for projects to submit to national funding agencies.

Heidi Schellman, Oregon State University