DUNE Workflow/load Development 9 Sept 2021
------------------------------------------

https://indico.fnal.gov/event/50838/

Present: Andrew McNab (AM) (chair/notes),
         Raja Nandakumar (RN)
         Doug Benjamin (DB)
         Fergus Wilson (FW)
         Ken Herner (KH)
         Steven Timm (ST)

Apologies: Chris Brew (CB)

Intro+news
----------
- Raja taking on workflow developmen half-post at RAL-PPD
- Chris suggested by email that we use CDR use cases to motivate designs 
- Doug: will each site have cache? What about inefficiency
- Andrew: the CDR is written in terms of services

GlideInWMS/HTCondor
-------------------
- Was a meeting yesterday assessing jobsub lite. Key to getting more
  people to submit. 
- GlideInWMS stakeholders meeting yesterday
  https://indico.fnal.gov/event/50032/contributions/219632/attachments/146985/188068/Glideinwms-StakeholdersMeeting-20210908.pdf

User tools
----------
- See above
- Beta version of jobsub lite out for testing. Plan is to try it with
  DUNE Global Pool.
- Doug offers to test from BNL
- Raja needs rucio-lite for ETF test jobs with storage

Workflow/production
-------------------
Workflow Dispatcher design: https://docs.google.com/document/d/1K5yijin-MAU-BXfa1rULfLFd1NjDQycitYgUK3Amiz8/edit?usp=sharing
- See comments on that document
- Major points
-- Want to avoid creating internal IDs for files etc that exist in MetaCat
   or RUCIO
-- But want to avoid doing joins on strings so use external numeric IDs 
   where available
-- Design can deal with the desirability of giving requests higher priority
   as they near completion to get them done
-- Will need to agree a unified set of terms (request vs project etc) across
   Workflow and Data. Document deliberately uses request and stage to avoid
   existing project meanings
-- HEPCloud currently relies on job pressure so as long as things appear
   that way it will pick up the requirement to create VMs on Amazon etc
-- We could merge the Pilot Factory and Generic Job Factory agents. This
   could be done by getting the existing Pilot Factories to run the 
   generic job client instead of the HTCondor client
   
HEPCloud
--------

- 

Pilot Factories
---------------

-

Other topics
------------

AOB
---

Meetings continue in this slot, 8am CT Thursdays.