Attendees: Rob Ross, Chris Jones, Doug Benjamin, Ken Herner, Kyle Knoepfel, Lisa Goodenough, Paolo, PeterVG, Phillippe Canal, Saba Sehrish, Suren Byna, Tammy Walton, Liz Sexton-Kennedy
**Paolo:**
- Papers to acknowledge the project. Author list to use.
- Would like an IOS image for the web site. Data throughput requirements?
- Reminder that this is just "flashy" thing, PMs to look at, not deeply technical.
**Darshan for DUNE.** Ken:
- Same run as he showed last time, cleaner plots (just LArSoft executable).
- Thinks he has convinced himself that the results re: reading are in fact accurate.
- Q: Why don't the columns in the File Count Summary add up?
**Darshan for ATLAS.** Doug: Massive flameouts that are hard to debug. Sending an update to Shane and CC'ing Rob/Peter. Does not work when submitting via Panda and the pilot.
**Darshan for CMS.** Chris: Nothing new.
**HDF.** Saba: No update. Had a brief chat with Chris, and she plans to use Chris's framework from now on, so we have the whole chain working. By "Chris's code" we mean the "testing framework" that Chris will present later in the meeting.
This code was in GitHub at one time; it is still there. Will eventually move to IOS GitHub. Next week is out, the following list there will be progress.
**Testing Framework and Early Performance Results.** Chris Jones.
Purpose:
- mimic characteristics of HEP data processing framework
- multi-threaded
- similar I/O
- like CMS, ATLAS, and DUNE
- allow for trying different I/O implementations
- some ability to shift between ROOT files/inputs
- easier to measure I/O and thread scaling
Mimicry:
- only processing events
- collection of data products
- source - mimics reading
- outputer - mimics writing
- waiter - mimics processing time
Processing:
- specify max. threads (TBBs)
- thread pool model, tasks handed to the pool when threads are available
- last step of task is often to make more tasks
- "lane" concept is used to specify how many events are concurrently processed
Lane:
- Source is a thing that gets us events
- each lane has an instance of a source
- source is given an event index
- lets framework know when it's all out of events
- source can wait until data is requested (vs. prefetching all of something)
Source Types:
- Empty makes no data products
- RootSource reads a ROOT file
- RepeatingRootSource reads first N entries, caches, cycles through them
- HDF55Source coming soon
- All sources report total time taken to read all the events
Waiters:
- parameter to control usleep(). Linearly related to Data Product size in bytes.
Outputter:
- all lanes share the same outputted
- calls are asynchronous
- outputter can know when data products are ready so it can split out serialization
- also knows when data products are finished if you need to write all data products for the same event at once
Outputter Types:
- dummy does nothing
- SerializeOutputer does ROOT serialization during the ready call and does nothing at finish
- RootOutputer coming soon, will write into ROOT
- HDF5Outputer also coming soon
Has done some evaluation to exercise this framework. Discussion of various results.
Here:
- https://github.com/Dr15Jones/root_serialization