Weekly CCE-IOS tele-conference (25 Mar 2020)
Chaired by: Ross, Rob; Dr. van Gemmeren, Peter
Attend: Salman, Paolo, RobR, Peter, RobL, Saba, Shane, Suren, Mathieu Liz, Philippe
Management news:
Salman: successfully negotiated for a little additional funding. Numbers are available. Priority are PPS and IOS.
Quarterly reports need to be written, followup by email
Received IOS effort request and will get back at us.
Intro:
Discussion:
Rob Latham: HPC I/O
ROOT is HEP High-Level I/O Library
Liz: Discussion of what I/O middleware is in HEP. Coordination of writing and threads within the root output module. Is that the place where the middleware concepts are applied.
Philippe: TMPIFile is new ROOT ~I/O forwarding layer.
Liz: Needs to be some effort on the experiment side to use what exists maximally, should inform the program of work.
Peter: Consider aligning (compressed) baskets (branches) in a cluster to better align/respect block bounderies.
Discussion of compression, how to coordinate, etc. Not immediately obvious how to approach, whether to do this inside ROOT, above ROOT, etc.
Compressing to near block boundaries, aggregation/sorting? Not clear what the right algorithmic approach is. Other costs to consider.
Two observations:
- Usually the higher-level library hands things off to a layer that does shuffling, if we're going to do that, across nodes.
- Also useful to put the bundles of data in a separate file from the metadata/index.