Minutes of the May 22, 2013 Liquid Argon Simulations/LBNE reco meeting
Present:
Tom Junk, Alan Hahn, Michelle Stancari, Mike Kirby, Brian Rebel,
Eric Church, Herb Greenlee, Tyler Alion, Jonathan Insler, Jim Stewart,
Brett Viren, Maxim Potekhin, Stan Seibert
Apologies to those omitted.
J. Insler, T. Alion, all:
Work is proceeding to address the two issues facing LBNE running existing
reconstruction algorithms -- computer time and memory efficiency, and
hit ambiguities. The simulation is efficient enough to generate
enough events to get us practice with reconstruction. But there are
some inefficient steps in reconstruction that we have to make better.
Currently, the standard reconstruction steps are to unpack the data
(unzerosuppress it) and run the appropriate module in CalData
(e.g. CalWireLBNE10kt_module.cc) which deconvolutes the electronics
response, the field response, and applies the signal filter function,
for all wires in the detector, and produces recob::Wire objects.
Then a hit finder is called, with the recommendation being GausHitFinder.
Subsequent reconstruction finds clusters and tracks, but we need to
work out the kinks in this step.
A problem with running the standard steps is that ART calls ROOT
output streamer functions (though the output file does not grow in size)
which take at least 30 minutes of time writing out recob::Wire objects.
One can put recob::Wire on the output droplist and speed this up.
ROOT has some nonlinear inefficiencies writing many objects to a single
output file that are not in a TTree. As it is, the recob::Wire objects
are in a TTree, but all 300K+ of them are in the same branch of the same
event. The output streamer function in ROOT spends a lot of time
allocating and deallocating memory. A solution may be to use
TClonesArray objects, but ART objects do not inherit from TObject.
We may want to talk to the Artists about how to deal with this.
Nonetheless, we probably only rarely want to save all the recob::Wire
objects anyhow, as for the 10 kT detector, that's 2 Gbytes per event
(at least, not counting pre and post-spill data). Indeed, in order to
fit into our batch worker memory limit, we may not even want to store
recob::Wire objects in memory, even if we're not writing them out.
A job running CalData on 10 kT takes about 7 Gbytes of virtual memory
on an interactive node, and batch workers are limited to 4 Gbytes.
A solution to this is to make the hit finder into an algorithm that
can be called from a module that works on the event one wire at a time
and discards the intermediate deconvoluted digits, and even the
unpacked digits. recob::Hit has a pointer to the raw digits so we
don't need to keep intermediate results. If we're pressed later on
we can always recompute the deconvoluted data, though saving it
may be cumbersome.
Tyler has started going down this path with his AmbigHits made
by a module called APAHitFinder. This method currently carries
a copy of the GausHitFinder code that actually finds the hits.
We'd like to call the official version, but that would require
the official version not to have a loop over the recob::Wire objects
but to take one wire's worth of data at a time. To negotiate
with Jonathan Asaadi.
Tyler's APAHitFinder is set up to make a list of ambiguous hits and
call an algorithm to disambiguate them. He has a simple starting-point
algorithm that uses the arrival time of the hit to associate
collection and induction hits together. More sophisticated pattern
recognition can be included, though the structure is there.
Jonathan Insler will work on a version of CalWire that doesn't unpack
the zero-suppressed data, but instead deconvolutes the small chunks
of the nonzero data which are sparsely distributed throughout the TPC's.
This may save CPU and memory as deconvoluting long strings of zeros
(or almost zeros) is most of what takes the time in CalData.
The question arises of what repository to put this in. Currently
LArSoft has some experiment-specific code dealing with
geometry and electronics response. Refactoring the hit-finding so as
not to constantly allocate and deallocate memory and store lots
of intermediate results could have CPU and storage benefits for all
experiments, so this may be good for everyone.
Tom will talk with Rick Snider about setting up an LBNE-specific
repository. We'll need setup scripts, nightly builds, and a (set of)
responsible maintainer(s).
A redmine site, lbne-software
https://cdcvs.fnal.gov/redmine/projects/lbne-software
has been set up by Brett, as a grandchild of the LBNE redmine site
https://cdcvs.fnal.gov/redmine/projects/lbne
Brett has set up build scripts to build LArSoft which work on RACF
at BNL. It works on Scientific Linux, and Brett is working on
Ubuntu, gauged to be another popular distribution on LBNE.
The scripot is called beanie, and can be found in the lbne-software
repository under "build"
https://cdcvs.fnal.gov/redmine/projects/lbne-software/repository/show/build
ART wouldn't build in a 32 bit environment; it needed 64 bits.
There are minutes attached to this event.
Show them.