Storage Access Architecture (20 Oct 2015)
* attendance: Burt, Gene, Andrew, Stu, Margaret, OLI
* went through slides attached to indico agenda
* in short: store configuration files as blobs in either Postgres or Couchbase and serve it out through engineX http caches like conditions
* number of files:
* mu2e: ~4-5E6
* NOvA: ~1E3
* experiments are using SAM projects for accounting of their workflows
* interaction rates needed: 10k-100k concurrent accesses
* development effort
* reuses a lot of components
* no modifications to SAM
* minor modifications to ifdh -> Marc estimates one afternoon development and testing
* slight modifications to webserver frontend to access new backend
* new backend
* Postgres is straight forward -> Igor estimates 3 days for development and testing
* Couchbase takes longer -> no estimate
* talked to Mitch about Couchbase, they are positive, they envision that they have to do this (or similar nosql db technology?) anyway
* Couchbase was already used for Big Data studies, there is already experience
* what about data safety
* experiments stored their files on tape for long term storage
* database under backup would be equally acceptable, like for conditions
* there are examples that this works, this could be a generic solution for any kind of blob access from jobs
* tests
* Couchbase achieved 10E6 transactions per second
* Postgres: unknown the blob transaction rate
* user interface plan:
* users get ifdh mode to post configurations/blobs to DB
* migration
* straight forward using tools to ingest data to DB and SAM replication, could be lengthy for mu2e with many million files
* Architecture
* add new storage location: blob DB to read support files
* support files will be enabled for this storage location
* need decision whether to remove support files from non-tape-backed storage
* depends on how big effort is to create new DB for blobs, if small, remove
* action items
* Andrew: answer questions:
* Postgres blob size limitation and scaling estimates
* Overhead to create new blob DB
* Andrew and Margaret
* estimate effort to run this new service in production
* schedule SPPM outside the usual schedule this or next week to get sign off from Panagiotis
* OLI
* modify storage access architecture with new input
There are minutes attached to this event.
Show them.