Please read these instructions before posting any event on Fermilab Indico

Indico will be unavailable on Wed., Dec 18th from 7 - 7:30am due to server maintenance.

XCache DevOps Meeting Aug 20, 2020

US/Central
Derek Weitzel (University of Nebraska - Lincoln), Marian Zvada (US CMS)
Description

Time: 11AM PT/1PM CT/2PM ET/8PM CET
Where: ZOOM.US

Join from PC, Mac, Linux, iOS or Android: https://unl.zoom.us/j/651969661
Meeting is password-protected: ask #xcache slack channel or xcache@opensciencegrid.org

Or iPhone one-tap :
    US: +16699006833,,651969661#  or +14086380968,,651969661#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 669 900 6833  or +1 408 638 0968  or +1 646 876 9923
    Meeting ID: 651 969 661
    International numbers available: https://unl.zoom.us/zoomconference?m=wxCNSgMZiA-cVKSNowGYlQ

	• Presentation from Georg Schuhardt (15m) regarding XCache and direct filesystem access: 
		• Many questions about the read performance of direct access in 5.0.  Possible bug, Andy and Georg will investigate.
		• Possibly a bug in the ACL’s of direct access.
		• Slides: https://drive.google.com/file/d/1kCt0IqMFY45XDp6aJsfzC4nnj0kVyGsI/view?usp=sharing

	• Development Updates
		• 5.0.1 is released, and built in OSG
		• 4.12.4-rc1 was just released
			• Staggering the release with the 5.0.1
			• May put more patches into the 4.12.4 before release.
		• OSG will product compat libraries in order to allow 5.0 be uploaded to EPEL
		• 5.1 manual now live in on xrootd website: https://xrootd.slac.stanford.edu/doc/dev51/xrd_config.htm
		• New monitoring and gstream configuration formats
		• 5.1 each server will have a fingerprint in the gstream.
			• Will be documented in the monitoring data.
	• ATLAS Updates
		• Status of caching g-stream changes
	• CMS Updates
		• UCSD cooling issues.
		• High availability setup with redirector seems to work for CMSSW jobs, but not for xrdcp.  Always picks the “first” one.
		• But xrdcp should move on to the next one.
		• Discussion of whether it was DNS issues, or something else.
		• CMS XCaches in the US: 3
		• CMS XCaches in containers: 0
	• StashCache (OSG) Updates
		• some discussion of stashcp’s behavior.  How many caches to try before failing? (2? 3? all? 4?)
		• (John Hicks) 
			• Houston node back on line, will be part of the PRP nautilus cluster
			• Kansas is back up as well, but still something wrong.
			• Amsterdam is made through customs.  Racked, but still being setup.
		• (Brian) Hot fix detection and image build in today
		• StashCache on EL8 works fine.
	• AOB
		• Discussion of how to pick the nearest server
			• Can always perform a xrootd ping (in the xrootd protocol)
			• Currently use GeoIP in stashcache world
	• Next week:
		• Edgar’s talk about the CMS datalake
		• GSOC presentation from riccardo’s student.
		• Move meeting ahead 30 minutes next week to accommodate Eurpoean time zones.
There are minutes attached to this event. Show them.
    • News
      Conveners: Dr Derek Weitzel (University of Nebraska - Lincoln), Mr Marian Zvada (US CMS)
    • XRootD development for XCache

      General status of development, XRootD releases with new cache features, bug fixes, etc...

      Conveners: Mr Andrew Hanushevsky (SLAC National Accelerator Laboratory), Matevz Tadel (UCSD), Wei Yang (SLAC)
    • XCache issues

      XCache issues experienced during the deployment, configuration or otherwise causing problems in production or development.

    • ATLAS XCache deployment
      Conveners: Dr Ilija Vukotic (LAL), Wei Yang (SLAC)
    • CMS XCache deployment
      Conveners: Diego Ciangottini (Universita e INFN, Perugia (IT)), Mr Edgar Fajardo Hernandez (UCSD), Prof. Frank Wuerthwein (UCSD)
    • OSG StashCache deployment

      Status of OSG deployment, origin/caches operations, stashcp, usage, etc...

      Conveners: Brian Bockelman, Brian Lin (University of Wisconsin-Madison), Dr Derek Weitzel (University of Nebraska - Lincoln), Mr Edgar Fajardo Hernandez (UCSD), John Hicks (Internet2), Mr John Thiltges (University of Nebraska - Lincoln), Mr Marian Zvada (US CMS), Mr Matyas Selmeci (University of Wisconsin-Madison)
    • XRootD transfers monitoring
      Conveners: Dr Derek Weitzel (University of Nebraska - Lincoln), Diego Davila (CMS), Mr Edgar Fajardo Hernandez (UCSD)
    • AOB