Please read these instructions before posting any event on Fermilab Indico

Indico will be down for maintenance on Thursday, May 16th from 6:00PM - 8:00PM CST.

9 AM Meeting

US/Central
Huddle

Huddle

Description
Accelerator Scheduling Meeting

9 AM Meeting (21 Mar 2018)

Crew Chief Operations Summary
•Monday
–MSEPTA, MBLM10, MBLM20 needed reboots. MSEPTA code reverted.
–Booster high field fallout issued solved.
–D:KPS5A tripped off and needed expert repair.
•Tuesday
–MC1 open access started 02:20.
–Mtest finished at 03:00.
–Booster studies.
–Beam off at 06:00.
–MI52 mini-loop interlocks dropped during access. Re-secured.
–Enclosure C interlocks dropped during gate repair. Re-secured.
–Beam to Linac at 13:30, rad trips during emittance scans.
•Wednesday
–06:30 keys issued for 8GeV line ion pump cables.
–Booster studies pending.
–Rebooted  MBLM10 and MBLM20.

Linac
Marx upgrade LRF5.  LRF5 water leak patch.  Emittance scans.  NIF (NTF) timing investigation. 
 

 

Booster


Successful shutdown. Established beam to booster dump by 1600 yesterday. Thanks to all involved in work. Studies on evening shift. Turned off this morning for MI-8 access.  Started booster studies @ 0845.
 

 

Booster Neutrino


Reference to beam spot motion that could be related to booster high field fallout.  Tunnel work today. Possible beam this evening.
 

 

BNB Experiments


Work on DAQ.
 

 

Main Injector & Recycler


Going well. MI52 roughing overnight. Plan to have beam this evening.  Reboot for RR & MI LLRF systems will be necessary.  RAD suvey of tunnel completed. MI-8 ion pump cabling this morning.
 

 

NuMI


MW3 FE mentioned. Horn1 hanger valved out; leak appears to be gone (mostly). Seresco repare complete. Absorber Hall & Muon alcove will need secure.
 

 

NuMI Experiments


Took advantage of shutdown for various repairs. 
 

 

Muon Rings


Made access into AP0 vault;  vertical motor on target ground fault. (found wire in gears).  Did leak chaecking, found small leak on ion pump feed thru. (vac sealed)  Leak checking contiued. May ask operations to unlock supply(s).
 

 

Muon Experiments


Trolly run Monday evening. Used DT for maintence. Vacuum pump died, had spark started access early. May start a bit later this evening.  Would like to go to 16 pulse operation over the weekend. 
 

 

SY120


Gate between enc B & C fixed.  H201 water leak discovered.  (work in progress)
 

 

SY120 Experiments


Waiting for beam.
 

 

ES&H


Not to run BNB from 0600 to 1600.  Statement about access requirements. PPE and such.
 

 

EE Support


Linac    LRF2     Swap Marx cells; replace short studs                   Done
MI8      -        Swap-in ramped correctors (10)                         Done
MI8      -        MI8 access; fixed IP cables 835, 836, 838              Done
MI/RR
 MI520   -        Electrical support for kicker magnet removal           Done
 MI60S   -        Swap-in spare Hipot controller                         Done
 MI60N   I_PS60Q  Swap-in re-hosed rectifier modules                     Done
 MI60N   I_PS60L  Mitigate intermittent “in-ckt”; replace switch         Done
 MI60S   I_PS60U  Mitigate ISO amp offset (wye bridge)                   Done
 MI14    R_LAM10  Swap-in repaired Magnapower switching PS            Pending
 MI40    R_LAM40  Repair indexing of LOC/REM switch (Dynapower)       Pending
 MI10-60N         Service air filters, MIPS start panels              Pending
 MI14,30,52       Service air filters, Magna PSs                      Pending
Muon Campus
 F27     D_Q709   Investigate PS transductor grounding issues     In-progress
 F27     D_Q707   Swap-out “now tested” Phase Control Module             Done
 AP0     D_H812   Install rebuilt cap bank                               Done
 AP30    D_ICMAG  PS studies (Dynapower output filter)                Pending
 AP30    D_ELAM   Replace PS main contactor                              Done
 AP50    D_IB     Inspect substation for future switchgear upgrade       Done
P1/P2/P3
 F0      I_Q710   Swap-in upgraded gate drive cards                   Pending
 F0      I_HV703  Replace leaking rectifier module                In-progress
 F0      -        Replace control relay in (4) 500kW PSs                 Done
 F1      I_F17B3  Swap-in repaired Magnapower switching PS               Done
 F2      -        Isolate legacy PS switchgear                           Done
 F4      S_HP3DS  Restore oil level interlock                            Done
Other
 T-Hall  -        TEL1 cable mitigation                               Pending

 

Mechanical
MSD Machine Reports (Mechanical/Vacuum/Fluids):
Proton Source (Kubinski):

 

  • In Booster, Leak checked performed in Periods 24, 1 &2.
  • Repaired multiple leaks that they found on the kickers in period 2. Leaks were found and epoxy used to repair the leaks they could address. Ion pump 2.1 was reading 1.5 10-6e before the repair was made its. As of Wednesday at 0700 its reading 3.510 -6e.
  • Plan today: replacing the turbo pumps on the A LEBT
  • Fluids (Wallace):
  • Inspected and patch the leak on the RF 5 skid system the leak has slowed  the leak down to one drip Every 90 seconds.
  • Replacement for bad Debuncher LCW pump underway.  Installation underway.
  • Booster Anode East/West skids topped off with water/glycol make up.

 
MI/RR (Juarez):

  • Thanks for the report and the hard work today, Fernando. I turned on the RGA filament from home and started scanning. The vacuum on the ion gauge spiked but came back down within an hour. Attached is a scan. We’ll leave the RGA filament on and keep scanning while you make the RGA vacuum common with the MI vacuum. I’ll check on the RGA in the morning before 8:00, but after 8:00 you’ll have to contact Kyle if you need to do anything with the RGA controls.
  • Make sure that the ion pump power supply for IP520D was changed to a positive supply before turning it on (since it used to be a negative supply for the triode).
  • Replaced the SEPTA’s 300 l/s ion pumps with the 400 l/s ion pumps.
  • 520 region kicker magnets have been removed and replaced with the beam pipe.
  • the 200 AMU RGA manifold is installed, under vacuum and scanning at MI52.
  • Everything mentioned under vacuum overnight into this morning.
  • Assisted EE support and MI with the In 8 GEV ion pump repair.  3 of 4 ion pump cables or cable fittings successfully replaced.
  • Plan today: and today we will be able to proceed with the leakage check.
  • Fluids (De Leon):
  • MI Pond Pump bypass jumpers were implemented to facilitate pond strainer cleaning.  Upon completion, bypass jumpers were reset, ACNET status verified Pond Pump/LCW operation looks good.
  • DI bottles in MI-60 RF LCW system successfully replaced with 4 new DI bottles.

 
Beamlines (Vrbos):

  • Oil changed and filters added to pumping station 110 in enclosure C
  • Interlock gate between enclosure B and C was repaired 
  • Vacuum Leak check in muon campus D30 area took place. small leak found on an ion pump.  Leak repaired using vac seal as a short term fix.  Leak check underway to verify.
  • Fluids (Rivera/Clemons):
  • The leaking Dipole in Enclosure B has a bad ceramic. Repairs continuing through today.   
  • Water leak in meson MS2 system found to be an old, rusty, leaking gauge.  Old Gauge was removed and a plug put in place
  • F sector patches were inspected and looking very good. no new leaks found
  • A1 Pond Pump test will be performed today to verify pond pump configuration flow through to Wilson Hall east reflecting pond.

 
 
 

RF
RF 'A' & 'B' phase matched amps changed (120 Hz ripple found.) DI bottles replaced @ MI-60
 

 

Controls


Reference to splitting MW3 & MI60SB FE's from same crate. MSEPTA still an issue.  Work in progress.
 

 

Instrumentation


Full report to be given Friday.
 

 

FESS


MECH  

                                                 Casey's Pond-  WCI cut out old Gas main, control wires traced, need to be identified for reconnection impact                                                   Andy's Pond and Swan Lake; All mechanical seals  replaced.  Both back in full operation, Valve Orientation incorrect!                                                   MINOS- Chiller Glycol Leak- Make -up Tank being monitored for Leak Rate                                                      Metasys DX Controller for Detector Hall/Nova CHE Loop replaced, Alarms programmed!                                                   WH 11E- Plumbing Release for cast iron Y replacement, may be related to "Brown Water" dripping at urinal on 10E                                                    M05 MESON BERM -INSTALL BELTS ON EXHAUST FAN                                                         MI10-20 30 40 50 52 60 -- 12"SINGLE BASKET MANUAL STRAINER FOR POND WATER -ANN                                                   APT30-REPAIR PIN HOLE LEAK IN SUMP DISCHARGE PIPING                                                       MI65 MEZZ- CLEAN CHW COILS ON AHU-2 AND VERIFY PROPER ELECTRIC HEAT OPERATION (requires heater removal)                                                   TEV RING A1-B4  TEST BY-PASS PUMP ROUTE today                                                                                                LARTF - CHECK & REPAIR A FAILED SUMP PUMP - replace piping to failed check valve                                                     MS3/F38W1/F38W2EXHAUST FAN, BELT DRIVE-ANN                                                         WHL - IR - LEAKING HP-10 SEAL - CREATE A MECHANICAL WORK ORDER                            WHLF07W - REPAIR WATER FOUNTAIN SPEWING WATER WHEN IT RUNS.   

There are minutes attached to this event. Show them.
    • 09:00 09:01
      Crew Chief Operations Summary 1m
      Slides

      •Monday

      –MSEPTA, MBLM10, MBLM20 needed reboots. MSEPTA code reverted.

      –Booster high field fallout issued solved.

      –D:KPS5A tripped off and needed expert repair.

      •Tuesday

      –MC1 open access started 02:20.

      –Mtest finished at 03:00.

      –Booster studies.

      –Beam off at 06:00.

      –MI52 mini-loop interlocks dropped during access. Re-secured.

      –Enclosure C interlocks dropped during gate repair. Re-secured.

      –Beam to Linac at 13:30, rad trips during emittance scans.

      •Wednesday

      –06:30 keys issued for 8GeV line ion pump cables.

      –Booster studies pending.

      –Rebooted  MBLM10 and MBLM20.

    • 09:01 09:02
      Linac 1m

      Marx upgrade LRF5.  LRF5 water leak patch.  Emittance scans.  NIF (NTF) timing investigation. 

    • 09:02 09:03
      Booster 1m

      Successful shutdown. Established beam to booster dump by 1600 yesterday. Thanks to all involved in work. Studies on evening shift. Turned off this morning for MI-8 access.  Started booster studies @ 0845.

    • 09:03 09:04
      Booster Neutrino 1m

      Reference to beam spot motion that could be related to booster high field fallout.  Tunnel work today. Possible beam this evening.

    • 09:04 09:13
      BNB Experiments 9m

      Work on DAQ.

    • 09:13 09:18
      Main Injector & Recycler 5m

      Going well. MI52 roughing overnight. Plan to have beam this evening.  Reboot for RR & MI LLRF systems will be necessary.  RAD suvey of tunnel completed. MI-8 ion pump cabling this morning.

    • 09:18 09:19
      NuMI 1m

      MW3 FE mentioned. Horn1 hanger valved out; leak appears to be gone (mostly). Seresco repare complete. Absorber Hall & Muon alcove will need secure.

    • 09:19 09:20
      NuMI Experiments 1m

      Took advantage of shutdown for various repairs. 

    • 09:20 09:21
      Muon Rings 1m

      Made access into AP0 vault;  vertical motor on target ground fault. (found wire in gears).  Did leak chaecking, found small leak on ion pump feed thru. (vac sealed)  Leak checking contiued. May ask operations to unlock supply(s).

    • 09:21 09:22
      Muon Experiments 1m

      Trolly run Monday evening. Used DT for maintence. Vacuum pump died, had spark started access early. May start a bit later this evening.  Would like to go to 16 pulse operation over the weekend. 

    • 09:22 09:23
      SY120 1m

      Gate between enc B & C fixed.  H201 water leak discovered.  (work in progress)

    • 09:23 09:24
      SY120 Experiments 1m

      Waiting for beam.

    • 09:25 09:26
      ES&H 1m

      Not to run BNB from 0600 to 1600.  Statement about access requirements. PPE and such.

    • 09:27 09:28
      EE Support 1m

       

      Linac    LRF2     Swap Marx cells; replace short studs                   Done

      MI8      -        Swap-in ramped correctors (10)                         Done

      MI8      -        MI8 access; fixed IP cables 835, 836, 838              Done

       

      MI/RR

       MI520   -        Electrical support for kicker magnet removal           Done

       MI60S   -        Swap-in spare Hipot controller                         Done

       MI60N   I_PS60Q  Swap-in re-hosed rectifier modules                     Done

       MI60N   I_PS60L  Mitigate intermittent “in-ckt”; replace switch         Done

       MI60S   I_PS60U  Mitigate ISO amp offset (wye bridge)                   Done

       MI14    R_LAM10  Swap-in repaired Magnapower switching PS            Pending

       MI40    R_LAM40  Repair indexing of LOC/REM switch (Dynapower)       Pending

       MI10-60N         Service air filters, MIPS start panels              Pending

       MI14,30,52       Service air filters, Magna PSs                      Pending

       

      Muon Campus

       F27     D_Q709   Investigate PS transductor grounding issues     In-progress

       F27     D_Q707   Swap-out “now tested” Phase Control Module             Done

       AP0     D_H812   Install rebuilt cap bank                               Done

       AP30    D_ICMAG  PS studies (Dynapower output filter)                Pending

       AP30    D_ELAM   Replace PS main contactor                              Done

       AP50    D_IB     Inspect substation for future switchgear upgrade       Done

       

      P1/P2/P3

       F0      I_Q710   Swap-in upgraded gate drive cards                   Pending

       F0      I_HV703  Replace leaking rectifier module                In-progress

       F0      -        Replace control relay in (4) 500kW PSs                 Done

       F1      I_F17B3  Swap-in repaired Magnapower switching PS               Done

       F2      -        Isolate legacy PS switchgear                           Done

       F4      S_HP3DS  Restore oil level interlock                            Done

       

      Other

       T-Hall  -        TEL1 cable mitigation                               Pending

    • 09:28 09:29
      Mechanical 1m

      MSD Machine Reports (Mechanical/Vacuum/Fluids):

       

      Proton Source (Kubinski):

      • In Booster, Leak checked performed in Periods 24, 1 &2.
      • Repaired multiple leaks that they found on the kickers in period 2. Leaks were found and epoxy used to repair the leaks they could address. Ion pump 2.1 was reading 1.5 10-6e before the repair was made its. As of Wednesday at 0700 its reading 3.510 -6e.
      • Plan today: replacing the turbo pumps on the A LEBT
      • Fluids (Wallace):
      • Inspected and patch the leak on the RF 5 skid system the leak has slowed  the leak down to one drip Every 90 seconds.
      • Replacement for bad Debuncher LCW pump underway.  Installation underway.
      • Booster Anode East/West skids topped off with water/glycol make up.

       

      MI/RR (Juarez):

      • Thanks for the report and the hard work today, Fernando. I turned on the RGA filament from home and started scanning. The vacuum on the ion gauge spiked but came back down within an hour. Attached is a scan. We’ll leave the RGA filament on and keep scanning while you make the RGA vacuum common with the MI vacuum. I’ll check on the RGA in the morning before 8:00, but after 8:00 you’ll have to contact Kyle if you need to do anything with the RGA controls.
      • Make sure that the ion pump power supply for IP520D was changed to a positive supply before turning it on (since it used to be a negative supply for the triode).
      • Replaced the SEPTA’s 300 l/s ion pumps with the 400 l/s ion pumps.
      • 520 region kicker magnets have been removed and replaced with the beam pipe.
      • the 200 AMU RGA manifold is installed, under vacuum and scanning at MI52.
      • Everything mentioned under vacuum overnight into this morning.
      • Assisted EE support and MI with the In 8 GEV ion pump repair.  3 of 4 ion pump cables or cable fittings successfully replaced.
      • Plan today: and today we will be able to proceed with the leakage check.
      • Fluids (De Leon):
      • MI Pond Pump bypass jumpers were implemented to facilitate pond strainer cleaning.  Upon completion, bypass jumpers were reset, ACNET status verified Pond Pump/LCW operation looks good.
      • DI bottles in MI-60 RF LCW system successfully replaced with 4 new DI bottles.

       

      Beamlines (Vrbos):

      • Oil changed and filters added to pumping station 110 in enclosure C
      • Interlock gate between enclosure B and C was repaired 
      • Vacuum Leak check in muon campus D30 area took place. small leak found on an ion pump.  Leak repaired using vac seal as a short term fix.  Leak check underway to verify.
      • Fluids (Rivera/Clemons):
      • The leaking Dipole in Enclosure B has a bad ceramic. Repairs continuing through today.   
      • Water leak in meson MS2 system found to be an old, rusty, leaking gauge.  Old Gauge was removed and a plug put in place
      • F sector patches were inspected and looking very good. no new leaks found
      • A1 Pond Pump test will be performed today to verify pond pump configuration flow through to Wilson Hall east reflecting pond.

       

       

       

       

    • 09:29 09:30
      Fluids 1m
    • 09:30 09:31
      RF 1m

      RF 'A' & 'B' phase matched amps changed (120 Hz ripple found.) DI bottles replaced @ MI-60

    • 09:32 09:33
      Controls 1m

      Reference to splitting MW3 & MI60SB FE's from same crate. MSEPTA still an issue.  Work in progress.

    • 09:33 09:34
      Instrumentation 1m

      Full report to be given Friday.

    • 09:34 09:35
      FESS 1m

      MECH                                                   
      Casey's Pond-  WCI cut out old Gas main, control wires traced, need to be identified for reconnection impact                                                   
      Andy's Pond and Swan Lake; All mechanical seals  replaced.  Both back in full operation, Valve Orientation incorrect!                                                   
      MINOS- Chiller Glycol Leak- Make -up Tank being monitored for Leak Rate                                                     
       Metasys DX Controller for Detector Hall/Nova CHE Loop replaced, Alarms programmed!                                                   
      WH 11E- Plumbing Release for cast iron Y replacement, may be related to "Brown Water" dripping at urinal on 10E                                                    
      M05 MESON BERM -INSTALL BELTS ON EXHAUST FAN                                                         
      MI10-20 30 40 50 52 60 -- 12"SINGLE BASKET MANUAL STRAINER FOR POND WATER -ANN                                                   
      APT30-REPAIR PIN HOLE LEAK IN SUMP DISCHARGE PIPING                                                       
      MI65 MEZZ- CLEAN CHW COILS ON AHU-2 AND VERIFY PROPER ELECTRIC HEAT OPERATION (requires heater removal)                                                   
      TEV RING A1-B4  TEST BY-PASS PUMP ROUTE today                                                         
                                             
      LARTF - CHECK & REPAIR A FAILED SUMP PUMP - replace piping to failed check valve                                                     
      MS3/F38W1/F38W2EXHAUST FAN, BELT DRIVE-ANN                                                         
      WHL - IR - LEAKING HP-10 SEAL - CREATE A MECHANICAL WORK ORDER                            
      WHLF07W - REPAIR WATER FOUNTAIN SPEWING WATER WHEN IT RUNS.   

       

    • 09:36 09:37
      Miscellaneous 1m
    • 09:55 09:56
      MTA 1m