AD Weekly Friday - 2018 Scheduling Meeting/Weekly Summary

US/Central
Curia II (WH2SW)

Curia II

WH2SW

Description

Weekly Scheduling, Planning and Status 
Performance Plots
Meetings will be held on Monday, Wednesday, and Friday of each week.

AD Weekly Friday - 2018 Scheduling Meeting/Weekly Summary (23 Mar 2018)


Crew Chief
See slide.


Operations Coordinator
Running at design rate, timeline full.


Linac
See slides. Maintenance period on Tuesday ~ 6 hrs. EESupp upgraded Marx Modulator cells. Patch redone on LRF5 LCW. Emittance scans Tuesday evening. Beam to NIF on Weds for most of the morning. LRF4 filament hose this morning, fixed just before 9:00, right now running OK after recovery


Booster
See slides. Good week, eff. remains ~94%, Came up slightly lower after downtime but they are running more beam (muon), Ops have been doing a good job balancing intensities/losses on TLMs/demands, SY has been running, supplying ~5E11 as req., shutdown was good, did both tunnel and upstairs work, patched some leaks in L2 area (vacuum improvement), did studies (BRF phasing @ inj), tuning for better efficiency to keep up with intensities and rep rate, downtime 30 mins (BRF trips, BRF5 util yard breaker), good going into weekend.


Main Injector & Recycler
See slides. (pics of MI52 region) Went to full intensity NuMI within 4 hours of turnon. Major downtime was MI52 kicker replacement. Had small vacuum burst during startup. New ion pumps installed on the MI52 septa, also inspected LAM40 leak, did rad studies around ring (DALE & barcode), MI-8 IP cableing repair (got 3/4 back on), studies requested next week—1 hr dedicated time (no NuMI) for measuring sextupole centers and collimator studies, 4 hours at 1.4 rep rate.Dedicated study needs BNB operation, targeting for Tuesday. Will finalize at Monday's 9:00.


NuMI
Valved out flow on Horn 1 hanger, leak went away. Borescoped cross-drain and that was clean. Rebooted orbit verifier with new code.


NuMI Experiments
As reported on Weds, both MINERvA and NOvA used the downtime to perform work: MINERvA swapped out some boards; NOvA also did (in their FD), they topped off their detector LCW reservoir, and did some expert training, running well.


Booster Neutrino
Took advantage of downtime during day to get head start on summer shutdown jobs, slightly higher rad in MI-8 as expected now with muon running design intensity. (LOTO training today 10:30 - Chris J.)


BNB Experiments
Used downtime to improve external trigger rate (doubled), water leak in sump discharge fixed. Thanks goes to FESS & company.


Muon Rings
Very good performance on beam, some of best. During shutdown fixed vertical target motor. found vacuum leak and fixed. Yesterday g-2 requested full rep rate (11.4 hz), successful to this point longest ever, essentially doubled rate. A few growing pains (airborne Rad mons get right limits, some losses in beamline, AP0 dump, doing some tuning/studies getting ready for weekend. Plan to run this mode over the weekend.


Muon Experiments
Running smooth, some of the pulses are “bouncing around”—jitter? want to revisit kicker timing and check RF stability.


SY120, Neutrino Muon, Meson Test, Meson Center
Another water leak discovered @ H201-12 in Encl C, but were not able to fix it in place (only patched), keeping an eye on remnant LCW level and have a viable spare ready to replace during summer shutdown. Option 'B', could valve out and run without cooling if necessary if patch fails.


SY120 Experiments
Running smoothly. Pleased with quick startup from shutdown. Switching to new experiment next Wednesday.


FAST/IOTA
All ion pump cables have been terminated. Termination of the Dubna quads will start next week. BPM terminations are nearly complete at the electrical service building. IOTA LCW manifold testing continues. Power supplies for the RadiaBeam quads should arrive next week. Controls installation for the ring power supplies should begin next week. Installation of the RF cavity beamtube should be completed within 2 weeks. Assembly and leak checking of IOTA vacuum spools continues at CMTF.


PIP-II Injector Test
Running well, all RF components operating in CW mode, working on getting higher power beam to dump, doing machine protection and studies. Meson processed spoke cavity 112 (went well), next cavity will be “fully loaded” (high powered coupler), will see how HP processing goes.


Alignment
No report.


Controls
Nothing to report. Work in progress - cause/fix for FE reboots.


Cryo
No report.


ESH&Q
We have started a review of the Radiation Work Permit system. During a tour it was noted that several personnel did not have the correct PPE on for a tour or for work according to the RWP.
Remind personnel to complete the RWP, all fields Proper PPE, level 1 for tours and walkthroughs only, Level 2 for any work where you might be beamline components Describe the activity briefly in the sign off sheet. Do not put Fishing or Party Hard.


EE Support
Many details covered in systems reports. LRF2 tech swapped out Marx cells, LRF3 swapped broken Marx cell, swapped out DC correctors and modified in MI-8, also fixed some IP cables (835,836,838), supported kicker removal @ MI52 and did MIPS replacement mostly at MI-60, misc PMs on various ps.
Muon: repairs, PMAG (soft ground fault, replaced SCR), made mod on Lumina PS (on lens supp), AP50 subst., in P1 P2 P3 line made some repairs, on Muon campus replaced a control relay in PS, also lens PMAG pulsers control card needed to be reversed to earlier version.
MT5EL in SY needed PLC replacement. Want to do some LOTO training during BNB downtime. (today:10:30)


Instrumentation
Calibrated toroids at MI-40. BPM work attemped but need to modify tool for cable disconnect.


Mechanical & Fluids
MSD Machine Reports (Mechanical/Vacuum/Fluids):
Proton Source (Kubinski):
Wednesday - two 361 turbo pumps were installed on Source “A” LEBT. Test station klystrons were swapped. Fluids (Wallace):
On Wednesday, a new LCW pump on the LINAC Debuncher Skid was successfully installed and is in standby mode ready for operation. On Friday morning, a major water leak was reported on LRF4 RF skid. LCW supply hose on the LRF4 power amplifier filament failed. A patch and clamp was applied which slowed the leak down to a drip about every 60 seconds. The hope is that it holds until our recently ordered refurbished replacement hoses arrive, otherwise all our other systems look good.

MI/RR (Juarez):
Wednesday - Upon completion of Septa Ion pump installation, kicker removal, beam pipe installation, and RGA assembly installations, leak tests in the 520 area were performed a reading of 2.0 E-10. No leaks were indicated. IP’s were turned on and equipment removed from the tunnel. Thursday and currently - Vacuum bursts along MI and F0 portion of the P1 line. Vacuum recovering acceptable. Fluids (De Leon):
MI pond pumps switched to high speed (60 hp mode) summer mode A1 Pond Pump tested successfully. Bypass pump to reflecting pond. Main pond pump is off under configuration control. Leak on MI-40 Lambertson (section 624) inspected. Short term solution: Water routed away from the Lambertson using plastic. Long term solution is to pull Lambertson out in aisle and implement elbow assembly repair.

Beamlines (Vrbos):
Wednesday –
Performed vacuum leak checks in Muon Campus, D30, sector. Enclosure gate repaired between enclosure B and C. Blower station maintenance performed in G1 stub. Thursday and currently - Vacuum bursts along MI and F0 portion of the P1 line. Vacuum recovering acceptable. Did not get to inspect/tour F2 Sector for shutdown preparations
Fluids (Rivera/Clemons):
Performed water leak inspections/searches along external beamlines and switchyard. All installed magnet leak patches appear to be holding. THE DIPOLE 201-12 at the Enclosure C leak was sizable. Temporary patch applied, holding for now. Will be monitoring the Remnant LCW System

Fluids (Non-Machine Specific) (Rios/J. Todd):
FAST – Continued with IOTA project. Successfully performed low pressure gas test, held at 15 psig. Filled with LCW and started hydrostatic test. System held pressure of 150 psig for 1/2 hour. Found leak at girder 9 union, and relief valve I installed began to weep prematurely. Repairs made. Second hydrotest to be performed today, Friday. - Sehnal CUB – No LCW make-up needed today – John Todd


RF
Very successful shutdown
Booster Shutdown:
Preventative PA replacements on BRF4, 7, & 9 SS upgrade conducted on modulator at BRF6 APS glycol skids filled and measured for glycol Replaced LLRF distribution amplifier for both A & B Conducted LLRF calibration on fan out system
Main injector Shutdown
General PM Replaced MIRF02 Modulator with a SS upgraded modulator Valved in new DI bottles for the RF system, resistivity looks good
Other
Investigating cause of BRF05 bias supply yard breaker trip from Tuesday night. Call Victor with 20 minutes of opportunistic during the day shift.


FESS
MECHANICALM05 MESON BERM -INSTALLED BELTS ON EXHAUST FAN MI10-20 30 40 50 52 60 – Cleaned 12" SINGLE BASKET MANUAL STRAINER FOR POND WATER -ANN APT30-REPAIRED PIN HOLE LEAK IN SUMP DISCHARGE, REPLACED SECTION OF PIPING MI65 MEZZ- INVESTIGATED INTERMITTENT OPERATION OF AHU-2, RESOLVED DIFFERENTIAL PRESSURE SENSOR PROBLEM TEV RING A1 BYPASS PUMP, TEST RAN AND CONFIRMED VALVE CONFIGURATION FOR A-1 HX BYPASS LARTF – REPAIRED LEAK, REPIPED SUMP DISCAHRGE PIPING MS3 PERFORMED PM ON EXHAUST FANS, BELT DRIVE-ANN
ELECTRICAL MI12A – EXHAUST FAN - DISCONNECT & RECONNECT 110V POWER FOR REPLACEMENT EXHAUST FAN RSRF – INVESTIGATED AND UNWIRED DE-COMISSIONED 75 HP POND PUMP MCC BUCKETS FOR POWER / CONNECTED LOAD RSRF – INVESTIGATED AND RESOLVED PHASE IMBALANCE ON 5HP AIR COMPRESSOR #1, REPAIRED STARTER APTRT- EXIT AND EMERGENCY LIGHT REPAIRS NUTMINOS ABSORBER HALL-INVESTIGATED BEARING NOISE ON TUNNEL SUPPLY AIR FAN SF-M-3 Alcove #4 MT6 - PM- PANELBOARD PP-ML-10A. BREAKER # 13 Tripped, will not reset, check circuit loading APSB50 - ISOLATED TRANSFORMERS FOR EE SUPORT INSPECTION
MONDAY MARCH 26th CASEYS POND 1500 KVA TRANSFORMER WILL BE LIVENED BUT NO LOAD Thursday March 29th, CASEY’S POND LOAD TRANSFER, 400 HP PUMPS WILL BE RUN IN HAND
HVAC K-TEV, M02, NMS PERORMED ANNUAL PM ON HVAC UNITS


Miscellaneous
Scope change on BNB work yesterday, about a day behind due to design vs. as-built difference. Had planned to pour concrete today. Will be pouring Monday, Tuesday, Wednesday rain - may cause delay. Concrete forms will need extra day to removef forms. Can run BNB over weekend. Will be shutting off Monday morning at 6-6:30. Cons won’t be there next week (but will be reading email).

There are minutes attached to this event. Show them.
    • 09:00 09:03
      Crew Chief 3m
      Slides

      See slide.

    • 09:03 09:06
      Operations Coordinator 3m
      Plots

      Running at design rate, timeline full.

    • 09:06 09:09
      Linac 3m
      Slides

      See slides. Maintenance period on Tuesday ~ 6 hrs. EESupp upgraded Marx Modulator cells. Patch redone on LRF5 LCW. Emittance scans Tuesday evening.  Beam to NIF on Weds for most of the morning.   LRF4 filament hose this morning, fixed just before 9:00, right now running OK after recovery

    • 09:09 09:12
      Booster 3m
      Slides

      See slides. Good week, eff. remains ~94%, Came up slightly lower after downtime but they are running more beam (muon), Ops have been doing a good job balancing intensities/losses on TLMs/demands, SY has been running, supplying ~5E11 as req., shutdown was good, did both tunnel and upstairs work, patched some leaks in L2 area (vacuum improvement), did studies (BRF phasing @ inj), tuning for better efficiency to keep up with intensities and rep rate, downtime 30 mins (BRF trips, BRF5 util yard breaker), good going into weekend.

    • 09:12 09:15
      Main Injector & Recycler 3m
      Slides

      See slides. (pics of MI52 region)  Went to full intensity NuMI within 4 hours of turnon. Major downtime was MI52 kicker replacement.  Had small vacuum burst during startup. New ion pumps installed on the MI52 septa, also inspected LAM40 leak, did rad studies around ring (DALE & barcode), MI-8 IP cableing repair (got 3/4 back on), studies requested next week—1 hr dedicated time (no NuMI) for measuring sextupole centers and collimator studies, 4 hours at 1.4 rep rate.Dedicated study needs BNB operation, targeting for Tuesday.  Will finalize at Monday's 9:00.

    • 09:15 09:18
      NuMI 3m

      Valved out flow on Horn 1 hanger, leak went away.  Borescoped cross-drain and that was clean.  Rebooted orbit verifier with new code.

    • 09:18 09:21
      NuMI Experiments 3m

      As reported on Weds, both MINERvA and NOvA used the downtime to perform work: MINERvA swapped out some boards; NOvA also did (in their FD), they topped off their detector LCW reservoir, and did some expert training, running well.

    • 09:21 09:23
      Booster Neutrino 2m

      Took advantage of downtime during day to get head start on summer shutdown jobs, slightly higher rad in MI-8 as expected now with muon running design intensity. (LOTO training today 10:30 - Chris J.)

    • 09:23 09:24
      BNB Experiments 1m

      Used downtime to improve external trigger rate (doubled), water leak in sump discharge fixed. Thanks goes to FESS & company.

    • 09:24 09:26
      Muon Rings 2m

      Very good performance on beam, some of best. During shutdown fixed vertical target motor. found vacuum leak and fixed.  Yesterday g-2 requested full rep rate (11.4 hz), successful to this point longest ever, essentially doubled rate. A few growing pains (airborne Rad mons get right limits, some losses in beamline, AP0 dump, doing some tuning/studies getting ready for weekend.  Plan to run this mode over the weekend.

    • 09:27 09:28
      Muon Experiments 1m
      Speakers: Brendan Casey (FNAL), Brendan Kiburg (Fermilab), Joe Grange (Argonne National Lab)

      Running smooth, some of the pulses are “bouncing around”—jitter? want to revisit kicker timing and check RF stability. 

    • 09:30 09:32
      SY120, Neutrino Muon, Meson Test, Meson Center 2m

      Another water leak discovered @ H201-12 in Encl C, but were not able to fix it in place (only patched), keeping an eye on remnant LCW level and have a viable spare ready to replace during summer shutdown. Option 'B',  could valve out and run without cooling if necessary if patch fails.

    • 09:32 09:35
      SY120 Experiments 3m

      Running smoothly.  Pleased with quick startup from shutdown.  Switching to new experiment next Wednesday.

    • 09:35 09:38
      FAST/IOTA 3m
      • All ion pump cables have been terminated.  Termination of the Dubna quads will start next week.
      • BPM terminations are nearly complete at the electrical service building.
      • IOTA LCW manifold testing continues.
      • Power supplies for the RadiaBeam quads should arrive next week.
      • Controls installation for the ring power supplies should begin next week.
      • Installation of the RF cavity beamtube should be completed within 2 weeks.
      • Assembly and leak checking of IOTA vacuum spools continues at CMTF.
    • 09:38 09:40
      PIP-II Injector Test 2m

      Running well, all RF components operating in CW mode, working on getting higher power beam to dump, doing machine protection and studies.  Meson processed spoke cavity 112 (went well), next cavity will be “fully loaded” (high powered coupler), will see how HP processing goes.

    • 09:40 09:42
      Alignment 2m

      No report.

    • 09:42 09:45
      Controls 3m

      Nothing to report.  Work in progress - cause/fix for FE reboots.

    • 09:45 09:47
      Cryo 2m

      No report.

    • 09:47 09:50
      ESH&Q 3m

      We have started a review of the Radiation Work Permit system.  During a tour it was noted that several personnel did not have the correct PPE on for a tour or for work according to the RWP. 

      • Remind personnel to complete the RWP, all fields
      • Proper PPE, level 1 for tours and walkthroughs only, Level 2 for any work where you might be beamline components
      • Describe the activity briefly in the sign off sheet. Do not put Fishing or Party Hard.

       

    • 09:50 09:52
      EE Support 2m

      Many details covered in systems reports. LRF2 tech swapped out Marx cells, LRF3 swapped broken Marx cell, swapped out DC correctors and modified in MI-8, also fixed some IP cables (835,836,838), supported kicker removal @ MI52 and did MIPS replacement mostly at MI-60, misc PMs on various ps.

      Muon: repairs, PMAG (soft ground fault, replaced SCR), made mod on Lumina PS (on lens supp), AP50 subst., in P1 P2 P3 line made some repairs, on Muon campus replaced a control relay in PS, also lens PMAG pulsers control card needed to be reversed to earlier version.

      MT5EL in SY needed PLC replacement. Want to do some LOTO training during BNB downtime. (today:10:30)

    • 09:52 09:54
      Instrumentation 2m

      Calibrated toroids at MI-40. BPM work attemped but need to modify tool for cable disconnect.

    • 09:54 09:55
      Mechanical & Fluids 1m

      MSD Machine Reports (Mechanical/Vacuum/Fluids):

      Proton Source (Kubinski):

      • Wednesday - two 361 turbo pumps were installed on Source “A” LEBT.  Test station klystrons were swapped. 
      • Fluids (Wallace):
      • On Wednesday, a new LCW pump on the LINAC Debuncher Skid was successfully installed and is in standby mode ready for operation.
      • On Friday morning, a major water leak was reported on LRF4 RF skid. LCW supply hose on the LRF4 power amplifier filament failed.  A patch and clamp was applied which slowed the leak down to a drip about every 60 seconds.  The hope is that it holds until our recently ordered refurbished replacement hoses arrive, otherwise all our other systems look good.

       

      MI/RR (Juarez):

      • Wednesday - Upon completion of Septa Ion pump installation, kicker removal, beam pipe installation, and RGA assembly installations, leak tests in the 520 area were performed a reading of 2.0 E-10.  No leaks were indicated. IP’s were turned on and equipment removed from the tunnel.
      • Thursday and currently - Vacuum bursts along MI and F0 portion of the P1 line.  Vacuum recovering acceptable.
      • Fluids (De Leon):
      • MI pond pumps switched to high speed (60 hp mode) summer mode
      • A1 Pond Pump tested successfully.  Bypass pump to reflecting pond.  Main pond pump is off under configuration control.
      • Leak on MI-40 Lambertson (section 624) inspected.  Short term solution: Water routed away from the Lambertson using plastic.  Long term solution is to pull Lambertson out in aisle and implement elbow assembly repair.

       

      Beamlines (Vrbos):

      • Wednesday –
      • Performed vacuum leak checks in Muon Campus, D30, sector.
      • Enclosure gate repaired between enclosure B and C.
      • Blower station maintenance performed in G1 stub.
      • Thursday and currently - Vacuum bursts along MI and F0 portion of the P1 line.  Vacuum recovering acceptable.
      • Did not get to inspect/tour F2 Sector for shutdown preparations
      • Fluids (Rivera/Clemons):
      • Performed water leak inspections/searches along external beamlines and switchyard.  All installed magnet leak patches appear to be holding.
      • THE DIPOLE 201-12 at the Enclosure C leak was sizable.  Temporary patch applied, holding for now.  Will be monitoring the Remnant LCW System

       

      Fluids (Non-Machine Specific) (Rios/J. Todd):

      • FAST – Continued with IOTA project.  Successfully performed low pressure gas test, held at 15 psig.  Filled with LCW and started hydrostatic test.  System held pressure of 150 psig for 1/2 hour.  Found leak at girder 9 union, and relief valve I installed began to weep prematurely.  Repairs made.  Second hydrotest to be performed today, Friday. - Sehnal
      • CUB – No LCW make-up needed today – John Todd
    • 09:55 09:56
      RF 1m

       

      Very successful shutdown

      Booster Shutdown:

      • Preventative PA replacements on BRF4, 7, & 9
      • SS upgrade conducted on modulator at BRF6
      • APS glycol skids filled and measured for glycol
      • Replaced LLRF distribution amplifier for both A & B
      • Conducted LLRF calibration on fan out system

       

      Main injector Shutdown

      • General PM
      • Replaced MIRF02 Modulator with a SS upgraded modulator
      • Valved in new DI bottles for the RF system, resistivity looks good

      Other

      Investigating cause of BRF05 bias supply yard breaker trip from Tuesday night. Call Victor with 20 minutes of opportunistic during the day shift.

    • 09:56 09:58
      FESS 2m
      Speakers: Gilbert, Shirley, Sprosty

      MECHANICAL
      M05 MESON BERM -INSTALLED BELTS ON EXHAUST FAN                                                
      MI10-20 30 40 50 52 60 – Cleaned 12" SINGLE BASKET MANUAL STRAINER FOR POND WATER -ANN  
      APT30-REPAIRED PIN HOLE LEAK IN SUMP DISCHARGE, REPLACED SECTION OF PIPING                                                                                                                            
      MI65 MEZZ- INVESTIGATED INTERMITTENT OPERATION OF AHU-2, RESOLVED DIFFERENTIAL PRESSURE SENSOR PROBLEM                                                                                                              
      TEV RING A1 BYPASS PUMP, TEST RAN AND CONFIRMED VALVE CONFIGURATION FOR A-1 HX BYPASS                                                                                                                            
      LARTF – REPAIRED LEAK, REPIPED SUMP DISCAHRGE PIPING                                                          
      MS3 PERFORMED PM ON EXHAUST FANS, BELT DRIVE-ANN                                                        

      ELECTRICAL
                                                                                                         
      MI12A – EXHAUST FAN - DISCONNECT & RECONNECT 110V POWER FOR REPLACEMENT EXHAUST FAN                                                                                                                                                          
      RSRF – INVESTIGATED AND UNWIRED DE-COMISSIONED 75 HP POND PUMP MCC BUCKETS FOR POWER / CONNECTED LOAD                                                                                                                  
      RSRF – INVESTIGATED AND RESOLVED PHASE IMBALANCE ON 5HP AIR COMPRESSOR #1, REPAIRED STARTER                                                                                                                            
      APTRT- EXIT AND EMERGENCY LIGHT REPAIRS                                                                                                                
      NUTMINOS ABSORBER HALL-INVESTIGATED BEARING NOISE ON TUNNEL SUPPLY AIR FAN  SF-M-3 Alcove #4                                                                                                                             
      MT6 - PM- PANELBOARD PP-ML-10A. BREAKER # 13 Tripped, will not reset, check circuit loading          
      APSB50 - ISOLATED TRANSFORMERS FOR EE SUPORT INSPECTION

      MONDAY MARCH 26th CASEYS POND 1500 KVA TRANSFORMER WILL BE LIVENED BUT NO LOAD    Thursday March 29th, CASEY’S POND LOAD TRANSFER, 400 HP PUMPS WILL BE RUN IN HAND

      HVAC        
      K-TEV, M02, NMS PERORMED ANNUAL PM ON HVAC UNITS                                                    
              

       

    • 09:58 09:59
      Summer Shutdown 2018 1m
      Speaker: Mr Cons Gattuso (Fermi Lab)
    • 09:59 10:00
      Miscellaneous 1m

      Scope change on BNB work yesterday, about a day behind due to design vs. as-built difference. Had planned to pour concrete today. Will be pouring Monday, Tuesday, Wednesday rain - may cause delay.  Concrete forms will need extra day to removef forms. Can run BNB over weekend.  Will be shutting off Monday morning at 6-6:30. Cons won’t be there next week (but will be reading email).