EPICS Collaboration Meeting at the Fermi National Accelerator Laboratory.
I will present an overview and demonstration of the areaDetector framework. areaDetector is widely used for interfacing EPICS to a variety of 2-D detectors, including those for visible light imaging, spectroscopy, and X-rays. It has a rich set of plugins for image statistics, transformation, file saving, etc. I will present a demonstration using a simulation detector and a real camera.
Perspective gained in the process of learning about EPICS systems and architectures as part of Electron Ion Collider design work will be shared from the point of view of a developer that is new to the ecosystem. Highlights related to positive and negative experiences will be addressed.
A brief overview of the ACORN Project and current status.
In recent years, the construction of European Spallation Source, including the control system, has progressed very rapidly and we are operating a substantial number of IOCs in production. Our goal from early on has been to apply comprehensively the new emerging features of EPICS; accordingly the talk will cover the status of our use of EPICS 7. Our underlying system architecture, in particular our use of MTCA and our EPICS build and deployment systems will be presented. More details about our software layer will be covered in another talk.
Apart from moving from Debian to RHEL as a baseline operating system, the NSLS-II facility has embraced a plethora of new powerful automation and orchestration tools. Changes to how we manage our systems have brought along new considerations and approaches in the areas of EPICS software packaging, distribution, IOC code management, and deployment. The talk covers updates in these areas, and outlines further opportunities to greater standardizing and automating. Specifically outlined are:
1. EPICS base, modules, and tools RPM creation, distribution, and licensing.
2. IOC code management via a monorepo, and specialized tools created to support the workflow.
3. Deployment of select EPICS applications.
The Electron Ion Collider Project will involve the deployment of a large number of Front End Computers of a novel SoC design that is intended as a Common Platform across many sub-systems will a spectrum of performance requirements. We'll describe the previous experience building a custom carrier + daughter card platform, the current platform generation designs and statuses for EIC, as well as future development plans that will be needed to operate a large accelerator complex that is currently being developed.
ALS has a long history behind. The ALS-U upgrade project is on the way in parallel with the ALS routine operation. The intention of this presentation is to show the recent changes to the ALS control system which are aligned with the ALS-U upgrade project.
NASA's Neil A. Armstrong Test Facility (ATF, formerly Plum Brook Station) is a 6,400 acre (26 km^2) research campus in Sandusky, Ohio that operates a number of unique world-class test facilities which are used by NASA, other governmental agencies, and the private sector to conduct full service facility and test system preparation for complex and innovative research and flight programs. The test facilities at ATF are used to simulate the conditions of the upper atmosphere, deep space, planetary, lunar environments, and cryogenics. This talk will present a high-level overview of the ATF test facilities and test capabilities and highlight their individual data and control (DAC) system needs. The second part of the talk will describe a recent project at ATF to identify a common client-server DAC SW architecture that will result in safer and more reliable operations across the entire ATF DAC system portfolio and will lower total cost of ownership to ATF (and therefore) our customers.
Most APS beamlines are using VME crates as a major part of their control system.
The VME hardware is expensive and becoming obsolete, with replacements for many of the above items no longer available. The VxWorks software is also expensive.
I am leading an effort at sector 13 to completely eliminate the 7 VME systems during the dark year.
I plan to install the following to replace the VME systems:
• Galil DMC-4183 motor controllers to replace the OMS-58 and MAXv.
• Moxa terminal servers to replace the VME serial communication modules.
• Measurement Computing USB-CTR08 to replace the Joerger and SIS 3820 scalers and multi-channel scaler functions.
• Measurement Computing USB-3104 to replace the DAC128V for analog output.
• Measurement Computing USB-1808X to replace the IP-330.
The total hardware cost to replace 7 VME systems for 2 FOEs and 5 experimentation stations is less than $150K. This includes more than 370 motor channels.
Detectors based on TimePix3 chips are complementary to existing image-based detectors. In addition to the 2D image, the TimePix3 chip allows the collection of time information of each hit down to the nanosecond level. The timing capabilities allow new possibilities in particle and photon data collection, by providing not only characterization of accelerator source but also science related to timed dynamics. The timing capabilities of the TimePix3 detector allow neutron measurements such as energy using Time of Flight mode. EPICS driver for the TimePix3 detector is a major step in the development of the science within modern neutron and synchrotron facilities.
I will present the RFSoC based 500 MHz BxB Beam Position Monitor development status and epics control system interface.
The RFSoC-based advanced BxB provides many capabilities for measuring BxB beam transients, multi-bunch coupling, bunch charging, and beam trip transients.
Also, Xilinx RFSoC FPGA supports a hard ARM CPU core for interface with EPICS control systems.
I will introduce BxB hardware architecture, FPGA data processing, extensive data capturing, and epics interface for a control system.
In order to increase data acquisition rates and to lower complexity at the embedded level, Fermilab Instrumentation is developing a Ethernet field bus to communicate with a centralized middle layer utilizing Redis. Each embedded node will stream data to the middle layer where post processing can be done via micro services in containers. All data, including raw and processed wave forms, will then be presented to the control system. This architecture allows for easy scalability, lowers complexity, and allows easy adaptation to current (ACNET) and future controls systems (EPICS).
ESS moved to CS-Studio Phoebus in 2019 and we have since then worked with the framework together with the community.
We have put in a lot of effort with the Logbook (Olog) and Save & Restore (based on MASAR).
IPv6 has been around since 1995, but EPICS has so far only supported IPv4. This will be a high level introduction to IPv6, what basic changes are necessary to existing IPv4 code, and what has been done for PV Access.
ACORN Human Factors: This presentation details an introduction to human factors and how Fermilab is enabling user-centered designs & processes for the ACORN project. Presentation topics include how a combination of design standardization and customization will enable more intuitive operations and reduce cognitive overwhelm & burnout.
A high-performance data acquisition system (DAQ) has been under active development to meet APS-U needs. It takes data from underneath FPGA (Field Programmable Gate Array), and streams it to its downstream users. The APS-U DAQ system software framework is implemented as a major portion of APS-U new control system software infrastructure, which is called C2. To visualize the DAQ data on the fly, a C2 Data Viewer (C2DV) is has been implemented using Python, which can be used for displaying live PV data streams for monitoring, troubleshooting and diagnostics purposes. It is now capable of handling both EPICS pvAccess (PVA) and Channel Access (CA) data, and includes several different applications: a scope viewer for plotting PVA waveforms, an image viewer for displaying Area Detector image data, and a striptool for monitoring PVA as well as CA scalar PVs. In this presentation we discuss various C2DV features, its usage at the Advanced Photon Source, as well as plans for future development.
This talk contextualizes EPICS in accelerator optimization and "quite
big" data experiment support, for other AI/ML session talks. EPICS 7
had the aim of better computing supporting for the next generation of
machines and analytical tools. Many new data types, such as matrices,
tables, images, and statistical descriptions, plus users' own data
types, now supplement the simple scalar and waveform types of the
classic EPICS. New EPICS based computational architectures for
scientific computing are emerging for high-performance data
processing, high fidelity archiving, pipelining, and classically
offline physics. Python and matlab bindings support ML and science
users. The result has been that controls are now being integrated with
multi-particle modelling, machine learning, enterprise databases,
experiment DAQs, and large legacy physics systems.
Machine Learning is a field of study that uses advanced algorithms and mathematical models to enable computers or machines to learn from data and make predictions or decisions without being explicitly programmed. In ML, data is used as a basis for training algorithms or models, which then analyze the data and identify patterns or relationships within it. These patterns are used to make predictions or decisions when presented with new, unseen data.
For example, imagine you have a dataset of temperature measurements over time, and you want to predict the temperature for a future date. You can use ML algorithms to analyze the historical data, identify patterns in temperature changes, and create a model that can predict future temperatures based on those patterns.
Machine Learning is used in various applications, such as image and speech recognition, autonomous vehicles, predictive analytics, and many more.
This talk will include
A high-volume, time-series, database for experimental data. Presentation of initial work carried out by Osprey DCS towards providing a specially designed timeseries database that can ingest and correlate data at high-speed, and can be used to feed ML systems.
An overview of Machine Learning Operations (MLOps) for accelerator control.
This talk will cover the use cases, experience, and experimental results from using Machine Learning enhanced modeling and tuning methods at SLAC National Accelerator Laboratory and collaborating accelerator facilities. These applications range from intelligent online optimization with minimal prior data to deployment of fast-executing surrogate models to predict complicated beam dynamics behavior online. An emphasis will be placed on practical lessons learned and future needs, with an eye toward control system and computing infrastructure.
Live observation of synchronized accelerator and photon data is crucial for best use of machine development time, and to setup demanding FEL configurations for the users. A Matlab class and a GUI have been developed and used in the last decade at the LCLS to acquire accelerator and photon data from different sources via EPICS and to present them. Data are synchronized on a pulse-to-pulse basis, processed, and displayed with low latency time, thus being useful for discovering correlations, live tuning, and understanding underlying physics while the data are taken. Scans are available by setting EPICS process variables and validating machine status by data timestamp. Consideration on the temporal latency, and the amount of data required for the system to be effective will be given.
ChannelFinder started as a simple directory service to address some of the limitations of EPICS's flat name space. Over the past few release the ChannelFinder infrastructure has expanded to provide additional features to manage the configuration of multiple services to supporting a pvAccess name server.
Recent work on PVXS, including to as a replacement for pva2pva.
EPICS Base 7.0.7 was released in September 2022, and the changes introduced there were described in a talk given by Ralph Lange at the EPICS meeting in Slovenia. This talk will briefly recap the main changes included in 7.0.7 and will then describe what has happened in Base since, including the code and documentation enhancements developed by attendees of the EPICS Codeathon hosted by Diamond in March.
The Fermilab Accelerator Directorate Robotics Initiative is working to develop and deploy robots in the accelerator tunnels to aid in observation and measurements in harsh environments to reduce personnel exposure and beam-off time. Longer term goals of the collaboration include telemanipulation of parts, autonomous navigation, and automation of repetitive tasks. One possible method of control of the robots involves utilizing the accelerator controls system. Current status of the initiative will be presented along with possible functional requirements required from the controls system to enable these technologies.
Fermilab has not been a traditional EPICS house, though there have been and still are some instances on site. We have now deployed an infrastructure designed to simplify, non-expert building of EPICS applications, both IOCs and user applications using a CI/CD pipeline. The status of this EPICS deployment at Fermilab will be summarized.
A short overview of the EPICS documentation on the web will be presented, along with some new ideas to make the documentation easier to find and also make it easier to contribute to documentation.
In a collaborational effort (ITER/HZB-BESSY/ESS/PSI), a Device Support for the OPC UA industrial SCADA protocol is under development. Goals, status and roadmap will be presented.
To support ITER's remote participation plans while honoring cybersecurity requirements, we are currently developing a new implementation of an "EPICS Diode", mirroring EPICS PVs through a strictly one-directional network connection.
Back in March, Diamond was hosting the 2023 EPICS Codeathon event.
Diamond are working on developing Ophyd v2, part of the Bluesky framework.
Ophyd v2 uses lessons learnt from Ophyd and Malcolm, the hardware scanning engine at DLS.
The key design concerns are modularity, separation of logic and I/O,
easy to implement hardware triggered scanning solutions and a clear migration path.
Currently there are many new developments for RTEMS and VMEbus CPUs as a basis for EPICS 7.
These are briefly presented here.
The main focus is the support of PowerPC boards like beatnik (MVME6100), MVME3700, MVME5100 etc..
Also new board alternatives on QorIQ basis (MVME2500) are addressed.
The stored beam was achieved in the Storage Ring at NSLS-II (National Synchrotron Light Source II) in March 2014. Even since then, the beam diagnostics and control system has been providing valuable fill pattern measurement data for machine operation. The control hardware as well as software has evolved over the past decade.
In many facilities there is a need to acquire and store distributed timing-synchronized FPGA-generated data for offline data analysis, fault detection and more recently AI/ML applications. While FPGA devices can easily generate and transmit Gpbs worth of data, the task of acquiring, storing and retrieving them are not trivial. This talk, although not answering this question, tries to explore some options of doing that and asking for feedback and collaboration in ways of achieving that.
Control System Studio (CS-Studio) is currently in the process of transitioning from Eclipse RCP to the next UI framework called Phoebus. To ease the transition of OPI (Operator Interface) screens developed for Eclipse-based CS-Studio to Phoebus, the "opi-generator" package provides a solution that generates properly formatted XML files for both CS-Studio and Phoebus from the same Python script. "opi-generator" is designed with a mechanism that can handle the differences in XML formats supported by the two platforms, and it also offers customizable support for unified screen style configurations.
RegMap is a tool used at Spallation Neutron Source to assist PLC developers with creating and testing EPICS integration with ease. PLC tags are described as a single line in Excel or CSV spreadsheet with user friendly fields. The RegMap tool converts saved spreadsheet into an EPICS database file and an OPI screen file, and allows for quick development cycle of register based devices without extensive EPICS experience.
The Accelerator Controls Operation Research Network (ACORN) project is a DOE O 413.3B project with a projected cost over $135M that aims to modernize Fermilab’s accelerator control system by 2028. A part of this modernization effort is to replace the existing user applications that are used by operators and experts to control and tune the accelerator complex. These applications use a diverse menagerie of outdated technologies including low-level UI frameworks such as Xlib and Java Swing. A goal of ACORN is to leverage modern web-based application frameworks for control system applications. Three candidate web UI frameworks were selected as part of the ACORN R&D – React, Fresh and Flutter. To evaluate the developer experience with these frameworks, 5 developers were given a simple task: use a framework to implement the user interface for a simple control room application: the telephone index. We present our findings and the reasoning behind our selected framework.
There are many VME modules in difference system in difference facilities. These VME module become obsolete due to the VME hardware obsolescence (such as front-end computer) and lack of support of operating systems (such as RTEMS) running on the VME systems. In this talk, we will discuss the migration path of the VME board to Ethernet on FPGA for control. The upgraded module can have the same VME formfactor for hardware compatibility reasons.
Last year, Cosylab made a public release of Autoparam and ADS EPICS modules and we are happy to see that the community is adopting them: soon after the release, we started to receive pull requests for bugfixes and new features. Fermilab was the first user of Autoparam and has an earlier version of the module. With this report, I'd like to present what was improved since Fermilab adopted Autoparam, and what benefits upgrading would bring.
This talk will cover how Fermilab is currently handling the incorporation of EPICS IOCs into a traditionally all ACNET control system. Specifically, it will discuss the role of Fermilab's Data Pool Manager (DPM) which provides a common data acquisition interface across multiple hardware architectures and protocols thus simplifying the development of data analysis and display components.
An update on the recent developments from the EPICS Tools and Services collaborations. This includes the latest releases of Phoebus and improvements to middle layer services like Olog, Save Restore, ChannelFinder, etc.
The PiMega detector family deployed at Sirius uses CERN’s Medipix3RX ASIC, delivering nearly 210 Gbps worth of raw payload in the largest detectors. In new generations based on Timepix4, the worst case bandwidth considered can reach up to 1.8 Tbps.
Current software architecture has a strong decoupling between data acquisition software stack and the EPICS IOC, relegating to EPICS the role of slow viewer only at 10 Hz update rate. A great deal of AreaDetector functions such as buffering, file writing, ROIs, image viewers, flat field array processing, are reimplemented elsewhere.
We discuss possibilities of higher integration to AreaDetector framework in newer software designs and present the challenges considered in this effort, future perspectives for new high-throughput detectors and questions that arise about possible hardware acceleration implementations in the framework, focusing on making the best use out of the existing and open source frameworks such as the AreaDetector.
CSNS (China Spallation Neutron Source) has used PSI Elog as the logbook system for several years and has stored more than ten thousand logs. However, the Elog does not provide responsive or separate web pages for mobile web client. Therefore, a separate mobile web client has been developed and deployed at CSNS to provide functions of user login, reading logs, creating logs, editing logs and deleting logs for the Elog. This talk will cover the design and implementation of the mobile web client for PSI Elog deployed at CSNS.
node-epics-ca is an EPICS Channel Access client for Node.js, which is intended to be used in the Node.js web applications or web services that want to directly talk to EPICS IOCs instead of an HTTP PV server. It is an FFI (Foreign Function Interface) implementation that talks to the existing EPICS Channel Access shared libraries using a third-party Node.js FFI package called koffi. This talk will cover the implementation and usage of node-epics-ca.
Presenting a thread model and possibilities for applying Transport Layer Security (TLS) to the PVA protocol.
This talk describes four cyber security related activities of
SLAC. First, a broad review of accelerator computing cyber and mission
reliability; our analysis method, findings and outcomes. Second, lab
and accelerator penetration testing, in particular, methods to control
coordinate, and trap, potentially hazardous scans. Third, a summary gap
analysis of recent regulatory orders from common practice at
accelerators, and nominal plan. Finally, briefly, first steps in
adding authentication and encryption to EPICS itself, as described in
other talks.
Cybersecurity is a rising concern in real-time and industrial control systems due to increased security threats toward these systems. In this talk, we discuss our ongoing efforts in security testing through modularization and fuzzing of the software stacks to detect vulnerabilities in different components. We have previously worked on modularizing the network stack of RTEMS into linkable static libraries. Our current efforts revolve around fuzzing using techniques such as model inference and machine learning to learn the state machine model of ICS protocol implementations to fuzz deeper states for stateful ICS protocol implementations (channel access, EPICS pvxs, etc.). We also discuss some ideas for our future work on improving the security posture of RTEMS-EPICS integration.
High Level Application service options for the Electron Ion Collider Project are currently under evaluation, both from the existing Relativistic Heavy Ion Collider proprietary Controls System and from the EPICS ecosystem. We will describe some early observations, concerns, and opportunities from amongst the more urgently required areas, including Logging, Name Lookups, Alarm notifications, and Sequencing tools.
After 50 years of operation, the LANSCE Control System is now an entirely EPICS control system. The original RICE (Remote Instrumentation and Control Equipment) system was replaced mainly by two types of EPICS IOCs National Instrument’s (NI) cRIO and a 2-form factor (VPX/cPCI crate) supporting three capabilities. Slow controls/monitoring use our Industrial I/O software, with EPICS device support talking to the FPGA via NI's C application programming interface. Interceptive beam diagnostics use our lvPortDriver to interface between LabVIEW real-time software and the EPICS IOC. VPX/cPCI systems use an EPICS IOC running on a soft-core processor implemented in the FPGA fabric. The soft-core IOCs have a special version of the channel access server (Data Access) that allows filtering of subscription updates. I will describe each architecture (including some recent improvements) and compare hardware and software lifecycle challenges for the various systems.
A brief overview of EPICS IOC implementation on SoC FPGA based new LLRF controllers. This presentation will include details of work completed so far, advantages, challenges and future work.
Wir stellen ein System vor welches es möglich macht EPICS auch in kleinen Laboren (Chemie, Physik) zu etablieren.
The Component Database (CDB) (developed for the Advanced Photon Source Upgrade) and a tightly coupled version of the eTraveler (originally developed at FRIB) have been in use for over 5 years to support the design, fabrication, and installation of thousands of components. This talk will provide a brief introduction to the applications and then describe the numerous ways the captured data has been used … many of which are beyond what was originally envisioned. A summary will include important “lessons learned” for future users of the tools.
HEX (High Energy Engineering X-ray Scattering) beamline is the first beamline at NSLS-II to use Power PMAC controller as main motion controller. In this presentation, technical details from controller selection, configuration, feedback devices, coordinated motion, EPICS integration, GUI etc. will be discussed. Implementation examples on different beamline components like slits, mono, endstation etc. will be shared as well.
The effective management of distributed data in the EPICS control network is crucial for efficient data analysis and controls software application development. One approach that has gained popularity among a large group of individuals is utilizing spreadsheet applications like LibreOffice Calc for data processing. The development of "PhantasyCalc" offers a systematic solution for integrating information from the EPICS network into LibreOffice Calc, providing simple yet powerful ways to work with diverse data sources. As a result, the spreadsheet becomes a user-friendly and shareable data analysis solution.
Image recognition and other tasks with visual components typically performed by humans remain integral to the establishment and maintenance of sample environments, including sample alignment, in neutron scattering and similar experiments. During this presentation, I will discuss the development of a user-friendly, EPICS-based interface designed for use by beamline operators which provides high-level access to system controls with the additional capability of deploying machine-learned models that reduce and simplify the workload of humans in executing controls tasks.
We developed a native CA and PVA library in JavaScript’s local runtime environment, Node.js. By using the asynchronous features of this programming language, this library can efficiently process a large amount of CA connections.
Closing remarks for the meeting. Announcement of next collaboration meeting.