NOBUGS Conference 2024
25th of September, 2024
presentation by Matias Guijarro
well organized and hosted by
Integrating ESRF data
with PSI
About the speaker
Software Engineer
20-year career at ESRF,
Beamline Control Unit
BLISS project responsible: 2017-2023
In sabbatical leave since 01/24
Joined BEC Development Team at Science IT Infrastructure and Services Department, PSI
Talk outline
1. Introduction to BEC
2. Introduction to Bliss and Blissdata
3. Integrating Blissdata with BEC
4. Conclusion
Introduction to
About BEC
BEC is the new Experiments Control System for SLS 2.0 beamlines
inspired by (NSLS-II)
Open source software with BSD license
Code repository hosted on PSI gitlab
in-house development @ PSI
Data catalogue: SciCat
E-logbook: SciLog
Live data visualization
2D detectors
Jungfraujoch, std-daq, AreaDetector...
Data Archiving
Experiment scripts, user sequences
About BEC
Storage Infrastructure
User interaction
Beamline Device control
Data Acquisition: Ophyd devices
BEC Widgets
Data Processing
Technical choices
Written in Python
Services-oriented architecture
Redis as data buffer, message broker
IPython shell
Devices control via Ophyd (Bluesky)
sub-project: ophyd_devices
sub-project: bec_widgets
Graphical framework based on Qt
Architecture
Device Server
Scan Server
Scan Bundler Service
Data Processing Service
Scihub Service
File Writer Service
BEC shell
BEC core library
BEC Widgets app
Connecting to BEC services
RedisConnector is a BEC core library object to communicate with all services (using Redis as message broker)
RedisConnector
pub/sub events listener thread
streams
listener thread
BEC messages queue
msg processing thread
execute callbacks
set (store in Redis),
publish (event)
or add to stream
Messages that can be sent or received by the Redis Connector are defined using Pydantic models, and all derive from the BECMessage base class
Pydantic ensures data validation for the message data fields
BEC services connect to endpoints : named targets which specify the expected message type, and the action to take in Redis (store, publish, ...)
Scan
Server
Device
Server
RedisConnector
RedisConnector
endpoint:
device_instructions
endpoint:
device_instructions
BECMessage
DeviceInstruction
BECMessage
DeviceInstruction
serialization
{
dev name: "xxx",
action: "trigger",
parameters: { }
}
BEC Messages
data
Introduction to
Python library with tools
Command Line Interface, web terminal
Configuration application
Live visualization
Data access, File Writer
Developed at ESRF since 2016 - designed for the EBS Upgrade
Sequencer for any kind of acquisition procedures:
built-in drivers
or hardware control
About BLISS
Link to latest BLISS article
Guijarro, M., Felix, L., De Nolf, W., Meyer, J., & Götz, A. (2023). A Marriage Made in BLISS—An Integrated Beamline Control System. Synchrotron Radiation News, 36(6), 12–19.
scalar
Devices & sequences configuration
class: elmo
udp:
url: 160.103.51.174
axes:
- name: nth
acceleration: 36
steps_per_unit: 30577
velocity: 180.0
home_velocity: 15
velocity_high_limit: 360.0
backlash: 0.0
Web interface for configuration editing
Beacon server
BLISS ecosystem at a glance
blissdata is a separate package with limited dependencies, easy to integrate in any Python project
Blissdata
Blissdata publishes data (producer)
Blissdata can also retrieve previously published data (consumers)
Support technology:
+ extensions RedisJSON, RedisSearch
Blissdata
SCAN
{...} JSON description
Streams
...
{...} JSON description
{...} JSON description
{...} JSON description
Everything is built around Scans and Streams
Blissdata defines a model to represent scans, and provides acquisition streams to push data
Scans of scans are handled
BLISS relies on blissdata to publish acquisition data to Redis
For big data (mainly 2D), only references are published
Blissdata
Blissdata feature :
decoupling stream contents and data
streams are filled with data events, data can be elsewhere
Blissdata
Data
Data events
OR
Redis stream
Blissdata
2 cases for data
1. Can be shipped with the event:
0D, 1D, small 2D... are within the Redis stream
2. Data is too large, or data is stored by acquisition in its own file:
JSON events are emitted, need a Blissdata plugin to reference (and de-reference) data
Integrating Blissdata with BEC
BEC distributed architecture and clean message passing makes it easy to interface with any of the components
Or to add a new component connected to existing services
The chosen approach for the proof-of-concept is to minimize the number of processes, copies and serialization/deserialization of data, and to stick to the more direct implementation
Integrating Blissdata with BEC
Device Server
Scan Server
blissdata
PyMca:
A tool designed to assist with XRF data analysis and beyond.
Publishing BEC scans with blissdata gives access to ESRF HDF5 Writer, display and processing tools
Current proposal
Work in progress
BEC with Blissdata, Flint, BLISS Writer
What's next ?
Integrating 2D data
AreaDetector, Std-Daq (PSI), JungfrauJoch (PSI)
= writing Blissdata plugins, to connect via ZMQ or getting from file
Handling of EPICS devices writing data directly to files
= again writing a Blissdata plugin for those cases
(could be generic if data is in the same HDF5 format for example)
Next steps
Conclusion
Conclusion
BEC distributed architecture provides entry points for plugging in new services, or extending parts of the system
Blissdata can easily be embedded into any data acquisition system
- generic concept of scans and data streams
- on-going work at ALBA with Sardana, at DESY with Sardana, Bluesky (ROCK-IT project) and at PSI with BEC (this talk !)
What's next ?
- BEC+Blissdata: work ahead to support 2D detectors (outside )
- need for better specification of Blissdata schemas, and documentation
Small development teams, existing software: let's mutualize data handling, visualization, processing, and archiving tools ! At least aim for interoperability to better serve users community
Technology alert!
Redis is in-memory store
It would be valuable to extend storage to fast disk (SSD, nvme), to be able to store longer and more data, while still benefiting from data access provided by Blissdata...
Introducing KVRocks
- Redis-compatible protocol
- RedisSearch implementation
- RedisJSON implementation
- Built on top of RocksDB
nobugs24_blissdata_bec
By Matias Guijarro
