NOBUGS Conference 2024

25th of September, 2024

presentation by Matias Guijarro

well organized and hosted by

Integrating  ESRF        data

with PSI            

About the speaker

Software Engineer

 

20-year career at ESRF,

Beamline Control Unit

 

BLISS project responsible: 2017-2023

In sabbatical leave since 01/24

 

 

Joined BEC Development Team at Science IT Infrastructure and Services Department, PSI

Talk outline

1. Introduction to BEC

2. Introduction to Bliss and Blissdata

3. Integrating Blissdata with BEC

4. Conclusion

Introduction to

About BEC

BEC is the new Experiments Control System for SLS 2.0 beamlines

                       

inspired by                           (NSLS-II)

Open source software with BSD license

Code repository hosted on PSI gitlab

in-house development @ PSI

Data catalogue: SciCat

E-logbook: SciLog

Live data visualization

2D detectors

Jungfraujoch, std-daq, AreaDetector...

Data Archiving

Experiment scripts, user sequences

About BEC

Storage Infrastructure

User interaction

Beamline Device control

Data Acquisition: Ophyd devices

BEC Widgets

Data Processing

Technical choices

Written in Python

Services-oriented architecture

Redis as data buffer, message broker

IPython shell

Devices control via Ophyd (Bluesky)

sub-project: ophyd_devices

sub-project: bec_widgets

Graphical framework based on Qt

Architecture

Device Server

 

Scan Server

Scan Bundler Service

Data Processing Service

Scihub Service

File Writer Service

BEC shell

 

BEC core library

BEC Widgets app

 

Connecting to BEC services

RedisConnector is a BEC core library object to communicate with all services (using Redis as message broker)

RedisConnector

pub/sub events listener thread

streams

listener thread

BEC messages queue

msg processing thread

execute callbacks

set (store in Redis),

publish (event)

or add to stream

Messages that can be sent or received by the Redis Connector are defined using Pydantic models, and all derive from the BECMessage base class

 

Pydantic ensures data validation for the message data fields

 

BEC services connect to endpoints : named targets which specify the expected message type, and the action to take in Redis (store, publish, ...)

 

Scan

Server

 

 

Device

Server

 

RedisConnector

RedisConnector

endpoint:

device_instructions

endpoint:

device_instructions

BECMessage

DeviceInstruction

BECMessage

DeviceInstruction

serialization

{

dev name: "xxx",

action: "trigger",

parameters: { }

}

BEC Messages

image/svg+xml

data

Introduction to

Python library with tools

Command Line Interface, web terminal

Configuration application

Live visualization

Data access, File Writer

Developed at ESRF since 2016 - designed for the EBS Upgrade

Sequencer for any kind of acquisition procedures:

 

built-in drivers

or                        hardware control

About BLISS

image/svg+xml

Link to latest BLISS article

 

Guijarro, M., Felix, L., De Nolf, W., Meyer, J., & Götz, A. (2023). A Marriage Made in BLISS—An Integrated Beamline Control System. Synchrotron Radiation News, 36(6), 12–19.

scalar

Devices & sequences configuration

 class: elmo
  udp:
    url: 160.103.51.174
  axes:
  - name: nth
    acceleration: 36
    steps_per_unit: 30577
    velocity: 180.0
    home_velocity: 15
    velocity_high_limit: 360.0
    backlash: 0.0

Web interface for configuration editing

Beacon server

BLISS ecosystem at a glance

image/svg+xml

blissdata is a separate package with limited dependencies, easy to integrate in any Python project

Blissdata

image/svg+xml

Blissdata publishes data (producer)

 

Blissdata can also retrieve previously published data (consumers)

 

Support technology:            

+ extensions RedisJSON, RedisSearch

Blissdata

image/svg+xml

SCAN

{...} JSON description

Streams

...

{...} JSON description

{...} JSON description

{...} JSON description

Everything is built around Scans and Streams

 

Blissdata defines a model to represent scans, and provides acquisition streams to push data

 

Scans of scans are handled

BLISS relies on blissdata to publish acquisition data to Redis

For big data (mainly 2D), only references are published

Blissdata

image/svg+xml

Blissdata feature :

decoupling stream contents and data

streams are filled with data events, data can be elsewhere

Blissdata

image/svg+xml

Data

Data events

OR

Redis stream

Blissdata

image/svg+xml

2 cases for data

 

1. Can be shipped with the event:

0D, 1D, small 2D... are within the Redis stream

 

2. Data is too large, or data is stored by acquisition in its own file:

JSON events are emitted, need a Blissdata plugin to reference (and de-reference) data

Integrating Blissdata with BEC

BEC distributed architecture and clean message passing makes it easy to interface with any of the components

 

Or to add a new component connected to existing services

 

The chosen approach for the proof-of-concept is to minimize the number of processes, copies and serialization/deserialization of data, and to stick to the more direct implementation

Integrating Blissdata with BEC

image/svg+xml

Device Server

Scan Server

blissdata

PyMca:

A tool designed to assist with XRF data analysis and beyond.

Publishing BEC scans with blissdata gives access to ESRF HDF5 Writer, display and processing tools

image/svg+xml

Current proposal

image/svg+xml

Work in progress

BEC with Blissdata, Flint, BLISS Writer

image/svg+xml

What's next ?

Integrating 2D data

AreaDetector, Std-Daq (PSI), JungfrauJoch (PSI)

 

= writing Blissdata plugins, to connect via ZMQ or getting from file

Handling of EPICS devices writing data directly to files

 

= again writing a Blissdata plugin for those cases

(could be generic if data is in the same HDF5 format for example)

image/svg+xml

Next steps

Conclusion

Conclusion

BEC distributed architecture provides entry points for plugging in new services, or extending parts of the system

 

Blissdata can easily be embedded into any data acquisition system

  • generic concept of scans and data streams
  • on-going work at ALBA with Sardana, at DESY with Sardana, Bluesky (ROCK-IT project) and at PSI with BEC (this talk !)

 

What's next ?

  • BEC+Blissdata: work ahead to support 2D detectors (outside             )
  • need for better specification of Blissdata schemas, and documentation

 

Small development teams, existing software: let's mutualize data handling, visualization, processing, and archiving tools ! At least aim for interoperability to better serve users community

Technology alert!

Redis is in-memory store

 

It would be valuable to extend storage to fast disk (SSD, nvme), to be able to store longer and more data, while still benefiting from data access provided by Blissdata...

Introducing KVRocks        

  • Redis-compatible protocol
  • RedisSearch implementation
  • RedisJSON implementation
  • Built on top of RocksDB