3rdof December, 2021

BeamLine Instrumentation Support Software

Presentation for ACU

Project history and milestones

at the heart of ESRF data acquisition

TUPV043

baked with love by the ESRF Instrumentation Services & Development Division Software Group

Beamline Control Unit: A. Beteva, G. Berruyer, L. Claustre, S. Debionne, S. Fisher, M. Guijarro, P. Guillou, C. Guilloud, A. Homs, M.C. Lagier, D. Mammeri, A. Mauro, J. Meyer, C. Muzelle, M. Oscarsson, E. Papillon, S. Petitdemange,  V. Valls, H. Witsch / Data Analysis Unit: W. De Nolf

Beacon : web application to configure a whole beamline

Flint : data visualisation

based on the ESRF silx toolkit

Modular architecture, Abstraction layers and adapters

Data publishing and archiving enables online data analysis

Services built on top of

BLISS integrated environment covers a wide range of beamline control needs

BLISS shell integrates with Daiquiri UI the ESRF web-based framework for user interfaces

Daiquiri UI

see: THAR02

+

BLISS stands for BeamLine Instrumentation Support Software,

it is a new synchrotron experiments control system written in Python

BLISS controls today 24 beamlines (out of 48), full deployment is aimed in 2023

TUPV043

BLISS Key Principles

Direct hardware control

Persistent

settings cache

 Data publishing,

Transient storage

Scan acquisition chain represented as a tree

All-in-one solution

All I/O based on gevent

cooperative multi-tasking,

asyncio interoperability

 

BLISS Beacon server

Web interface for configuration editing

Beacon server

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment + Python setup file

configuration management

&

services

Transient data store

Settings cache

Centralized logging and log viewer web application

 class: elmo
  udp:
    url: 160.103.51.174
  axes:
  - name: nth
    acceleration: 36
    steps_per_unit: 30577
    velocity: 180.0
    home_velocity: 15
    velocity_low_limit: .inf
    velocity_high_limit: 360.0
    low_limit: -.inf
    high_limit: .inf
    sign: 1
    backlash: 0.0
- class: aerotech
  tcp:
    url: id31aerotech-1
  axes:
  - name: nth_aero
    acceleration: 5
    steps_per_unit: 1

database implementation

TUPV043

BLISS Modular Architecture

Beacon server

data

streams

Mini frameworks within BLISS, to ease integration of same kind of hardware

Adaptation layer, to make hardware controllers behaving properly in scans, depending on the scan

TUPV043

BLISS data publishing and archiving

Daiquiri web GUI

see: THAR02

  • redis streams are used to push data to redis during acquisition
  • only references to heavy data (2D) are published to redis, refs will be resolved at runtime when asked by a listener
  • online data analysis scripts can be easily written using BLISS Python API
  • BLISS fully supports ESRF data policy (based on PaNdata Europe and FAIR data initiatives), and interoperates with ICAT and ESRF data portal

Redis

1

What is Redis ?

  • { key: value, } store
    • binary safe string
    • max 512 MB for a string value
    • key can have a data type: List, Set, Sorted Set, Hash (containing multiple string values)
  • in memory
    • can also persist on disk, but we do not do that for data
  • since Redis 6: streams

1

producer

producer

producer

producer

consumer

consumer

consumer

XADD

XREAD

  • data publishing using redis streams
    • 1 stream per data channel

Redis streams

1

individual string values

we arbitrary limit streams to 2000 string values max

1 string value can correspond to multiple data events

BLISS client API: knows scans structure, read data

Note: Redis data is only transient !

Consumers need to continuously read from it, to get complete data

 

(Like Nexus Writer for example)

1

References

  • Heavy data (== 2D data, up to now) is not stored in redis streams
    • References are stored instead
  • When receiving a reference, the BLISS client API resolves the reference
    • get data from Tango server memory (if still available)
    • fallback: reading from disk

References

2

BLISS development tools

Issues list, sprints reviews, discussions and tasks to do on ESRF-hosted gitlab

Continuous Integration:

gitlab-ci

Deployment:

Documentation:

MkDocs

Code formatting

BLISS development methodology

Prioritized

todo

Analysis phase

Coding

Integration tests

Ready to merge

new features

bug fixes

backlog

(tasks)

Kanban

BLISS presentation for ACU (3/12/2021)

By Matias Guijarro

BLISS presentation for ACU (3/12/2021)

Presentation of BLISS to ACU colleagues

  • 195