presented by Matias Guijarro - BLISS team

Beamline Control Unit, Software Group

BLISS workshop DESY, 21/11/2023

 

Introduction to the

image/svg+xml

 

control system

ESRF Extremely Brilliant Source (EBS) update program

4th generation light source

150 M€ investment (2015-2022)

100x better

X-ray beams

New

state-of-the-art

beamlines

Data as a Service strategy

BLISS project goals

State-of-the-art beamline control

Advanced scans, Trajectories,

Data management

Integrated environment

Configuration, Command Line Interface, Live Data Display

Ready for new challenges

Online data analysis, live feedback, extensibility of the system

December, 2015

BLISS project starts

May, 2018

First tests with friendly users

October, 2017

1st talk about BLISS @ ICALEPCS '17 (Barcelona)

 

ESRF Upgrade shutdown period

BLISS team in September 2019

December, 2018

ESRF EBS shutdown

August, 2020

Restart of ESRF user operation

 August 2023

BLISS 1.0

image/svg+xml

Deployment on EBS beamlines

... continuing BLISS deployment on all beamlines ...

March 2020

COVID lockdown

BLISS 1.11

 December, 2020

 February, 2020

1st publication with BLISS data

(hal-02466436)

BLISS project timeline

Data portal: https://data.esrf.fr

E-logbook

Live data visualization

Online data analysis workflows

Graphical user interface for data acquisition

Faster detectors, data production

Efficient data format, storage

Today's experiment control system challenges

image/svg+xml

Experiment scripts, user sequences

Jungfrau @ ID29

EWOKS

BLISS philosophy

Python library shipped with tools

Command Line Interface, web terminal

Configuration application

Live visualization

Data service & file writer

At the heart of the ESRF software ecosystem

Implementation of Data Policy

Graphical Interfaces

Online Data Analysis

Holistic approach to run synchrotron experiments

Sequencer for any kind of acquisition procedures

 

hardware control

or via built-in drivers

BLISS architecture and ecosystem

scalar

Aspects of BLISS

  1. Beacon server

  2. Shell

  3. Data Management

  4. Graphical Interfaces

  5. Integration of hardware devices

  6. Scanning engine

  7. Data publishing

  8. Live visualization

  9. Data saving

10. Data processing

Beacon server

1

Beacon server: configuration and services

Transient data store

Settings cache

1

BLISS shell

2

BLISS command line interface

bliss -s <session_name>

User sessions group beamline devices for an experiment + a Python setup file

2

based on ptpython: completion, typing helper, function parameter hints, syntax highlighting...

tmux is a terminal multiplexer used to run multiple terminals within the same window - multiple people can join the same BLISS session

BLISS web terminal

image/svg+xml

ptpython

interpreter

blissterm server

ptpython

interpreter

API

blissterm application

socket io

xtermjs

Daiquiri UI components library

logic

blissdata

2

Data management

3

Data management

3

Users can feed the ESRF Data Portal directly from the BLISS command line

 

Proposal

Datasets

Metadata

  • acquisition parameters
  • sample parameters

Gallery

  • images, plots, associated to a dataset

Tape interface

newcollection("sample1")

Data management: electronic logbook

3

IH-LS-3167: elemental distributions in microalgae

BLISS commands to submit comments, plots, execution output, logs

 

elog_add("...")

 

Automatic notifications

Manually editable from the Data Portal

searchable

Graphical interfaces

Introducing Daiquiri

1. White Rum

2. Lima juice

3. Fine sugar

4. a drop of

4

Daiquiri

A web-based User Interface framework, designed to build data acquisition and beamline control applications

4

Daiquiri architecture

4

image/svg+xml

BLISS

daiquiri-local

Daiquiri example: integrated user interfaces

4

Daiquiri example: parameterisation

4

4

Daiquiri example: monitoring

Integration of hardware devices

5

Integration of hardware devices

BLISS provides communication helpers to implement protocols more easily, and mini-frameworks for each category of devices

5

abstraction level

Generic controllers like "attribute as Counter" makes integration of TANGO devices without coding anything

scalar

"Counters" for scans

- class: tango_attr_as_counter
  uri: id00/undulator_dev/u35c
  counters:
    - name: u35c_pos
      attr_name: position
      mode: MEAN
      unit: mm

synchronization

low latency

flexibility

BLISS scans

6

Beacon static configuration service

BLISS scan engine

All scans in BLISS are based on the same Scan object

 

The Scan object iterates through the Acquisition Chain

 

The Acquisition Chain is a tree structure, with Acquisition Masters as nodes and Acquisition Slaves as leaves

 

The chain describes what triggers what (software or hardware triggering)

 

Both Acquisition Masters and Slaves can read data ;

data is sent to Redis via Acquisition Channels

6

Beacon static configuration service

Scan counters

The name "counter" comes from SPEC - in BLISS it represents a placeholder for the data to be recorded, that users have to put in scans

Sampling counters example

Integrating counters example

6

Beacon static configuration service

Scan measurement groups

A Measurement Group is a Python object containing Counters

 

Allow user to enable/disable counters

 

There is 1 global Measurement Group per BLISS session

(indicates "default" counters == what to count when a scan is launched without counter argument)

DEMO [1]: align_counters
 Out [1]: MeasurementGroup: align_counters (state='default')
        - Existing states : 'default'

        Enabled                          Disabled
        -------------------------------  --------------------
        sim_diode_sampling_ctrl:diode
        sim_diode_sampling_ctrl:diode2
        sim_diode_sampling_ctrl:diode3
- class: MeasurementGroup
  plugin: session
  name: align_counters
  counters: [diode, diode2, diode3]

6

Beacon static configuration service

BLISS standard scans

Step-by-step scans, not requiring any special hardware

ct

 

 

 

single count

loopscan

 

 

 

repeated counts

timescan

 

 

 

infinite counts

ascan

 

 

 

move a single motor, N steps

a2scan

 

 

 

moves 2 motors, N steps

amesh

 

 

 

moves 2 motors, N*M steps

pointscan

 

 

 

moves given motors, list of coordinates, N steps

Note: for standard scans, the acquisition chain is built automatically, trying to guess the more optimized settings

 

The "default chain" can be customized to tell to use a counting card instead of a software timer as a master to trigger a 2D detector (for example)

6

Beacon static configuration service

Common parameters for standard scans

scan_command(...,
             *counters, #optional list of counters, or mix of counters+MG
             name="scan", 
             title=None, #default (None): scan cmd+arguments separated by space
             save=True,
         save_images=None, #default (None): follows the "save" argument
             sleep_time=0 #float, sleep time after counting (between 2 points)
             stab_time=0 #float, stabilization time after a move, before counting
             run=True,
             return_scan=True)

 

6

Software continuous scan example

Continuous scan with a motor triggering MCA and 2D detector acquisition, while a timer triggers diode readings

image/svg+xml

6

Data publishing

7

Data publishing

7

BLISS relies on blissdata to publish acquisition data to Redis

For big 2D data, only references are published

blissdata is a separate package, it can be used to read data (incl. during acquisition)

Daiquiri

web GUI

BLISS data flow

7

Live visualization

8

Live visualization

8

Flint is a PyQt graphical application shipped with BLISS

 

It is based on the ESRF silx toolkit

 

It relies on blissdata to display live data from BLISS acquisition

Interactive control of Flint

8

Flint can be remote controlled from BLISS scripts, thanks to dedicated Python functions

 

This allows to create interactive experiment sequences with no PyQt code to write

TOMO_SESSION [1]: edit_roi_counters(tomocam, 1.0)
WARNING 2020-10-16 15:39:45,049 flint: Flint starting...
Waiting for ROI edition to finish on tomocam [default, default]...
WARNING 2020-10-16 15:39:46,999 flint: Waiting for selection in Flint window.

Data saving

9

Data saving

9

The BLISS Nexus Writer is a TANGO server that uses blissdata to listen to scans and to save them in a

Nexus-compliant HDF5 file

Data processing

10

Data processing

10

Processing data is hard and takes time !

What processing do I need to do?

Where is my data ?

What software do I need to install? And how ?

There is too much data!
I will never be able to process it all !

Oh god, I forgot to apply the mask !

The software is not working and the PhD/post-doc who wrote it left !

Data processing

10

Problem: how to improve data processing experience, and how to apply processing as soon as possible?

  • To make informed decisions during acquisition
  • To provide to users to help/encourage  further data analysis

i.e, to go from this...

to this:

Data processing: workflows

10

Ewoks (Esrf WorKflowS) to the rescue

Workflows are data processing pipelines composed of several steps (tasks)

image/svg+xml

Acquisition control

Execute ewoks workflow on

  • local machines (immediate feed)
  • slurm nodes

Tasks can be rearranged or reused in other workflows without deep knowledge of the task content

Upload result to data portal

Persist result for further analysis

Visualization, feedback

image/svg+xml

EWOKS example: Exafs visualization

10

Raw parameter space

Workflow based on xraylarch

Parameter space in which scientific decisions are made

Last but not least...

Selected features praised by ESRF scientists

 

1. Python!

2. Software axes and counters

3. Software regulation loops

Python!

Example from BM23 (Kirill Lomachenko): beamline "health check" for local contacts via Telegram, thanks to Telegram Python API

Scientists can take advantage of the huge Python ecosystem in their own scripts

1.1

Python!

from bliss.setup_globals import fsh, diffrz
import numpy as np

def ftomo_series(scanname, start, num_scans, sleep_time=0, pars=None):
    for i in np.arange(start, start+num_scans): 
        newdataset(f"{scanname}{i}")
        umv(diffrz, pars['start_pos'])
        with fsh.closed_context():
            print("taking dark images")
            ftimescan(pars['exp_time'], pars['nref'], 0)
        if i==start or not(i%pars['ref_int']):
            print("taking flat images")
            ref_scan(pars['ref_mot'], pars['exp_time'], pars['nref'], pars['ref_step'], pars['start_pos'], pars['scan_mode'])
        else:
            print("skipping flat images - stay where we are")
            ftimescan(pars['exp_time'], 10,0)
        print("taking projections...")
        fscan(diffrz, pars['start_pos'], pars['step_size'], pars['num_proj'], pars['exp_time'], scan_mode = pars['scan_mode'])
        print("resetting diffrz to 0")
        umv(diffrz, pars['start_pos']+360)
        diffrz.position=pars['start_pos']
        sleep(sleep_time)

Example from ID11: tomography @ nanofocus endstation

25 degrees/s.

diffrz

500 fps

700 GB per hour of (compressed) data

User-written acquisition sequence == night shift macro

continuous, hardware-synchronized scans

direct import of beamline objects from BLISS setup in user scripts

nice Python features: context manager, to ensure fast shutter is closed

whole Python ecosystem can be available

Python functions, with named arguments and default values

1.2

Software pseudo axes

Example from BM32: Laue diffraction

Pseudo axis to switch from Energy (HKL reciprocal space)

to crystal angle (real hardware, rotation axis)

Energy to Bragg angle abacus




from bliss.controller.motor import CalcController

class EnergyCalc(CalcController):
  def initialize(self):
    """Load abacus data, to be able to do the calculation"""
    ...
    
  def calc_to_real(self, **positions):
    """Take pseudo axis position, and return real axis position"""
    ...
    
  def calc_from_real(self, **positions):
    """Return calculated axis position from real axis position"""
    ...
    
    
- controller:
    class: EnergyCalc
    package: bm32.ctrls.pseudo_laue
    axes:

    -   name: $thf
        tags: real thf

    -   name: Edia
        tags: Edia
        unit: eV

file: ~/local/bm32/ctrls/pseudo_laue.py

2.1

Software pseudo counters

In BLISS, a counter is a Python object representing an experimental parameter which can be measured during a scan

A Software counter takes multiple counters as inputs and produces multiple calculated counters as outputs

Software counters can be created in code, or declared from the BLISS configuration:

- class: ExpressionCalcCounterController
  name: calc_diodes
  inputs:
      - counter: $diode1
        tags: d1
      - counter: $diode2
        tags: d2
      - counter: $diode3
        tags: d3
      - counter: $diode4
        tags: d4
  constants:
       m : 0.5
       n : 0.8
  outputs:
      - name: intensity
        expression:  (d1 + d2 + d3 + d4 )
      - name: cen_x
        expression:  m*(d2-d1) + m*(d4-d3)
      - name: cen_y
        expression:  m*(d3-d1) + m*(d4-d2)

2.2

Software regulation loops

Example: beam position regulation

Compensation of low frequency drifts due to thermal load changes or mechanical instability

BLISS provides a Software Loop object, that knows how to regulate with PID parameters

 

Existing BLISS objects with "read" interface can be taken as Input, and objects with "set" interface can be taken as Output

3

DEMO

And now, ladies and gentlemen...

Thanks for your attention

Questions ?

Acknowledgements

BLISS Core Development team 2015-2023

 

T. Coutinho*, W. De Nolf (DAU), L. Felix, B. Formet* , P. Guillou, C. Guilloud,

M. Guijarro,  V. Michel*, P. Pancino*, E. Papillon, L. Pithan*,

S. Petitdemange, V. Valls

Beamline Operation team

A. Beteva, G. Berruyer, L. Claustre, S. Debionne, S. Fisher, A. Homs, M-C. Lagier, A. Mauro, C. Muzelle, M. Oscarsson, D. Naudet, H. Witsch  

head of Software Group: A. Gotz, head of BCU: J. Meyer

(*: left ESRF)

Data Automation Unit: M. Bodin, A. De Maria, L. Huder