From spec to Python, BLISS for EBS @ ESRF

From Spec to Python:

BLISS for ESRF EBS

presented by Matias Guijarro - BLISS team

Beamline Control Unit / Software Group

ESRF, Grenoble, France

Beamline Control Automation & Data Acquisition Workshop - 17^th January, 2019

ESRF Extremely Brilliant Source (EBS) project

4th generation light source

150 M€ investment (2015-2022)

100x better

X-ray beams

New

state-of-the-art

beamlines

Data as a Service strategy

From Spec to Python

Spec has been driving ESRF experiments for 26 years

                   Welcome to "spec" Release 6.03.11
         Copyright (c) 1987-2016 Certified Scientific Software
                          All rights reserved

 (Portions derived from a program developed at Harvard University.)
 (Linked with BSD libedit library for command line editing.)

 Using "/bliss/users/blissadm/spec/redhate4/spec.d"
  for auxiliary file directory (SPECD).

 Getting configuration parameters from "SPECD/matias/config".

 =
 Type h changes for info on latest changes.
 Browse to http://www.certif.com for complete documentation.
 =

 Reading file "SPECD/standard.mac".
 Warning:  No open data file.  Using "/dev/null".
 Type "startup" to initialize data file, etc.

 Reading file "SPECD/site.mac".
 Reading file "/bliss/users/blissadm/local/spec/macros/ID00setup.mac" (level 2).
 Reading file "/bliss/users/blissadm/spec/macros/lock.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlocal.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlist.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/pseudo.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/tw.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/joy.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/cplotarray.mac" (level 3).

 Doing SETUP.
 Reading file "SPECD/setup".
 Reading file "SPECD/matias/setup".

 1.MATIAS>

Poor macro language

No continuous scan framework

Limited data management

Simplistic configuration

No extensibility

Lots of workarounds

High operating cost

Revolution !

BeamLine Instrumentation Support Software

Welcome,

BLISS project goals

State-of-the-art beamline control

Continuous scans, Trajectories,

Data management

Integrated environment

Command Line Interface, Configuration, Data display

Ready for new challenges

Online data analysis, live feedback, full-featured programming language

BLISS philosophy

Integrated beamline control system

Software library
Command Line Interface
Configuration application
Online visualization

Python everywhere

command line, user sequences, etc.

Generic scan engine

Continuous, step-by-step, multiple masters

Data management, decoupling acquisition and archiving

BLISS project development tools

Issues list, sprints reviews, discussions and tasks to do on ESRF-hosted gitlab

Continuous Integration:

gitlab-ci

Deployment:

Documentation:

MkDocs

Code formatting: Black

All I/O based on gevent

cooperative multi-tasking

Direct hardware control

Distributed

objects state

Persistent

settings cache

Data publishing,

Transient storage

Scan acquisition chain, represented as a tree

BLISS design principles

online data analysis

data visualisation

data archiving

2D detector

Beacon server

Beacon static configuration service

BLISS modular architecture

user sequences

Beacon static configuration service

Web interface for configuration editing

Beacon server

.yml

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment,

Python setup file

Can replace TANGO DB

Conversion script provided

Beacon server

configuration

Beacon static configuration service

Beacon, example configuration file

sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│   ├── diode.yml
│   ├── __init__.yml
│   └── motors
│       ├── bv.yml
│       ├── DtoX.yml
│       ├── __init__.yml
│       ├── md2.yml
│       ├── mirror1.yml
│       ├── slits.yml
│       └── table.yml
├── oh
│   ├── bpm.yml
│   ├── __init__.yml
│   ├── motors
│   │   ├── bv.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── slits.yml
│   │   └── transfocators.yml
│   └── wagos.yml
└── sessions
    ├── id232_setup.py
    ├── id232.yml
    └── __init__.yml

- controller:
    class: IcePAP
    host: iceid2322
    axes:
    - name: mbv4mot
      address: 1
      steps_per_unit: 817
      velocity: 0.3
      acceleration: 3

example configuration of a beam viewer motor

Beacon static configuration service

Web interface for configuration editing

Beacon server

.yml

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment,

Python setup file

Can replace TANGO DB

Conversion script provided

Beacon server

configuration

+ services based on

Message broker: state sharing,

distributed lock

Transient data store

Persistent settings cache

Beacon static configuration service

BLISS scan framework

synchronization

low latency

flexibility

Beacon static configuration service

BLISS scan framework

Data writer

HDF5

synchronization

low latency

flexibility

Acquisition chain

a tree with master & slave nodes
master triggers data acquisition
slave takes data

AcquisitionMaster, AcquisitionDevice

wrappers around BLISS control objects

BLISS scan example

Continuous scan with a motor triggering MCA and 2D detector acquisition, while a timer triggers diode readings

"Classic" step-by-step scans

Directly available as functions from 'bliss.common.standard'
- Example: ascan(axis, start, stop, npoints, count_time, *counters)
Default acquisition chain
Use the same underlying framework as continuous scans

BLISS online data publishing

While a scan is running, data is published to the redis database provided by Beacon

scalar values are stored directly
bigger data (images, spectra) is just referenced
configurable time to live (TTL)

Any external process can access redis data to perform online data analysis or live feedback for example

BLISS scan data flow

Writing data acquisition sequences

from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import numpy # I know you dreamt of it
import gevent

def set_detector_cover(in):
    wcidxx.set('detcover', in)
 
    # 5 seconds timeout waiting for detector cover to move
    with gevent.Timeout(5):
        while wcidxx.get('detcover_in') == in:
          time.sleep(0.1)

def my_super_experiment(name):
    safety_shutter.open()

    old_att = attenuators.get()

    def restore_beamline():
        set_detcover_open(False)
        attenuators.set(old_att)

    with cleanup(safety_shutter.close):
        with error_cleanup(restore_beamline):
            attenuators.set(50)
            set_detcover_open(True)

            SCAN_SAVING.name = name
            MEASUREMENT_GROUP.enable('diode')
            
            data_node = dscan(m0, -5, 5, 10, 0.1)
            
            for data in data_node.walk_data():
              # do something useful with data...

with cleanup(safety_shutter.close): # cleanup is always called at the end
    with error_cleanup(restore_beamline):  # only called in case of error
        ...

Use of Python context managers for cleanup

Normal Python functions

Easy timeouts with gevent.Timeout

# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
    while wcidxx.get('detcover_in') == in:
        time.sleep(0.1)

BLISS development phases (2017-2019)

1. Coding sprints with different "product owners"

2. Assessing the system on different beamlines before the EBS shutdown, real-world experiments

3. Deployment on 15 (20?) beamlines during EBS shutdown

We are here

Software Development

Kanban methodology

Full transitions, no more Spec on target beamlines

Scrum methodology

ID31

BM32

ID21

BM16

ID15

ID13

ID11

ID10

1^st development phase beamlines

Text

Real experiments with users: MX, ID10, ID11, ID13, ID31

State of BLISS (november 2018)

parallel HDF5 data writing

PEPU, Speedgoat

scans tree

data selection, statistics

features list, HKL Conda package

Counter aliases, counter modes

Web application

scanning engine, counters

BCU documentation writing event

Bugs hunting

Technical debt, QA

UX improvement

Changes: new ideas, failed ideas

Many controllers to integrate (motors, devices, etc.)

Training (for ESRF staff + users...)

Lessons learned (so far...)

Commissioning time with beam + scientists is invaluable

Human adventure: anticipation, transparency, help with transition... Mutual understanding

As much as possible, aim for complete transitions: no legacy system running along with the new system

One single git repository is much easier to work with compared to several sub-projects in separate repositories

Kanban helps reducing pressure on development team: happier people, better code ; importance of "Analysis" step

Conclusion

Spec has reached end-of-life at ESRF

BLISS is being designed for the needs of the EBS beamlines,

and already running experiments

Half of the beamlines will be converted during the EBS shutdown (2019-2020)

Would be happy to start collaborations around BLISS with interested people

Acknowledgements

BLISS core development team

E. Papillon

V. Michel*

J. Bodera

C. Guilloud

M. Guijarro

T. Coutinho*

S. Petitdemange

+ P. Guillou, P. Pancino, L. Pithan

BCU engineers

A. Beteva, G. Berruyer, L. Claustre, M-C. Dominguez*, M-C. Lagier, A. Homs, R. Homs, A. Mauro, S. Ohlsson, M. Perez*, F. Sever, H. Witsch

head of BCU: J. Meyer, head of Software Group: A. Gotz

BLISS demo