From Spec to Python:

BLISS for ESRF EBS

presented by Matias Guijarro - BLISS team

 

Beamline Control Unit / Software Group

 

ESRF, Grenoble, France

Beamline Control Automation & Data Acquisition Workshop - 17th January, 2019

ESRF Extremely Brilliant Source (EBS) project

4th generation light source

150 M€ investment (2015-2022)

100x better

X-ray beams

New

state-of-the-art

beamlines

Data as a Service strategy

From Spec to Python

Spec has been driving ESRF experiments for 26 years

                   Welcome to "spec" Release 6.03.11
         Copyright (c) 1987-2016 Certified Scientific Software
                          All rights reserved

 (Portions derived from a program developed at Harvard University.)
 (Linked with BSD libedit library for command line editing.)

 Using "/bliss/users/blissadm/spec/redhate4/spec.d"
  for auxiliary file directory (SPECD).

 Getting configuration parameters from "SPECD/matias/config".

 =
 Type h changes for info on latest changes.
 Browse to http://www.certif.com for complete documentation.
 =

 Reading file "SPECD/standard.mac".
 Warning:  No open data file.  Using "/dev/null".
 Type "startup" to initialize data file, etc.

 Reading file "SPECD/site.mac".
 Reading file "/bliss/users/blissadm/local/spec/macros/ID00setup.mac" (level 2).
 Reading file "/bliss/users/blissadm/spec/macros/lock.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlocal.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlist.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/pseudo.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/tw.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/joy.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/cplotarray.mac" (level 3).

 Doing SETUP.
 Reading file "SPECD/setup".
 Reading file "SPECD/matias/setup".

 1.MATIAS> 

Poor macro language

No continuous scan framework

Limited data management

Simplistic configuration

No extensibility

Lots of workarounds

High operating cost

Revolution !

BeamLine Instrumentation Support Software

Welcome,

BLISS project goals

State-of-the-art beamline control

Continuous scans, Trajectories,

Data management

Integrated environment

Command Line Interface, Configuration, Data display

Ready for new challenges

Online data analysis, live feedback, full-featured programming language

BLISS philosophy

Integrated beamline control system

  • Software library
  • Command Line Interface
  • Configuration application
  • Online visualization

 

Python everywhere

command line, user sequences, etc.

 

Generic scan engine

Continuous, step-by-step, multiple masters

 

Data management, decoupling acquisition and archiving

BLISS project development tools

Issues list, sprints reviews, discussions and tasks to do on ESRF-hosted gitlab

Continuous Integration:

gitlab-ci

Deployment:

Documentation:

MkDocs

Code formatting: Black

All I/O based on gevent

cooperative multi-tasking

Direct hardware control

Distributed

objects state

Persistent

settings cache

 Data publishing,

Transient storage

Scan acquisition chain, represented as a tree

BLISS design principles

online data analysis

data visualisation

data archiving

2D detector

Beacon server

Beacon static configuration service

BLISS modular architecture

1

2

3

user sequences

4

Beacon static configuration service

Web interface for configuration editing

Beacon server

.yml

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment,

Python setup file

Can replace TANGO DB

Conversion script provided

Beacon server

configuration

1

Beacon static configuration service

Beacon, example configuration file

sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│   ├── diode.yml
│   ├── __init__.yml
│   └── motors
│       ├── bv.yml
│       ├── DtoX.yml
│       ├── __init__.yml
│       ├── md2.yml
│       ├── mirror1.yml
│       ├── slits.yml
│       └── table.yml
├── oh
│   ├── bpm.yml
│   ├── __init__.yml
│   ├── motors
│   │   ├── bv.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── slits.yml
│   │   └── transfocators.yml
│   └── wagos.yml
└── sessions
    ├── id232_setup.py
    ├── id232.yml
    └── __init__.yml
- controller:
    class: IcePAP
    host: iceid2322
    axes:
    - name: mbv4mot
      address: 1
      steps_per_unit: 817
      velocity: 0.3
      acceleration: 3

example configuration of a beam viewer motor

1

Beacon static configuration service

Web interface for configuration editing

Beacon server

.yml

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment,

Python setup file

Can replace TANGO DB

Conversion script provided

Beacon server

configuration

+ services based on

Message broker: state sharing,

distributed lock

Transient data store

Persistent settings cache

1

Beacon static configuration service

BLISS scan framework

synchronization

low latency

flexibility

2

Beacon static configuration service

BLISS scan framework

Data writer

  • HDF5

synchronization

low latency

flexibility

Acquisition chain

  • a tree with master & slave nodes
  • master triggers data acquisition
  • slave takes data

AcquisitionMaster, AcquisitionDevice

  • wrappers around BLISS control objects

2

BLISS scan example

Continuous scan with a motor triggering MCA and 2D detector acquisition, while a timer triggers diode readings

2

"Classic" step-by-step scans

2

  • Directly available as functions from 'bliss.common.standard'
    • Example: ascan(axis, start, stop, npoints, count_time, *counters)
  • Default acquisition chain
  • Use the same underlying framework as continuous scans

BLISS online data publishing

While a scan is running, data is published to the redis database provided by Beacon

  • scalar values are stored directly

  • bigger data (images, spectra) is just referenced

  • configurable time to live (TTL)

Any external process can access redis data to perform online data analysis or live feedback for example

3

BLISS scan data flow

3

Writing data acquisition sequences

from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import numpy # I know you dreamt of it
import gevent

def set_detector_cover(in):
    wcidxx.set('detcover', in)
 
    # 5 seconds timeout waiting for detector cover to move
    with gevent.Timeout(5):
        while wcidxx.get('detcover_in') == in:
          time.sleep(0.1)

def my_super_experiment(name):
    safety_shutter.open()

    old_att = attenuators.get()

    def restore_beamline():
        set_detcover_open(False)
        attenuators.set(old_att)

    with cleanup(safety_shutter.close):
        with error_cleanup(restore_beamline):
            attenuators.set(50)
            set_detcover_open(True)

            SCAN_SAVING.name = name
            MEASUREMENT_GROUP.enable('diode')
            
            data_node = dscan(m0, -5, 5, 10, 0.1)
            
            for data in data_node.walk_data():
              # do something useful with data...
with cleanup(safety_shutter.close): # cleanup is always called at the end
    with error_cleanup(restore_beamline):  # only called in case of error
        ...

Use of Python context managers for cleanup

Normal Python functions

Easy timeouts with gevent.Timeout

# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
    while wcidxx.get('detcover_in') == in:
        time.sleep(0.1)

4

BLISS development phases (2017-2019)

1. Coding sprints with different "product owners"

2. Assessing the system on different beamlines before the EBS shutdown, real-world experiments

3. Deployment on 15 (20?) beamlines during EBS shutdown

We are here 

Software Development

Kanban methodology

Full transitions, no more Spec on target beamlines

Scrum methodology

ID31

BM32

ID21

BM16

ID15

ID13

ID11

ID10

1st development phase beamlines

MX

Text

Real experiments with users: MX, ID10, ID11, ID13, ID31

State of BLISS (november 2018)

parallel HDF5 data writing

PEPU, Speedgoat

scans tree

data selection, statistics

features list, HKL Conda package

Counter aliases, counter modes

Web application

scanning engine, counters

BCU documentation writing event

Bugs hunting

Technical debt, QA

UX improvement

Changes: new ideas, failed ideas

Many controllers to integrate (motors, devices, etc.)

Training (for ESRF staff + users...)

Lessons learned (so far...)

Commissioning time with beam + scientists is invaluable

Human adventure: anticipation, transparency, help with transition... Mutual understanding

As much as possible, aim for complete transitions: no legacy system running along with the new system

One single git repository is much easier to work with compared to several sub-projects in separate repositories

Kanban helps reducing pressure on development team: happier people, better code ; importance of "Analysis" step

Conclusion

Spec has reached end-of-life at ESRF

BLISS is being designed for the needs of the EBS beamlines,

and already running experiments

Half of the beamlines will be converted during the EBS shutdown (2019-2020)

Would be happy to start collaborations around BLISS with interested people

Acknowledgements

BLISS core development team

E. Papillon

V. Michel*

J. Bodera

C. Guilloud

M. Guijarro

T. Coutinho*

S. Petitdemange

+ P. Guillou, P. Pancino, L. Pithan

BCU engineers

A. Beteva, G. Berruyer, L. Claustre, M-C. Dominguez*, M-C. Lagier, A. Homs, R. Homs, A. Mauro, S. Ohlsson, M. Perez*, F. Sever, H. Witsch  

head of BCU: J. Meyer, head of Software Group: A. Gotz

BLISS demo

From spec to Python, BLISS for EBS @ ESRF

By Matias Guijarro

From spec to Python, BLISS for EBS @ ESRF

  • 197
Loading comments...

More from Matias Guijarro