From Spec to Python:
BLISS for ESRF EBS
presented by Matias Guijarro - BLISS team
Beamline Control Unit / Software Group
ESRF, Grenoble, France
Beamline Control Automation & Data Acquisition Workshop - 17th January, 2019
ESRF Extremely Brilliant Source (EBS) project
4th generation light source
150 M€ investment (2015-2022)
100x better
X-ray beams
New
state-of-the-art
beamlines
Data as a Service strategy
From Spec to Python
Spec has been driving ESRF experiments for 26 years
Welcome to "spec" Release 6.03.11
Copyright (c) 1987-2016 Certified Scientific Software
All rights reserved
(Portions derived from a program developed at Harvard University.)
(Linked with BSD libedit library for command line editing.)
Using "/bliss/users/blissadm/spec/redhate4/spec.d"
for auxiliary file directory (SPECD).
Getting configuration parameters from "SPECD/matias/config".
=
Type h changes for info on latest changes.
Browse to http://www.certif.com for complete documentation.
=
Reading file "SPECD/standard.mac".
Warning: No open data file. Using "/dev/null".
Type "startup" to initialize data file, etc.
Reading file "SPECD/site.mac".
Reading file "/bliss/users/blissadm/local/spec/macros/ID00setup.mac" (level 2).
Reading file "/bliss/users/blissadm/spec/macros/lock.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/stlocal.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/stlist.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/pseudo.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/tw.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/joy.mac" (level 3).
Reading file "/bliss/users/blissadm/spec/macros/cplotarray.mac" (level 3).
Doing SETUP.
Reading file "SPECD/setup".
Reading file "SPECD/matias/setup".
1.MATIAS>
Poor macro language
No continuous scan framework
Limited data management
Simplistic configuration
No extensibility
Lots of workarounds
High operating cost
Revolution !
BeamLine Instrumentation Support Software
Welcome,
BLISS project goals
State-of-the-art beamline control
Continuous scans, Trajectories,
Data management
Integrated environment
Command Line Interface, Configuration, Data display
Ready for new challenges
Online data analysis, live feedback, full-featured programming language
BLISS philosophy
Integrated beamline control system
- Software library
- Command Line Interface
- Configuration application
- Online visualization
Python everywhere
command line, user sequences, etc.
Generic scan engine
Continuous, step-by-step, multiple masters
Data management, decoupling acquisition and archiving
BLISS project development tools
Issues list, sprints reviews, discussions and tasks to do on ESRF-hosted gitlab
Continuous Integration:
gitlab-ci
Deployment:
Documentation:
MkDocs
Code formatting: Black
All I/O based on gevent
cooperative multi-tasking
Direct hardware control
Distributed
objects state
Persistent
settings cache
Data publishing,
Transient storage
Scan acquisition chain, represented as a tree
BLISS design principles
online data analysis
data visualisation
data archiving
2D detector
Beacon server
Beacon static configuration service
BLISS modular architecture
1
2
3
user sequences
4
Beacon static configuration service
Web interface for configuration editing
Beacon server
.yml
Devices & sequences configuration in YAML format
User sessions to group beamline devices for an experiment,
Python setup file
Can replace TANGO DB
Conversion script provided
Beacon server
configuration
1
Beacon static configuration service
Beacon, example configuration file
sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│ ├── diode.yml
│ ├── __init__.yml
│ └── motors
│ ├── bv.yml
│ ├── DtoX.yml
│ ├── __init__.yml
│ ├── md2.yml
│ ├── mirror1.yml
│ ├── slits.yml
│ └── table.yml
├── oh
│ ├── bpm.yml
│ ├── __init__.yml
│ ├── motors
│ │ ├── bv.yml
│ │ ├── __init__.yml
│ │ ├── mono.yml
│ │ ├── slits.yml
│ │ └── transfocators.yml
│ └── wagos.yml
└── sessions
├── id232_setup.py
├── id232.yml
└── __init__.yml- controller:
class: IcePAP
host: iceid2322
axes:
- name: mbv4mot
address: 1
steps_per_unit: 817
velocity: 0.3
acceleration: 3
example configuration of a beam viewer motor
1
Beacon static configuration service
Web interface for configuration editing
Beacon server
.yml
Devices & sequences configuration in YAML format
User sessions to group beamline devices for an experiment,
Python setup file
Can replace TANGO DB
Conversion script provided
Beacon server
configuration
+ services based on
Message broker: state sharing,
distributed lock
Transient data store
Persistent settings cache
1
Beacon static configuration service
BLISS scan framework
synchronization
low latency
flexibility
2
Beacon static configuration service
BLISS scan framework
Data writer
- HDF5
synchronization
low latency
flexibility
Acquisition chain
- a tree with master & slave nodes
- master triggers data acquisition
- slave takes data
AcquisitionMaster, AcquisitionDevice
- wrappers around BLISS control objects
2
BLISS scan example
Continuous scan with a motor triggering MCA and 2D detector acquisition, while a timer triggers diode readings
2
"Classic" step-by-step scans
2
-
Directly available as functions from 'bliss.common.standard'
- Example: ascan(axis, start, stop, npoints, count_time, *counters)
- Default acquisition chain
- Use the same underlying framework as continuous scans
BLISS online data publishing
While a scan is running, data is published to the redis database provided by Beacon
-
scalar values are stored directly
-
bigger data (images, spectra) is just referenced
-
configurable time to live (TTL)
Any external process can access redis data to perform online data analysis or live feedback for example
3
BLISS scan data flow
3
Writing data acquisition sequences
from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import numpy # I know you dreamt of it
import gevent
def set_detector_cover(in):
wcidxx.set('detcover', in)
# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
while wcidxx.get('detcover_in') == in:
time.sleep(0.1)
def my_super_experiment(name):
safety_shutter.open()
old_att = attenuators.get()
def restore_beamline():
set_detcover_open(False)
attenuators.set(old_att)
with cleanup(safety_shutter.close):
with error_cleanup(restore_beamline):
attenuators.set(50)
set_detcover_open(True)
SCAN_SAVING.name = name
MEASUREMENT_GROUP.enable('diode')
data_node = dscan(m0, -5, 5, 10, 0.1)
for data in data_node.walk_data():
# do something useful with data...with cleanup(safety_shutter.close): # cleanup is always called at the end
with error_cleanup(restore_beamline): # only called in case of error
...Use of Python context managers for cleanup
Normal Python functions
Easy timeouts with gevent.Timeout
# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
while wcidxx.get('detcover_in') == in:
time.sleep(0.1)
4
BLISS development phases (2017-2019)
1. Coding sprints with different "product owners"
2. Assessing the system on different beamlines before the EBS shutdown, real-world experiments
3. Deployment on 15 (20?) beamlines during EBS shutdown
We are here
Software Development
Kanban methodology
Full transitions, no more Spec on target beamlines
Scrum methodology
ID31
BM32
ID21
BM16
ID15
ID13
ID11
ID10
1st development phase beamlines
MX
Text
Real experiments with users: MX, ID10, ID11, ID13, ID31
State of BLISS (november 2018)
parallel HDF5 data writing
PEPU, Speedgoat
scans tree
data selection, statistics
features list, HKL Conda package
Counter aliases, counter modes
Web application
scanning engine, counters
BCU documentation writing event
Bugs hunting
Technical debt, QA
UX improvement
Changes: new ideas, failed ideas
Many controllers to integrate (motors, devices, etc.)
Training (for ESRF staff + users...)
Lessons learned (so far...)
Commissioning time with beam + scientists is invaluable
Human adventure: anticipation, transparency, help with transition... Mutual understanding
As much as possible, aim for complete transitions: no legacy system running along with the new system
One single git repository is much easier to work with compared to several sub-projects in separate repositories
Kanban helps reducing pressure on development team: happier people, better code ; importance of "Analysis" step
Conclusion
Spec has reached end-of-life at ESRF
BLISS is being designed for the needs of the EBS beamlines,
and already running experiments
Half of the beamlines will be converted during the EBS shutdown (2019-2020)
Would be happy to start collaborations around BLISS with interested people
Acknowledgements
BLISS core development team
E. Papillon
V. Michel*
J. Bodera
C. Guilloud
M. Guijarro
T. Coutinho*
S. Petitdemange
+ P. Guillou, P. Pancino, L. Pithan
BCU engineers
A. Beteva, G. Berruyer, L. Claustre, M-C. Dominguez*, M-C. Lagier, A. Homs, R. Homs, A. Mauro, S. Ohlsson, M. Perez*, F. Sever, H. Witsch
head of BCU: J. Meyer, head of Software Group: A. Gotz
BLISS demo
From spec to Python, BLISS for EBS @ ESRF
By Matias Guijarro
