Experiments control
of EBS beamlines
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1413529/ESRF-Logo-RGB.png)
CBS day out, 24 March 2017 - Château de la Baume, Seyssins (38), France
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1413514/avatar.png)
presented by Matias Guijarro, Beamline Control Unit, Software Group, ESRF
"A new sequencer for the beamlines will be developed to replace SPEC.
[...] The new sequencer will be based on Python. [...]
The new sequencer will rewrite the experiment macros so that
they are easier to maintain."
(excerpt from the ESRF Phase II TDS Orange Book)
SPEC
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3628199/darthvader.jpeg)
The good in SPEC
Direct control of devices
- (much) easier to debug / understand
- reconfig = reset
- no additional data transfers
Integrated tool
- plots display
- configuration
The dark side
Poor macro language, no extensibility
Single task operation
Exclusive control
the BLISS project
BeamLine Instrumentation Support Software
- project repository: gitlab.esrf.fr/bliss/bliss
- started in December, 2014
- 2700+ commits
- more than 50.000 lines of Python code
- 4 core developers, 10 participants
made with by BCU
BLISS project overview
Key concepts
- Python library + tools
- Modular architecture
- Synchronous API on top of evented I/O for cooperative multi-tasking
- Direct HW control from 1 BLISS process at a time, support for several peers via state sharing
- Centralized configuration
- Control objects
- Master/Slave Tree-based acquisition chains
- Data management
- Sequences as genuine Python functions
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3623949/stairwaytoheaven.jpg)
Multi-tasking + synchronous API
Python library with modular architecture
Configuration
Tools / User interfaces
Scans
User sequences
Data management
The path to BLISS
(bliss) sybil:~ % python
Python 2.7.9 (default, Mar 1 2015, 12:57:24)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from bliss.common.axis import Axis
>>> from bliss.controllers.motors import IcePAP
>>> iceid2322 = IcePAP.IcePAP("iceid2322",
{"host": "iceid2322"},
[("mbv4mot", Axis, { "address": 1,
"steps_per_unit": 80,
"velocity": 125,
"acceleration": 500
}
)],
[])
>>> iceid2322.initialize()
>>> iceid2322.axes
{'mbv4mot': <bliss.common.axis.Axis object at 0x7fdb0defb410>}
>>> m = iceid2322.get_axis("mbv4mot")
>>> m.velocity()
125.0
>>> m.acceleration()
500.0
>>> m.position()
252.23750000000001
>>>
Multitasking ?
Evented I/O ?
Synchronous API ?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1717563/225px-Dexter2.png)
Experiment Control & Data Acquisition
heavily rely on Input/Output (IO) operations
run
run
run
run
Task execution
I/O system call
- For I/O, tasks must wait (sleep)
- Behind the scene, the underlying system will carry the I/O operation and wake up the task when buffers are ready
Task execution vs. I/O system call
Evented I/O is an operating system feature that allows to not block during I/O calls
Evented I/O
- System will fire READ or WRITE events when buffers are ready, so I/O can be performed in no time
- Multi-tasking can be achieved by switching tasks between read or write events
Task A
run
run
Task B
run
run
gevent Python library offers a "task" abstraction,
and uses evented I/O for system calls
- gevent adds a synchronous API on top of evented I/O
- Python library is patched to be gevent-friendly
- Task context switch becomes transparent
gevent synchronous API
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1717576/dexter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3104795/wizard.jpeg)
Effective multi-tasking without
too much worries
Centralized Configuration:
introducing Beacon
Beacon, brilliant Beamline Configuration
- Client / server architecture
- Set of YAML files (.yml) containing objects configuration
- Services built on top of
- Web application : user interface for configuration editing
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1417084/redis-white.png)
Beacon: example configuration from ID23-2
sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│ ├── diode.yml
│ ├── __init__.yml
│ └── motors
│ ├── bv.yml
│ ├── DtoX.yml
│ ├── __init__.yml
│ ├── md2.yml
│ ├── mirror1.yml
│ ├── slits.yml
│ └── table.yml
├── oh
│ ├── bpm.yml
│ ├── __init__.yml
│ ├── motors
│ │ ├── bv.yml
│ │ ├── __init__.yml
│ │ ├── mono.yml
│ │ ├── slits.yml
│ │ └── transfocators.yml
│ └── wagos.yml
└── sessions
├── id232_setup.py
├── id232.yml
└── __init__.yml
- controller:
class: IcePAP
host: iceid2322
axes:
- name: mbv4mot
address: 1
steps_per_unit: 817
velocity: 0.3
acceleration: 3
- name: wcid232a
class: wago
controller_ip: wcid232a
counter_gain_names: wbd_g
counter_names: wbd
mapping:
-
type: 750-516
logical_names: wbd_g,wbd_g,wbd_g,_
-
type: 750-630
logical_names: psenc
-
type: 750-630
logical_names: psenc
-
type: 750-630
logical_names: wbvenc
-
type: 750-513
logical_names: psrelay, _
-
type: 750-467
logical_names: wbd, _
-
type: 750-469
logical_names: pst,pst
-
type: 750-469
logical_names: pst,pst
-
type: 750-469
logical_names: pst,pst
Beacon: web application
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3621771/beacon-config.png)
Beacon: Sessions
- Sessions allow to group control objects under a logical entity
- the same object can be present in several sessions
- by default, a session contains all objects; otherwise one can define 'config-objects' or 'exclude-objects'
- a setup file can be specified, to be executed after session is loaded
class: Session
name: eh3
setup-file: ./eh3.py
config-objects:
- aerox
- aeroy
- musst
- pcoedge
- measurement_eh3
- multiplexer_eh3
class: Session
name: id29
default: True
setup-file: ./id29_setup.py
exclude-objects: pmbpress pmb
example from id29
example from id15
BLISS tools
Command Line Interface
Based on ptpython
sybil:~ % bliss
>>> config.names_list
['ss4hg', 'ss4ho', 'ss2vg', 'bm2v1', 'wbvmot', 'ss2vo', 'trot', 'm2theta', 'thetaenc', 'mbv2mot', 'tf1m4', 'psb', 'm1theta',
'psf', 'psd', 'mbv3mot', 't2ry', 't2rz', 'tyrot', 'bm2v2', 'psu', 'wbv', 't1rz', 't1ry', 'tf1m1', 'tf1m3', 'tf1m2', 'sampx',
'sampy', 'ss2hg', 'mbv1mot', 'ss2ho', 'diode', 't2y', 't2z', 'bm1v1enc', 'psvo', 'm1ty', 'theta', 'm2ty', 'psvg', 'thgt',
'ss4vo', 'tf2m4', 'tf2m2', 'tf2m3', 'tf2m1', 'ts1vg', 'ts1vo', 'ss4u', 'ss4b', 'ss4f', 't1y', 't1z', 'mbv2d', 'txrot', 'ss4vg',
'mbv4mot', 'ts1hg', 'ts1ho', 'ss2b', 'wcid232a', 'ss2d', 'ss2f', 'mbv3d', 'ttrans', 'ss2u', 'pshg', 'bm1v2', 'bm1v1', 'psho',
'ss1b', 'ss1d', 'ss1f', 'ss1u', 'ss1ho', 'ss1hg', 'tz1', 'tz3', 'tz2', 'DtoX', 'tyf', 'tyb', 'mbv1d', 'id23-2', 'ss1vg',
'ss4d', 'ss1vo', 'phiz', 'phiy', 'phix']
>>> config.get("mbv4mot")
<bliss.common.axis.Axis object at 0x7fc279b72fd0>
>>> mbv4mot = config.get("mbv4mot")
>>> mbv4mot.position()
2.0
>>>
[F4] Emacs 15/15 [F3] History [F6] Paste mode [F2] Menu - CPython 2.7.9
Command Line Interface
Session example
- from ID23-2
sybil:~ % bliss --show-sessions
Session name(s):
id23-2
(bliss) sybil:~ % bliss -s id23-2
Initializing 'ss4hg`
Initializing 'ss4ho`
Initializing 'ss2vg`
Initializing 'bm2v1`
Initializing 'wbvmot`
Initializing 'ss2vo`
Initializing 'trot`
Initializing 'm2theta`
Initializing 'thetaenc`
Initializing 'mbv2mot`
Initializing 'tf1m4`
Initializing 'psb`
Initializing 'm1theta`
Initializing 'psf`
Initializing 'psd`
Initializing 'mbv3mot`
Initializing 't2ry`
Initializing 't2rz`
Initializing 'tyrot`
Initializing 'bm2v2`
Initializing 'psu`
Initializing 'wbv`
Initializing 't1rz`
Initializing 't1ry`
Initializing 'tf1m1`
Initializing 'tf1m3`
Initializing 'tf1m2`
Initializing 'sampx`
Initializing 'sampy`
Initializing 'ss2hg`
Initializing 'mbv1mot`
Initializing 'ss2ho`
Initializing 'diode`
Initializing 't2y`
Initializing 't2z`
Initializing 'bm1v1enc`
Initializing 'psvo`
Initializing 'm1ty`
Initializing 'theta`
Initializing 'm2ty`
Initializing 'psvg`
Initializing 'thgt`
Initializing 'ss4vo`
Initializing 'tf2m4`
Initializing 'tf2m2`
Initializing 'tf2m3`
Initializing 'tf2m1`
Initializing 'ts1vg`
Initializing 'ts1vo`
Initializing 'ss4u`
Initializing 'ss4b`
Initializing 'ss4f`
Initializing 't1y`
Initializing 't1z`
Initializing 'mbv2d`
Initializing 'txrot`
Initializing 'ss4vg`
Initializing 'mbv4mot`
Initializing 'ts1hg`
Initializing 'ts1ho`
Initializing 'ss2b`
Initializing 'wcid232a`
Initializing 'ss2d`
Initializing 'ss2f`
Initializing 'mbv3d`
Initializing 'ttrans`
Initializing 'ss2u`
Initializing 'pshg`
Initializing 'bm1v2`
Initializing 'bm1v1`
Initializing 'psho`
Initializing 'ss1b`
Initializing 'ss1d`
Initializing 'ss1f`
Initializing 'ss1u`
Initializing 'ss1ho`
Initializing 'ss1hg`
Initializing 'tz1`
Initializing 'tz3`
Initializing 'tz2`
Initializing 'DtoX`
Initializing 'tyf`
Initializing 'tyb`
Initializing 'mbv1d`
Initializing 'id23-2`
Initializing 'ss1vg`
Initializing 'ss4d`
Initializing 'ss1vo`
Initializing 'phiz`
Initializing 'phiy`
Initializing 'phix`
Done.
>>>
[F4] Emacs 90/90 [F3] History [F6] Paste mode [F2] Menu - CPython 2.7.9
Web Application Interface
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3626160/id30b.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3628317/BlissGUI_id30.jpg)
BLISS scans
sybil:~ % bliss
>>> from bliss.scanning.chain import AcquisitionChain
>>> from bliss.scanning.acquisition.motor import SoftwarePositionTriggerMaster
>>> from bliss.scanning.acquisition.lima import LimaAcquisitionDevice
>>> from PyTango.gevent import DeviceProxy
>>> m0 = config.get("m0")
>>> lima_dev = DeviceProxy("id30a3/limaccd/simulation")
>>> chain = AcquisitionChain()
>>> chain.add(SoftwarePositionTriggerMaster(m0, start=5, end=10, npoints=10, time=5),
LimaAcquisitionDevice(lima_dev, acq_nb_frames=5, acq_expo_time=0.03, acq_trigger_mode="INTERNAL_TRIGGER_MULTI"))
Continuous scan example
5
10
m0 position
m0 speed
detector frame triggering
1. Building the Acquisition Chain
m0: master
Lima device: slave
>>> SCAN_SAVING.template = '/data/id23eh2/inhouse/{date}/{sample}'
>>> SCAN_SAVING.sample = 'HAK1234'
>>> SCAN_SAVING.get_path()
"/data/id23eh2/inhouse/20170324/HAK1234"
Continuous scan example
2. Setting up where to save data
>>> from bliss.scanning.scan import Scan
>>> my_continuous_scan = Scan(chain)
>>> my_continuous_scan.prepare()
>>> my_continuous_scan.start()
3. Running scan
Scan can take additional arguments
- scan_info : dictionary with scan metadata
- writer : specifies how data has to be written ; defaults to bliss.scanning.writer.hdf5.HDF5Writer - setting to None means data is not saved
- scan_data_callback : in-process fast path to data being acquired, receives data events
3'. Running scan in background
>>> from bliss.scanning.scan import Scan
>>> my_continuous_scan = Scan(chain)
>>> my_continuous_scan.prepare()
>>> background_scan = gevent.spawn(my_continuous_scan.start)
>>> background_scan.ready() # is it finished ?
>>> background_scan.kill() # interrupt
Generic step-by-step scans
ascan, dscan, a2scan, d2scan, timescan
- functions from bliss.common.standard
- API example: ascan(axis, start, stop, npoints, count_time, *counters)
- uses the same underlying scan framework
chain = AcquisitionChain()
timer = SoftwareTimerMaster(count_time, npoints=npoints)
if not counters:
for cnt in MEASUREMENT_GROUP.enabled:
chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))
else:
for cnt in counters:
chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))
chain.add(LinearStepTriggerMaster(motor, start, end, npoints), timer)
scan = Scan(chain)
scan.prepare()
scan.start()
...
Measurement Groups
Mainly for 'ct' (counting for a period of time) or generic scans
- by default, the current active measurement group is used
MeasurementGroup objects
- only one active at a time, MEASUREMENT_GROUP global
- defined in configuration (.yml file)
- ability to enable/disable counters, and to save state for later
- default measurement group contains all counters
Data Management
Data Management
First-class citizen
- built in the scan framework from the ground up
How is it done ?
- flexible template for scan data base path (SCAN_SAVING)
- hierarchical organization of data, mirroring the Acquisition Chain tree
- Online data publishing
Hierarchical organization of data
Mirroring of the Acquisition Chain tree
- each device in the chain has a name, e.g. axis name for motors or 'timer' for Timer objects
- each device can define AcquisitionChannel objects, e.g. counters for a P201 card or image for a Lima camera
- Acquisition channels must have a name, a type and a shape
- scan_info dictionary ({ key: value, ... }) for metadata
Well-defined model for converting scan data into nicely organized directories and files
Online data publishing
Based on
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1417084/redis-white.png)
As data acquisition (scan) is running, data is published to the redis database provided by Beacon
- scalar values are stored directly
- bigger data (images, spectra) is just referenced
- data is kept in Redis for 1 day by default
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1964817/beacon_iconic.png)
Online data publishing
Any external process can read data from Redis
- using standard Redis tools or libraries for any language
- BLISS provides a data iterator
>>> from bliss.data.node import get_node
>>> session_scans = get_node("my_session_name")
>>> def analyse_data(node):
... for data in node.iterator().walk_data():
... <do something useful with data>
>>> for node in session_scans.iterator().walk():
... analyse_data(node)
A perfect fit with silx-based data processing
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3628032/silx_small.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1413529/ESRF-Logo-RGB.png)
User Sequences
Python functions as sequences
from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import time, numpy # I know you dreamt of it
import gevent
def set_detector_cover(in):
wcidxx.set('detcover', in)
# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
while wcidxx.get('detcover_in') == in:
time.sleep(0.1)
def my_super_experiment(name):
safety_shutter.open()
old_att = attenuators.get()
def restore_beamline():
set_detcover_open(False)
attenuators.set(old_att)
# cleanup is always called at the end
with cleanup(safety_shutter.close):
# this will only be called in case of error
with error_cleanup(restore_beamline):
attenuators.set(50)
set_detcover_open(True)
SCAN_SAVING.name = name
MEASUREMENT_GROUP.enable('diode')
data_node = dscan(m0, -5, 5, 10, 0.1)
for data in data_node.walk_data():
# do something useful with data...
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/1720106/under_construction.png)
Conclusion
In the next months...
Scan framework being finalized (only a few weeks left !)
MCAs
New controllers to be added
- Eurotherm, Lakeshore
- future P201 replacement card
- N354 bench clock card
- BCDU8
- ...
KB alignment functions
Beamlines already using BLISS today
MX (ID29, ID30A-1, ID30A-3, ID30B, ID23-1, ID23-2):
80% SPEC free - ID23-1 not migrated yet, will be done soon
ID31, ID15: ongoing migration
ID28 side station: partially done (only for newly installed experiment)
ID16, ID09: only for some motors control and P201, Keithley
Careful deployment, development in parallel: slow progress, to ensure experiments stability
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/2793638/warning.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/331850/images/3628405/unclesam.png)
... but not all at once
Thanks for your attention !
Feel free to ask questions
bliss@cbs_day_out
By Matias Guijarro
bliss@cbs_day_out
- 1,040