Experiments control

of EBS beamlines

MEX day out, 2nd of June 2017 - Château de la Commanderie, Eybens (38), France

presented by Matias Guijarro, Beamline Control Unit, Software Group, ESRF

"A new sequencer for the beamlines will be developed to replace SPEC.

[...] The new sequencer will be based on Python. [...]

The new sequencer will rewrite the experiment macros so that

they are easier to maintain."

(excerpt from the ESRF Phase II TDS Orange Book)

the BLISS project

BeamLine Instrumentation Support Software

  • project repository: gitlab.esrf.fr/bliss/bliss
  • started in December, 2014
  • 3000+ commits
  • more than 50.000 lines of Python code
  • 4 core developers, 10 participants

made with        by BCU

BLISS project overview

Key concepts

  • Python library + tools
  • Modular architecture
  • Cooperative multi-tasking with a synchronous API
  • Centralized configuration
  • Direct hardware control
  • Control objects
  • Master/Slave Tree-based acquisition chains
  • Data management
  • Sequences as genuine Python functions

Multi-tasking + synchronous API

Python library with modular architecture

Configuration

Tools / User interfaces

Scans

User sequences

Data management

The path to BLISS:

talk outline

Hardware access and state sharing


(bliss) sybil:~ % python
Python 2.7.9 (default, Mar  1 2015, 12:57:24)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from bliss.common.axis import Axis
>>> from bliss.controllers.motors import IcePAP
>>> iceid2322 = IcePAP.IcePAP("iceid2322",
                              {"host": "iceid2322"},
                              [("mbv4mot", Axis, { "address": 1, 
                                                   "steps_per_unit": 80,
                                                   "velocity": 125,
                                                   "acceleration": 500
                                              }
                              )],
                              [])
>>> iceid2322.initialize()
>>> iceid2322.axes
{'mbv4mot': <bliss.common.axis.Axis object at 0x7fdb0defb410>}
>>> m = iceid2322.get_axis("mbv4mot")
>>> m.velocity()
125.0
>>> m.acceleration()
500.0
>>> m.position()
252.23750000000001
>>>

 

Multitasking ?

Synchronous API ?

Experiment Control & Data Acquisition

heavily rely on Input/Output (IO) operations

run

run

run

run

Task execution

I/O system call

  • For I/O, tasks must wait (sleep)
  • Behind the scene, the underlying system will carry the I/O operation and wake up the task when buffers are ready

Task execution vs. I/O system call

Evented I/O is an operating system feature that allows to not block during I/O calls

Evented I/O

  • System will fire READ or WRITE events when buffers are ready, so I/O can be performed in no time
  • Multi-tasking can be achieved by switching tasks between read or write events

Task A

run

run

Task B

run

run

gevent Python library offers a "task" abstraction,

and uses evented I/O for system calls

  • gevent adds a synchronous API on top of evented I/O
  • Python library is patched to be gevent-friendly
  • Task context switch becomes transparent

gevent synchronous API

Effective multi-tasking without

too much worries

Centralized Configuration:

introducing Beacon

Beacon, brilliant Beamline Configuration

  • Client / server architecture
  • Set of YAML files (.yml) containing objects configuration
  • Services built on top of
  • Web application : user interface for configuration editing

Beacon: example configuration from ID23-2

sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│   ├── diode.yml
│   ├── __init__.yml
│   └── motors
│       ├── bv.yml
│       ├── DtoX.yml
│       ├── __init__.yml
│       ├── md2.yml
│       ├── mirror1.yml
│       ├── slits.yml
│       └── table.yml
├── oh
│   ├── bpm.yml
│   ├── __init__.yml
│   ├── motors
│   │   ├── bv.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── slits.yml
│   │   └── transfocators.yml
│   └── wagos.yml
└── sessions
    ├── id232_setup.py
    ├── id232.yml
    └── __init__.yml
- controller:
    class: IcePAP
    host: iceid2322
    axes:
    - name: mbv4mot
      address: 1
      steps_per_unit: 817
      velocity: 0.3
      acceleration: 3

- name: wcid232a
  class: wago
  controller_ip: wcid232a
  counter_gain_names: wbd_g
  counter_names: wbd
  mapping:
    -
        type: 750-516
        logical_names: wbd_g,wbd_g,wbd_g,_
    -
        type: 750-630
        logical_names: psenc
    -
        type: 750-630
        logical_names: psenc
    -
        type: 750-630
        logical_names: wbvenc
    -
        type: 750-513
        logical_names: psrelay, _
    -
        type: 750-467
        logical_names: wbd, _
    -
        type: 750-469
        logical_names: pst,pst
    -
        type: 750-469
        logical_names: pst,pst
    -
        type: 750-469
        logical_names: pst,pst

Beacon: web application

Beacon: Sessions

  • Sessions allow to group control objects under a logical entity
  • the same object can be present in several sessions
  • by default, a session contains all objects; otherwise one can define 'config-objects' or 'exclude-objects'
  • a setup file can be specified, to be executed after session is loaded
class: Session
name: eh3
setup-file: ./eh3.py
config-objects:
  - aerox
  - aeroy
  - musst
  - pcoedge
  - measurement_eh3
  - multiplexer_eh3
class: Session
name: id29
default: True
setup-file: ./id29_setup.py
exclude-objects: pmbpress pmb

example from id29

example from id15

Direct Hardware Control

HW controller

Controller instance

Axis objects

BLISS process

Ethernet, serial line, USB, GPIB, etc.

Communication module

Concurrent access to hardware

State sharing

HW controller

Axis objects

BLISS process

Axis objects

BLISS process

axis state

transfer

can I move?

yes

settings are saved

BLISS tools

Command Line Interface

Based on ptpython 

sybil:~ % bliss
>>> config.names_list
['ss4hg', 'ss4ho', 'ss2vg', 'bm2v1', 'wbvmot', 'ss2vo', 'trot', 'm2theta', 'thetaenc', 'mbv2mot', 'tf1m4', 'psb', 'm1theta', 
'psf', 'psd', 'mbv3mot', 't2ry', 't2rz', 'tyrot', 'bm2v2', 'psu', 'wbv', 't1rz', 't1ry', 'tf1m1', 'tf1m3', 'tf1m2', 'sampx',
'sampy', 'ss2hg', 'mbv1mot', 'ss2ho', 'diode', 't2y', 't2z', 'bm1v1enc', 'psvo', 'm1ty', 'theta', 'm2ty', 'psvg', 'thgt', 
'ss4vo', 'tf2m4', 'tf2m2', 'tf2m3', 'tf2m1', 'ts1vg', 'ts1vo', 'ss4u', 'ss4b', 'ss4f', 't1y', 't1z', 'mbv2d', 'txrot', 'ss4vg',
'mbv4mot', 'ts1hg', 'ts1ho', 'ss2b', 'wcid232a', 'ss2d', 'ss2f', 'mbv3d', 'ttrans', 'ss2u', 'pshg', 'bm1v2', 'bm1v1', 'psho',
'ss1b', 'ss1d', 'ss1f', 'ss1u', 'ss1ho', 'ss1hg', 'tz1', 'tz3', 'tz2', 'DtoX', 'tyf', 'tyb', 'mbv1d', 'id23-2', 'ss1vg',
'ss4d', 'ss1vo', 'phiz', 'phiy', 'phix']

>>> config.get("mbv4mot")
<bliss.common.axis.Axis object at 0x7fc279b72fd0>

>>> mbv4mot = config.get("mbv4mot")

>>> mbv4mot.position()
2.0

>>> 




 [F4] Emacs  15/15 [F3] History [F6] Paste mode                                                  [F2] Menu - CPython 2.7.9

Command Line Interface

Session example

  • from ID23-2
sybil:~ % bliss --show-sessions
Session name(s):
     id23-2
(bliss) sybil:~ % bliss -s id23-2
Initializing 'ss4hg`
Initializing 'ss4ho`
Initializing 'ss2vg`
Initializing 'bm2v1`
Initializing 'wbvmot`
Initializing 'ss2vo`
Initializing 'trot`
Initializing 'm2theta`
Initializing 'thetaenc`
Initializing 'mbv2mot`
Initializing 'tf1m4`
Initializing 'psb`
Initializing 'm1theta`
Initializing 'psf`
Initializing 'psd`
Initializing 'mbv3mot`
Initializing 't2ry`
Initializing 't2rz`
Initializing 'tyrot`
Initializing 'bm2v2`
Initializing 'psu`
Initializing 'wbv`
Initializing 't1rz`
Initializing 't1ry`
Initializing 'tf1m1`
Initializing 'tf1m3`
Initializing 'tf1m2`
Initializing 'sampx`
Initializing 'sampy`
Initializing 'ss2hg`
Initializing 'mbv1mot`
Initializing 'ss2ho`
Initializing 'diode`
Initializing 't2y`
Initializing 't2z`
Initializing 'bm1v1enc`
Initializing 'psvo`
Initializing 'm1ty`
Initializing 'theta`
Initializing 'm2ty`
Initializing 'psvg`
Initializing 'thgt`
Initializing 'ss4vo`
Initializing 'tf2m4`
Initializing 'tf2m2`
Initializing 'tf2m3`
Initializing 'tf2m1`
Initializing 'ts1vg`
Initializing 'ts1vo`
Initializing 'ss4u`
Initializing 'ss4b`
Initializing 'ss4f`
Initializing 't1y`
Initializing 't1z`
Initializing 'mbv2d`
Initializing 'txrot`
Initializing 'ss4vg`
Initializing 'mbv4mot`
Initializing 'ts1hg`
Initializing 'ts1ho`
Initializing 'ss2b`
Initializing 'wcid232a`
Initializing 'ss2d`
Initializing 'ss2f`
Initializing 'mbv3d`
Initializing 'ttrans`
Initializing 'ss2u`
Initializing 'pshg`
Initializing 'bm1v2`
Initializing 'bm1v1`
Initializing 'psho`
Initializing 'ss1b`
Initializing 'ss1d`
Initializing 'ss1f`
Initializing 'ss1u`
Initializing 'ss1ho`
Initializing 'ss1hg`
Initializing 'tz1`
Initializing 'tz3`
Initializing 'tz2`
Initializing 'DtoX`
Initializing 'tyf`
Initializing 'tyb`
Initializing 'mbv1d`
Initializing 'id23-2`
Initializing 'ss1vg`
Initializing 'ss4d`
Initializing 'ss1vo`
Initializing 'phiz`
Initializing 'phiy`
Initializing 'phix`
Done.
>>>     
                                                                                                                  
                                                                                                                  
                                                                                                                  


 [F4] Emacs  90/90 [F3] History [F6] Paste mode                                                                [F2] Menu - CPython 2.7.9

Web Application Interface

BLISS scans

sybil:~ % bliss
>>> from bliss.scanning.chain import AcquisitionChain

>>> from bliss.scanning.acquisition.motor import SoftwarePositionTriggerMaster

>>> from bliss.scanning.acquisition.lima import LimaAcquisitionDevice

>>> from PyTango.gevent import DeviceProxy

>>> m0 = config.get("m0")

>>> lima_dev = DeviceProxy("id30a3/limaccd/simulation")

>>> chain = AcquisitionChain()

>>> chain.add(SoftwarePositionTriggerMaster(m0, start=5, end=10, npoints=10, time=5),
              LimaAcquisitionDevice(lima_dev, acq_nb_frames=5, acq_expo_time=0.03, acq_trigger_mode="INTERNAL_TRIGGER_MULTI"))

Continuous scan example

5

10

m0  position

m0  speed

detector frame triggering

1. Building the Acquisition Chain

m0: master

Lima device: slave

>>> SCAN_SAVING.template = '/data/id23eh2/inhouse/{date}/{sample}'

>>> SCAN_SAVING.sample = 'HAK1234'

>>> SCAN_SAVING.get_path()
"/data/id23eh2/inhouse/20170324/HAK1234"

Continuous scan example

2. Setting up where to save data

>>> from bliss.scanning.scan import Scan

>>> my_continuous_scan = Scan(chain)

>>> my_continuous_scan.prepare()

>>> my_continuous_scan.start()

3. Running scan

Scan can take additional arguments

  • scan_info : dictionary with scan metadata
  • writer : specifies how data has to be written ; defaults to bliss.scanning.writer.hdf5.HDF5Writer - setting to None means data is not saved
  • scan_data_callback :  in-process fast path to data being acquired, receives data events

3'. Running scan in background

>>> from bliss.scanning.scan import Scan

>>> my_continuous_scan = Scan(chain)

>>> my_continuous_scan.prepare()

>>> background_scan = gevent.spawn(my_continuous_scan.start)

>>> background_scan.ready() # is it finished ?

>>> background_scan.kill() # interrupt

Generic step-by-step scans

ascan, dscan, a2scan, d2scan, timescan, mesh

  • functions from bliss.common.standard
  • API example: ascan(axis, start, stop, npoints, count_time, *counters)
  • uses the same underlying scan framework
chain = AcquisitionChain()

timer = SoftwareTimerMaster(count_time, npoints=npoints)

if not counters:
    for cnt in MEASUREMENT_GROUP.enabled:
        chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))
else:
    for cnt in counters:
        chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))

chain.add(LinearStepTriggerMaster(motor, start, end, npoints), timer)

scan = Scan(chain)

scan.prepare()

scan.start()

...

Measurement Groups

Mainly for 'ct' (counting for a period of time) or generic scans

  • by default, the current active measurement group is used

MeasurementGroup objects

  • only one active at a time, MEASUREMENT_GROUP global
  • defined in configuration (.yml file)
  • ability to enable/disable counters, and to save state for later
  • default measurement group contains all counters

Data Management

Data Management

First-class citizen

  • built in the scan framework from the ground up

How is it done ?

  • flexible template for scan data path (SCAN_SAVING)
  • hierarchical organization of data, mirroring the Acquisition Chain tree
  • Online data publishing

Hierarchical organization of data

Mirroring of the Acquisition Chain tree

  • each device in the chain has a name, e.g. axis name for motors or 'timer' for Timer objects
  • each device can define AcquisitionChannel objects, e.g. counters for a P201 card or image for a Lima camera
  • Acquisition channels must have a name, a type and a shape
  • scan_info dictionary ({ key: value, ... }) for metadata

Well-defined model for converting scan data into nicely organized directories and files

Online data publishing

Based on

As data acquisition (scan) is running, data is published to the redis database provided by Beacon

  • scalar values are stored directly
  • bigger data (images, spectra) is just referenced
  • data is kept in Redis for 1 day by default

Online data publishing

Any external process can read data from Redis

  • using standard Redis tools or libraries for any language
  • BLISS provides a data iterator
>>> from bliss.data.node import get_node

>>> session_scans = get_node("my_session_name")

>>> def analyse_data(node):
...     for data in node.iterator().walk_data():    
...         <do something useful with data>

>>> for node in session_scans.iterator().walk():
...     analyse_data(node)

A perfect fit with silx-based data processing

flint data visualization tool

(development started yesterday)

inside

opid30@aelita> flint -s massif3

User Sequences

Python functions as sequences

from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import time, numpy # I know you dreamt of it
import gevent

def set_detector_cover(in):
    wcidxx.set('detcover', in)
 
    # 5 seconds timeout waiting for detector cover to move
    with gevent.Timeout(5):
        while wcidxx.get('detcover_in') == in:
          time.sleep(0.1)

def my_super_experiment(name):
    safety_shutter.open()

    old_att = attenuators.get()

    def restore_beamline():
        set_detcover_open(False)
        attenuators.set(old_att)

    # cleanup is always called at the end
    with cleanup(safety_shutter.close):
        # this will only be called in case of error
        with error_cleanup(restore_beamline):
            attenuators.set(50)
            set_detcover_open(True)

            SCAN_SAVING.name = name
            MEASUREMENT_GROUP.enable('diode')
            
            data_node = dscan(m0, -5, 5, 10, 0.1)
            
            for data in data_node.walk_data():
              # do something useful with data...

Conclusion

In the next months...

Scan framework being finalized (only a few weeks left !)

 

MCAs

 

New controllers and control objects to be added

 

flint - data visualization tool with silx from DAU

 

KB alignment functions

Beamlines already using BLISS today

MX (ID29, ID30A-1, ID30A-3, ID30B, ID23-1, ID23-2):

80% SPEC free - ID23-1 not migrated yet, will be done soon

 

ID31, ID15: ongoing migration

 

ID28 side station: partially done (only for newly installed experiment)

 

ID16, ID09: only for some motors control and P201, Keithley

 

 

Careful deployment, development in parallel: slow progress, to ensure experiments stability

... but not all at once

Thanks for your attention !

 

Feel free to ask questions