"A new sequencer for the beamlines will be developed to replace SPEC.

[...] The new sequencer will be based on Python. [...]

The new sequencer will rewrite the experiment macros so that

they are easier to maintain."

(excerpt from the ESRF Phase II TDS Orange Book)

SPEC

The good in SPEC

Direct control of devices

• (much) easier to debug / understand
• reconfig = reset
• no additional data transfers

Integrated tool

• plots display
• configuration

the BLISS project

BeamLine Instrumentation Support Software

• project repository: gitlab.esrf.fr/bliss/bliss
• started in December, 2014
• 2700+ commits
• more than 50.000 lines of Python code
• 4 core developers, 10 participants

BLISS project overview

Key concepts

• Python library + tools
• Modular architecture
• Synchronous API on top of evented I/O for cooperative multi-tasking
• Direct HW control from 1 BLISS process at a time, support for several peers via state sharing
• Centralized configuration
• Control objects
• Master/Slave Tree-based acquisition chains
• Data management
• Sequences as genuine Python functions

Multi-tasking + synchronous API

Python library with modular architecture

Configuration

Tools / User interfaces

Scans

User sequences

Data management

The path to BLISS

Multitasking ?

Evented I/O ?

Synchronous API ?

Experiment Control & Data Acquisition

heavily rely on Input/Output (IO) operations

run

run

run

run

Task execution

• For I/O, tasks must wait (sleep)
• Behind the scene, the underlying system will carry the I/O operation and wake up the task when buffers are ready

Task execution vs. I/O system call

Evented I/O is an operating system feature that allows to not block during I/O calls

Evented I/O

• System will fire READ or WRITE events when buffers are ready, so I/O can be performed in no time
• Multi-tasking can be achieved by switching tasks between read or write events

Task A

run

run

Task B

run

run

gevent Python library offers a "task" abstraction,

and uses evented I/O for system calls

• gevent adds a synchronous API on top of evented I/O
• Python library is patched to be gevent-friendly
• Task context switch becomes transparent

gevent synchronous API

Centralized Configuration:

introducing Beacon

Beacon, brilliant Beamline Configuration

Beacon: example configuration from ID23-2

- controller:
    class: IcePAP
    host: iceid2322
    axes:
    - name: mbv4mot
      address: 1
      steps_per_unit: 817
      velocity: 0.3
      acceleration: 3

- name: wcid232a
  class: wago
  controller_ip: wcid232a
  counter_gain_names: wbd_g
  counter_names: wbd
  mapping:
    -
        type: 750-516
        logical_names: wbd_g,wbd_g,wbd_g,_
    -
        type: 750-630
        logical_names: psenc
    -
        type: 750-630
        logical_names: psenc
    -
        type: 750-630
        logical_names: wbvenc
    -
        type: 750-513
        logical_names: psrelay, _
    -
        type: 750-467
        logical_names: wbd, _
    -
        type: 750-469
        logical_names: pst,pst
    -
        type: 750-469
        logical_names: pst,pst
    -
        type: 750-469
        logical_names: pst,pst

Beacon: web application

Beacon: Sessions

BLISS tools

Command Line Interface

Based on ptpython

sybil:~ % bliss
>>> config.names_list
['ss4hg', 'ss4ho', 'ss2vg', 'bm2v1', 'wbvmot', 'ss2vo', 'trot', 'm2theta', 'thetaenc', 'mbv2mot', 'tf1m4', 'psb', 'm1theta', 
'psf', 'psd', 'mbv3mot', 't2ry', 't2rz', 'tyrot', 'bm2v2', 'psu', 'wbv', 't1rz', 't1ry', 'tf1m1', 'tf1m3', 'tf1m2', 'sampx',
'sampy', 'ss2hg', 'mbv1mot', 'ss2ho', 'diode', 't2y', 't2z', 'bm1v1enc', 'psvo', 'm1ty', 'theta', 'm2ty', 'psvg', 'thgt', 
'ss4vo', 'tf2m4', 'tf2m2', 'tf2m3', 'tf2m1', 'ts1vg', 'ts1vo', 'ss4u', 'ss4b', 'ss4f', 't1y', 't1z', 'mbv2d', 'txrot', 'ss4vg',
'mbv4mot', 'ts1hg', 'ts1ho', 'ss2b', 'wcid232a', 'ss2d', 'ss2f', 'mbv3d', 'ttrans', 'ss2u', 'pshg', 'bm1v2', 'bm1v1', 'psho',
'ss1b', 'ss1d', 'ss1f', 'ss1u', 'ss1ho', 'ss1hg', 'tz1', 'tz3', 'tz2', 'DtoX', 'tyf', 'tyb', 'mbv1d', 'id23-2', 'ss1vg',
'ss4d', 'ss1vo', 'phiz', 'phiy', 'phix']

>>> config.get("mbv4mot")
<bliss.common.axis.Axis object at 0x7fc279b72fd0>

>>> mbv4mot = config.get("mbv4mot")

>>> mbv4mot.position()
2.0

>>> 




 [F4] Emacs  15/15 [F3] History [F6] Paste mode                                                  [F2] Menu - CPython 2.7.9 

Command Line Interface

Session example

• from ID23-2

sybil:~ % bliss --show-sessions
Session name(s):
     id23-2
(bliss) sybil:~ % bliss -s id23-2
Initializing 'ss4hg`
Initializing 'ss4ho`
Initializing 'ss2vg`
Initializing 'bm2v1`
Initializing 'wbvmot`
Initializing 'ss2vo`
Initializing 'trot`
Initializing 'm2theta`
Initializing 'thetaenc`
Initializing 'mbv2mot`
Initializing 'tf1m4`
Initializing 'psb`
Initializing 'm1theta`
Initializing 'psf`
Initializing 'psd`
Initializing 'mbv3mot`
Initializing 't2ry`
Initializing 't2rz`
Initializing 'tyrot`
Initializing 'bm2v2`
Initializing 'psu`
Initializing 'wbv`
Initializing 't1rz`
Initializing 't1ry`
Initializing 'tf1m1`
Initializing 'tf1m3`
Initializing 'tf1m2`
Initializing 'sampx`
Initializing 'sampy`
Initializing 'ss2hg`
Initializing 'mbv1mot`
Initializing 'ss2ho`
Initializing 'diode`
Initializing 't2y`
Initializing 't2z`
Initializing 'bm1v1enc`
Initializing 'psvo`
Initializing 'm1ty`
Initializing 'theta`
Initializing 'm2ty`
Initializing 'psvg`
Initializing 'thgt`
Initializing 'ss4vo`
Initializing 'tf2m4`
Initializing 'tf2m2`
Initializing 'tf2m3`
Initializing 'tf2m1`
Initializing 'ts1vg`
Initializing 'ts1vo`
Initializing 'ss4u`
Initializing 'ss4b`
Initializing 'ss4f`
Initializing 't1y`
Initializing 't1z`
Initializing 'mbv2d`
Initializing 'txrot`
Initializing 'ss4vg`
Initializing 'mbv4mot`
Initializing 'ts1hg`
Initializing 'ts1ho`
Initializing 'ss2b`
Initializing 'wcid232a`
Initializing 'ss2d`
Initializing 'ss2f`
Initializing 'mbv3d`
Initializing 'ttrans`
Initializing 'ss2u`
Initializing 'pshg`
Initializing 'bm1v2`
Initializing 'bm1v1`
Initializing 'psho`
Initializing 'ss1b`
Initializing 'ss1d`
Initializing 'ss1f`
Initializing 'ss1u`
Initializing 'ss1ho`
Initializing 'ss1hg`
Initializing 'tz1`
Initializing 'tz3`
Initializing 'tz2`
Initializing 'DtoX`
Initializing 'tyf`
Initializing 'tyb`
Initializing 'mbv1d`
Initializing 'id23-2`
Initializing 'ss1vg`
Initializing 'ss4d`
Initializing 'ss1vo`
Initializing 'phiz`
Initializing 'phiy`
Initializing 'phix`
Done.
>>>     
                                                                                                                  
                                                                                                                  
                                                                                                                  


 [F4] Emacs  90/90 [F3] History [F6] Paste mode                                                                [F2] Menu - CPython 2.7.9 

Web Application Interface

BLISS scans

sybil:~ % bliss
>>> from bliss.scanning.chain import AcquisitionChain

>>> from bliss.scanning.acquisition.motor import SoftwarePositionTriggerMaster

>>> from bliss.scanning.acquisition.lima import LimaAcquisitionDevice

>>> from PyTango.gevent import DeviceProxy

>>> m0 = config.get("m0")

>>> lima_dev = DeviceProxy("id30a3/limaccd/simulation")

>>> chain = AcquisitionChain()

>>> chain.add(SoftwarePositionTriggerMaster(m0, start=5, end=10, npoints=10, time=5),
              LimaAcquisitionDevice(lima_dev, acq_nb_frames=5, acq_expo_time=0.03, acq_trigger_mode="INTERNAL_TRIGGER_MULTI"))
  

Continuous scan example

5

10

m0  position

m0  speed

detector frame triggering

1. Building the Acquisition Chain

m0: master

Lima device: slave

>>> SCAN_SAVING.template = '/data/id23eh2/inhouse/{date}/{sample}'

>>> SCAN_SAVING.sample = 'HAK1234'

>>> SCAN_SAVING.get_path()
"/data/id23eh2/inhouse/20170324/HAK1234"

Continuous scan example

2. Setting up where to save data

>>> from bliss.scanning.scan import Scan

>>> my_continuous_scan = Scan(chain)

>>> my_continuous_scan.prepare()

>>> my_continuous_scan.start()

3. Running scan

Scan can take additional arguments

• scan_info : dictionary with scan metadata
• writer : specifies how data has to be written ; defaults to bliss.scanning.writer.hdf5.HDF5Writer - setting to None means data is not saved
• scan_data_callback :  in-process fast path to data being acquired, receives data events

Generic step-by-step scans

ascan, dscan, a2scan, d2scan, timescan

• functions from bliss.common.standard
• API example: ascan(axis, start, stop, npoints, count_time, *counters)
• uses the same underlying scan framework

chain = AcquisitionChain()

timer = SoftwareTimerMaster(count_time, npoints=npoints)

if not counters:
    for cnt in MEASUREMENT_GROUP.enabled:
        chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))
else:
    for cnt in counters:
        chain.add(timer, CounterAcqDevice(cnt, count_time, npoints))

chain.add(LinearStepTriggerMaster(motor, start, end, npoints), timer)

scan = Scan(chain)

scan.prepare()

scan.start()

...

Measurement Groups

Mainly for 'ct' (counting for a period of time) or generic scans

• by default, the current active measurement group is used

MeasurementGroup objects

• only one active at a time, MEASUREMENT_GROUP global
• defined in configuration (.yml file)
• ability to enable/disable counters, and to save state for later
• default measurement group contains all counters

Data Management

Data Management

First-class citizen

• built in the scan framework from the ground up

How is it done ?

• flexible template for scan data base path (SCAN_SAVING)
• hierarchical organization of data, mirroring the Acquisition Chain tree
• Online data publishing

Hierarchical organization of data

Mirroring of the Acquisition Chain tree

• each device in the chain has a name, e.g. axis name for motors or 'timer' for Timer objects
• each device can define AcquisitionChannel objects, e.g. counters for a P201 card or image for a Lima camera
• Acquisition channels must have a name, a type and a shape
• scan_info dictionary ({ key: value, ... }) for metadata

Online data publishing

Based on

As data acquisition (scan) is running, data is published to the redis database provided by Beacon

• scalar values are stored directly
• bigger data (images, spectra) is just referenced
• data is kept in Redis for 1 day by default

Online data publishing

Any external process can read data from Redis

• using standard Redis tools or libraries for any language
• BLISS provides a data iterator

User Sequences

Python functions as sequences

from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import time, numpy # I know you dreamt of it
import gevent

def set_detector_cover(in):
    wcidxx.set('detcover', in)
 
    # 5 seconds timeout waiting for detector cover to move
    with gevent.Timeout(5):
        while wcidxx.get('detcover_in') == in:
          time.sleep(0.1)

def my_super_experiment(name):
    safety_shutter.open()

    old_att = attenuators.get()

    def restore_beamline():
        set_detcover_open(False)
        attenuators.set(old_att)

    # cleanup is always called at the end
    with cleanup(safety_shutter.close):
        # this will only be called in case of error
        with error_cleanup(restore_beamline):
            attenuators.set(50)
            set_detcover_open(True)

            SCAN_SAVING.name = name
            MEASUREMENT_GROUP.enable('diode')
            
            data_node = dscan(m0, -5, 5, 10, 0.1)
            
            for data in data_node.walk_data():
              # do something useful with data...

Conclusion

In the next months...

Scan framework being finalized (only a few weeks left !)

MCAs

New controllers to be added

• Eurotherm, Lakeshore
• future P201 replacement card
• N354 bench clock card
• BCDU8
• ...

KB alignment functions

Beamlines already using BLISS today

MX (ID29, ID30A-1, ID30A-3, ID30B, ID23-1, ID23-2):

80% SPEC free - ID23-1 not migrated yet, will be done soon

ID31, ID15: ongoing migration

ID28 side station: partially done (only for newly installed experiment)

ID16, ID09: only for some motors control and P201, Keithley

Careful deployment, development in parallel: slow progress, to ensure experiments stability

... but not all at once