presented by Matias Guijarro - BLISS team

Beamline Control Unit, Software Group

NOBUGS

10/23/2018

NSLS-II, BNL - USA

BLISS:

experiments control

of EBS beamlines

Extremely Brilliant Source (ESRF – EBS) project

150 M€ investment over the period 2015-2022
4th generation light source
100x improved brilliance and coherence of X-ray beams
New state-of-the-art beamline portfolio

Spec is driving ESRF experiments since 26 years

                   Welcome to "spec" Release 6.03.11
         Copyright (c) 1987-2016 Certified Scientific Software
                          All rights reserved

 (Portions derived from a program developed at Harvard University.)
 (Linked with BSD libedit library for command line editing.)

 Using "/bliss/users/blissadm/spec/redhate4/spec.d"
  for auxiliary file directory (SPECD).

 Getting configuration parameters from "SPECD/matias/config".

 =
 Type h changes for info on latest changes.
 Browse to http://www.certif.com for complete documentation.
 =

 Reading file "SPECD/standard.mac".
 Warning:  No open data file.  Using "/dev/null".
 Type "startup" to initialize data file, etc.

 Reading file "SPECD/site.mac".
 Reading file "/bliss/users/blissadm/local/spec/macros/ID00setup.mac" (level 2).
 Reading file "/bliss/users/blissadm/spec/macros/lock.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlocal.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlist.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/pseudo.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/tw.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/joy.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/cplotarray.mac" (level 3).

 Doing SETUP.
 Reading file "SPECD/setup".
 Reading file "SPECD/matias/setup".

 1.MATIAS>

Poor macro language

No continuous scan framework

Limited data management

Per-session configuration

No extensibility

Limitations

Workarounds

Maintenance cost

            Welcome to "spec" Release 6.03.11
         Copyright (c) 1987-2016 Certified Scientific Software
                          All rights reserved

 (Portions derived from a program developed at Harvard University.)
 (Linked with BSD libedit library for command line editing.)

 Using "/bliss/users/blissadm/spec/redhate4/spec.d"
  for auxiliary file directory (SPECD).

 Getting configuration parameters from "SPECD/matias/config".

 =
 Type h changes for info on latest changes.
 Browse to http://www.certif.com for complete documentation.
 =

 Reading file "SPECD/standard.mac".
 Warning:  No open data file.  Using "/dev/null".
 Type "startup" to initialize data file, etc.

 Reading file "SPECD/site.mac".
 Reading file "/bliss/users/blissadm/local/spec/macros/ID00setup.mac" (level 2).
 Reading file "/bliss/users/blissadm/spec/macros/lock.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlocal.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/stlist.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/pseudo.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/tw.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/joy.mac" (level 3).
 Reading file "/bliss/users/blissadm/spec/macros/cplotarray.mac" (level 3).

 Doing SETUP.
 Reading file "SPECD/setup".
 Reading file "SPECD/matias/setup".

 1.MATIAS>

Revolution !

Beamline Instrumentation Support Software

BLISS project goals

State-of-the-art beamline control

Continuous scans, Trajectories,

Data management

Integrated environment

Command Line Interface, Configuration, Online data display

Ready for new challenges

Extensibility, user sequences

All-in-one solution

The BLISS team

Core development team: 6 engineers + Scrum master

+ Product Owners from the Beamline Control Unit: proxy to scientists

BLISS project tools

Issues list, sprints reviews, discussions and tasks to do on ESRF-hosted gitlab

Continuous Integration:

gitlab-ci

Deployment:

Documentation:

MkDocs

Beamline Instrumentation Support Software

BLISS technical choices

Python library + set of tools

Configuration application, Command Line Interface, Online data visualisation application

All I/O based on gevent

cooperative multi-tasking

Direct hardware control

Distributed

objects state

Persistent

settings cache

Data publishing,

Transient storage

Scan acquisition chain, represented as a tree

(see poster)

BLISS key principles

online data analysis

data visualisation

data archiving

2D detector

Beacon: configuration, settings, broker

Beacon static configuration service

BLISS modular architecture

Beacon static configuration service

Beamline Instrumentation Support Software

BLISS Beacon server

Beacon static configuration service

Web interface for configuration editing

Beacon server

.yml

Devices & sequences configuration in YAML format

User sessions to group beamline devices for an experiment,

Python setup file

Can replace TANGO DB

Conversion script provided

Beacon configuration services

configuration

Beacon static configuration service

Beacon, example configuration file

sybil:~/local/beamline_configuration % tree
.
├── beacon.rdb
├── eh
│   ├── diode.yml
│   ├── __init__.yml
│   └── motors
│       ├── bv.yml
│       ├── DtoX.yml
│       ├── __init__.yml
│       ├── md2.yml
│       ├── mirror1.yml
│       ├── slits.yml
│       └── table.yml
├── oh
│   ├── bpm.yml
│   ├── __init__.yml
│   ├── motors
│   │   ├── bv.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── slits.yml
│   │   └── transfocators.yml
│   └── wagos.yml
└── sessions
    ├── id232_setup.py
    ├── id232.yml
    └── __init__.yml

- controller:
    class: IcePAP
    host: iceid2322
    axes:
    - name: mbv4mot
      address: 1
      steps_per_unit: 817
      velocity: 0.3
      acceleration: 3

example configuration of a beam viewer motor

Beacon static configuration service

Beacon dynamic services

Beacon server,

services built on top of

Transient data store

Persistent settings cache

Message broker: state sharing,

distributed lock

dynamic services based on

Beacon static configuration service

Interaction between Beacon and BLISS

BLISS scans

Acquisition chain
- a tree with master & slave nodes
- master triggers data acquisition
- slave takes data
AcquisitionMaster, AcquisitionDevice
- wrappers around BLISS control objects
Data writer
- HDF5

Continuous scan example

m0 position

m0 speed

detector frame triggering

Continuous scan example

sybil:~ % bliss
>>> from bliss.scanning.chain import AcquisitionChain
>>> from bliss.scanning.acquisition.motor import SoftwarePositionTriggerMaster
>>> from bliss.scanning.acquisition.lima import LimaAcquisitionDevice
>>> from PyTango.gevent import DeviceProxy

>>> m0 = config.get("m0")

>>> lima_dev = DeviceProxy("id30a3/limaccd/simulation")

>>> chain = AcquisitionChain()

>>> chain.add(SoftwarePositionTriggerMaster(m0, start=5, end=10,
                                                npoints=10, time=5),
              LimaAcquisitionDevice(lima_dev, acq_nb_frames=5, acq_expo_time=0.03,
                                    acq_trigger_mode="INTERNAL_TRIGGER_MULTI"))

Continuous scan example

>>> SCAN_SAVING.template = '/data/id23eh2/inhouse/{date}/{sample}'

>>> SCAN_SAVING.sample = 'HAK1234'

>>> SCAN_SAVING.get_path()
"/data/id23eh2/inhouse/20170324/HAK1234"

>>> from bliss.scanning.scan import Scan

>>> my_continuous_scan = Scan(chain)

>>> my_continuous_scan.start()

Classic step-by-step scans

Directly available as functions from 'bliss.common.standard'
- Example: ascan(axis, start, stop, npoints, count_time, *counters)
Default acquisition chain
Use the same underlying framework as continuous scans

Data Management

Model for organizing acquired data

Mirroring of the Acquisition Chain tree
- each device in the chain has a name
- each device define 1 or more 'AcquisitionChannel' objects
Acquisition channels
- must have a name, a type and a shape
Metadata
- scan_info dictionary ({ key: value, ... }) associated with scans

Online data publishing

While a scan is running, data is published to the redis database provided by Beacon
- scalar values are stored directly
- bigger data (images, spectra) is just referenced
- configurable time to live (TTL)
Any external process can access redis data to perform online data analysis or live feedback for example

User

Sequences

Sequences as Python functions

from bliss import * # imports generic scans, cleanup functions, etc
from bliss.setup_globals import * # imports objects from session (setup)
import numpy # I know you dreamt of it
import gevent

def set_detector_cover(in):
    wcidxx.set('detcover', in)
 
    # 5 seconds timeout waiting for detector cover to move
    with gevent.Timeout(5):
        while wcidxx.get('detcover_in') == in:
          time.sleep(0.1)

def my_super_experiment(name):
    safety_shutter.open()

    old_att = attenuators.get()

    def restore_beamline():
        set_detcover_open(False)
        attenuators.set(old_att)

    with cleanup(safety_shutter.close):
        with error_cleanup(restore_beamline):
            attenuators.set(50)
            set_detcover_open(True)

            SCAN_SAVING.name = name
            MEASUREMENT_GROUP.enable('diode')
            
            data_node = dscan(m0, -5, 5, 10, 0.1)
            
            for data in data_node.walk_data():
              # do something useful with data...

with cleanup(safety_shutter.close): # cleanup is always called at the end
    with error_cleanup(restore_beamline):  # only called in case of error
        ...

Use of Python context managers for cleanup

Normal Python functions

Easy timeouts with gevent.Timeout

# 5 seconds timeout waiting for detector cover to move
with gevent.Timeout(5):
    while wcidxx.get('detcover_in') == in:
        time.sleep(0.1)

Current state of BLISS

ID30A-1, ID30A-3, ID30B, ID23-2, ID29

ID23-1 (on going)

BM29 (still to be done)

BLISS in user operation since 2016

on SB beamlines

Current state of BLISS

Assessing the system on more beamlines before the EBS

ID31

BM32

ID21

BM16

ID15

ID13

ID11

ID10

Current state of BLISS

Aim for complete transitions: no dual spec & BLISS installations

Commissioning time (with beam) is invaluable

Need engagement from beamline scientists

Human adventure

Lessons learned

Conclusion

Spec has reached end-of-life at ESRF

BLISS is being designed for the needs of the EBS beamlines,

and already running experiments

Half of the beamlines will be converted during the EBS shutdown (2019-2020)

Would be happy to start collaborations around BLISS with interested people

BLISS:

experiments control

of EBS beamlines

Extremely Brilliant Source (ESRF – EBS) project

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

BLISS scans

BLISS scans

Continuous scan example

Continuous scan example

Continuous scan example

Classic step-by-step scans

Data Management

Model for organizing acquired data

Online data publishing

User

Sequences

Sequences as Python functions

BLISS in user operation since 2016

on SB beamlines

NOBUGS_BLISS_2018

NOBUGS_BLISS_2018

Matias Guijarro

BLISS:

experiments control

of EBS beamlines

Extremely Brilliant Source (ESRF – EBS) project

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

Beacon static configuration service

BLISS scans

BLISS scans

Continuous scan example

Continuous scan example

Continuous scan example

Classic step-by-step scans

Data Management

Model for organizing acquired data

Online data publishing

User

Sequences

Sequences as Python functions

BLISS in user operation since 2016

on SB beamlines

NOBUGS_BLISS_2018

More from Matias Guijarro