MASSIF control software stack:

leveraging latest ESRF developments and new insights

MXCuBE meeting, 1st of June 2015 - Helmholtz-Zentrum Berlin (HZB)

Matias Guijarro

Software Engineer @ Beamline Control Unit

European Synchrotron Radiation Facility

Grenoble, France

MASSIF (UPBL10), high-throughput MX sample characterization

ID30B

ID30A3

ID30A2

ID30A1

MASSIF (UPBL10), high-throughput MX sample characterization

 

ID30A-1

 

Dectris Pilatus 3 2M detector

Prosilica GC655C on-axis viewer

Oxford 700 cryo stream controller

Keithley 6485 picoammeter for diodes reading

Icepap-driven slits motors

Icepap-driven rotating fast shutter

+ a lot of Wago-controlled actuators

MASSIF (UPBL10), high-throughput MX sample characterization

 

ID30A-1

Staubli arm used both as sample changer and goniometer

Oscillation synchronisation done with ESRF MUSST card (Ethernet GPIB)

 

All axis are Icepap-driven

 

MASSIF (UPBL10), high-throughput MX sample characterization

 

ID30A-1

"Big dewar"

3 cells of 8 pucks, 10 SPINE samples each: 240 samples

1 Icepap-driven axis for cell selection

Keyence cameras for detection

Wago modules for operation

MASSIF (UPBL10), high-throughput MX sample characterization

 

ID30A-3

Dectris Pilatus 3 2M (will be replaced by Eiger 4M, when Eiger will be ready)

 

KB mirror with NF8753-driven motors

 

MD2M diffractometer (MD2 little brother with Icepap-driven axes,  Galil DMC213-driven axis for omega)

 

EMBL SC3 sample changer

 

Other parts are similar to ID30A-1

MASSIF (UPBL10), high-throughput MX sample characterization

 

ID30B, collaboration with

Dectris Pilatus 6M 

 

Tunable wavelength, fluorescence detector

 

Bended single mirror, IcePAP driven

 

MD2S diffractometer (latest EMBL development)

 

EMBL SC3 sample changer + new EMBL sample changer and plate robot

ESRF Big Dewar

MASSIF software components map

 Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

 

Antoine de Saint-Exupéry

We don't need

MAGIC

Configuring with Beacon

M. Guijarro, S. Petitdemange

Beacon, brilliant Beamline Configuration

Client/server architecture

 

Objects configuration repository

 

Settings cache based on

 

Channels: peer-to-peers data exchange between Beacon clients

 

Distributed Lock Manager

 

Drop-in replacement for Tango database (experimental)

Beacon objects configuration repository

A set of .yml files organized in directories

 

A file can contain multiple objects

 

Each object must have an unique name

 

Beacon objects configuration repository

bacon:~/local/beamline_control/configuration % tree
.
├── calibrated_diodes.dat
├── eh
│   ├── bpms.yml
│   ├── detector_cover.yml
│   ├── fshutter.yml
│   ├── i0i1.yml
│   ├── __init__.yml
│   ├── MD2S.yml
│   ├── motors
│   │   ├── DtoX.yml
│   │   ├── ehtable.yml
│   │   ├── fshut.yml
│   │   ├── __init__.yml
│   │   ├── mirror.yml
│   │   ├── slitbox.yml
│   │   └── ss2lits.yml
│   ├── musst.yml
│   ├── tango_shutters.yml
│   ├── transmission.yml
│   └── wagos.yml
├── __init__.yml
├── oh
│   ├── bpms.yml
│   ├── frontend.yml
│   ├── __init__.yml
│   ├── mbv1_diode.yml
│   ├── monocal_diode.yml
│   ├── motors
│   │   ├── energy_wl.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── pslits.yml
│   │   ├── ss1lits.yml
│   │   ├── transfocator.yml
│   │   └── undulators.yml
│   ├── transfocator.yml
│   └── wagos.yml
├── percent.arr
└── undulators.dat
name: tfmad
class: transfocator
controller_ip: 160.103.50.57
lenses: 7
pinhole: 2
>>> from beacon import static
>>> cfg = static.get_config()
>>> tfmad = cfg.get("tfmad")
>>> tfmad
filename:<tfmad.yml>,plugin:None,{'controller_ip': '160.103.50.57', 'pinhole': 2, 'name': 'tfmad', 'lenses': 7, 'class': 'transfocator'}
>>> tfmad["lenses"]
7
>>> tfmad["lenses"] = 6
>>> tfmad.save()

Beacon objects configuration repository

Objects are singletons

 

By default, a dict-like object is returned

 

The plugin key allow to specify particular functions for object instantiation

        - compatible with any library

 

Setting keys in __init__.yml assign properties to a whole set of objects in a directory

 

Beacon settings cache

Beacon client API offers various Settings objects

 

Settings are a thin helper layer on top of Redis structures

>>> from beacon import settings
>>> d = settings.HashSetting("example")
>>> d
<HashSetting name=example value={}>
>>> d["foo"]="bar"
>>> d
<HashSetting name=example value={'foo': 'bar'}>
>>> 
linguijarro:~ % redis-cli
127.0.0.1:6379> keys *
1) "example"
127.0.0.1:6379> hgetall example
1) "foo"
2) "bar"
127.0.0.1:6379> 

Server-side

Client-side

Beacon channels

Peer-to-peers data exchange between clients connected to the same beacon server

 

Serialization with standard "pickle" Python module

 

Designed for small amount of data, ie. states, scalar values, simple strings

 

Under the hood: nanomsg (SURVEY and BUS scalability protocols)

+ gevent

 

Beacon channels example

Client A

>>> from beacon import channels
>>> c = channels.Channel("test", "hello, world")
>>> c.value
'hello, world'
>>> 

Client B

>>> from beacon import channels
>>> c = channels.Channel("test")
>>> c.value
'hello, world'
>>> c.value = 42
>>>
>>> c.value
42

Client C

>>> from beacon import channels
>>> c = channels.Channel("test")
>>> c.value
42
>>> c.value = "unlimited power"
>>> c.value
'unlimited power'

Positioning with the ESRF Motion (Emotion) library

C. Guilloud, M. Guijarro, M. Perez

Emotion features

  • 4 well-defined base classes: Controller, Axis, Group & Encoder
  • powered by gevent
  • Virtual (pseudo) axes
  • State of the art control: software limits, user positions, backlash handling, smooth stopping, stepper motors under-resolution move compensation
  • Events
  • Easy to configure, multiple back-ends support
  • Mockup object for simulation / testing
  • Unit tests suite, 70% code coverage
  • Comes with a generic Tango server

Default back-end is XML

 

Beacon back-end

  • integration with full beamline configuration
  • Beacon plugin to get Emotion objects from Beacon

Emotion configuration with Beacon

controller:
    class: IcePAP
    host: iceid306
    axes:
         -
            name: DtoX
            address: 26
            steps_per_unit: 80
            velocity: 125
            acceleration: 500
>>> from beacon import static
>>> cfg = static.get_config()
>>> dtox = cfg.get("DtoX")
>>> dtox
<emotion.axis.Axis object at 0x2729190>
>>> dtox.position()
700.0
>>> dtox.acceleration()
500.0
>>> dtox.velocity()
250.0
bacon:~/local/beamline_control/configuration % tree
.
├── calibrated_diodes.dat
├── eh
│   ├── bpms.yml
│   ├── detector_cover.yml
│   ├── fshutter.yml
│   ├── i0i1.yml
│   ├── __init__.yml
│   ├── MD2S.yml
│   ├── motors
│   │   ├── DtoX.yml
│   │   ├── ehtable.yml
│   │   ├── fshut.yml
│   │   ├── __init__.yml
│   │   ├── mirror.yml
│   │   ├── slitbox.yml
│   │   └── ss2lits.yml
│   ├── musst.yml
│   ├── tango_shutters.yml
│   ├── transmission.yml
│   └── wagos.yml
├── __init__.yml
├── oh
│   ├── bpms.yml
│   ├── frontend.yml
│   ├── __init__.yml
│   ├── mbv1_diode.yml
│   ├── monocal_diode.yml
│   ├── motors
│   │   ├── energy_wl.yml
│   │   ├── __init__.yml
│   │   ├── mono.yml
│   │   ├── pslits.yml
│   │   ├── ss1lits.yml
│   │   ├── transfocator.yml
│   │   └── undulators.yml
│   ├── transfocator.yml
│   └── wagos.yml
├── percent.arr
└── undulators.dat
plugin: emotion

Emotion Virtual Axes

Example: 2-leg table, can translate and rotate

controller:
    class: tabsup
    d1: 730
    d2: 1208
    axes:
        -
            name: $tyf
            tags: real front
        -
            name: $tyb
            tags: real back
        -
            name: ttrans
            tags: ttrans
        -
            name: tzrot
            tags: trot

Configuration (Beacon)

$ indicates a reference to another object

from emotion import CalcController
import math


class tabsup(CalcController):
    def __init__(self, *args, **kwargs):
        CalcController.__init__(self, *args, **kwargs)

        self.d1 = self.config.get("d1", float)
        self.d2 = self.config.get("d2", float)

    def calc_from_real(self, positions_dict):
        tyf = positions_dict["front"]
        tyb = positions_dict["back"]
        d1 = self.d1
        d2 = self.d2

        return {"ttrans": (d1*tyb+d2*tyf)/(d1+d2),
                "trot": math.degrees(math.atan((tyf-tyb)/(d1+d2))) }

    def calc_to_real(self, axis_tag, positions_dict):
        ttrans = positions_dict["ttrans"]
        trot = positions_dict["trot"]
        d1 = self.d1
        d2 = self.d2

        return {"back": ttrans - d2*math.tan(math.radians(trot)),
                "front": ttrans + d1*math.tan(math.radians(trot)) }

Supported motor controllers

Hardware

PI E517, PI E712, PI E753

Galil DMC 213

PMD 206

NewFocus 8753

FlexDC

IcePAP

Virtual

Slits

KB mirror

2-leg table

3-leg table

Energy

 

Software

Tango Undulator

MD2 Exporter (for MD2 motors)

PiezoJack

Add new controllers easily

# mandatory methods
def initialize_axis(axis)
def start_one(motion)
def stop(axis)
def state(axis)
def read_position(axis)
# optional
def read_velocity(axis)
def set_velocity(axis)
def read_acceleration(axis)
def set_acceleration(axis, new_acc)
def set_position(axis, new_position)

def start_all(motion)
def stop_all()

def set_on(axis)
def set_off(axis)

def initialize_encoder(encoder)
def read_encoder(encoder)
def set_encoder(encoder, new_value)

def home_search(axis)
def home_set_hardware_position(axis, home_pos)
def home_state(axis)

def limit_search(axis, limit)

MASSIF beamlines are the first ESRF beamlines without spec

MXCuBE is the main software for users

 

But...

 

What about beamline commissioning ?

Doing calibration and alignment procedures ?

Testing the beamline ?

 

Introducing

K _ _ _ _ s

 

 

 

A. Beteva,

M. Guijarro,

S. Petitdemange

Khoros, MASSIF sequencer and beamline control software

Beamline components

Scanning

Sequences

  • Python functions
  • Helpers: cleanup context manager, Task abstraction
  • Step by step: timescan, "ascan", "dscan", "a2scan", "d2scan"
  • Data Manager, writes different file formats (spec, HDF5, ...)
  • High-level objects (actuator, shutter, diffractometer, pixel detector, etc.)
  • Configuration with Beacon (Khoros Beacon plugin)
  • Counters & measurement groups
  • Motors via Emotion

Khoros library

>>> from khoros import setup
>>> setup("/home/guijarro/local/test_setup.py")
Reading setup file '/home/guijarro/local/test_setup.py`
Initializing 'i0`
Initializing 'm0`
Initializing 'm1`
Done.
True
>>> DEFAULT_OFFSET
0.0
>>> add(3,6)
9
>>> m0.move(1)
>>> m0.position()
1.0
>>> ascan(m0,1,2,10,0.1,i0)
>>> last_scan_data()
array([[  1.        ,   0.375     ],
       [  1.11111111,   5.625     ],
       [  1.22222222,  -7.        ],
       [  1.33333333,   8.625     ],
       [  1.44444444, -25.5       ],
       [  1.55555556,   0.125     ],
       [  1.66666667,  -8.125     ],
       [  1.77777778,   2.28571429],
       [  1.88888889,  -3.125     ],
       [  2.        ,  10.625     ]])
>>>

# Khoros setup file
# -----------------
# nothing is imported by default,
# let's import basic scans
from khoros.core.scans import *

# basically any Python code can be
# imported; objects goes to the
# 'setup_globals' namespace.
# When setup is executed from the
# interpreter, objects are exposed in
# the global namespace too

# for demonstration purpose let's
# define some functions
def double(n):
  return 2*n

def add(x,y):
  return x+y

# Beamline components are loaded prior
# to setup file execution, so it's
# possible to use beamline components here:
DEFAULT_OFFSET = m0.position()

test_setup.py

Khoros Beamline Components

from khoros.core.measurement import CounterBase, AverageMeasurement
import PyTango.gevent
import time

class tango_attr_as_counter(CounterBase):
    def __init__(self, name, config):
        CounterBase.__init__(self, name)
        tango_uri = config.get("uri")
        self.__control = PyTango.gevent.DeviceProxy(tango_uri)
        self.attribute = config.get("attr_name")

    def read(self, acq_time=None):
        meas = AverageMeasurement()
        for reading in meas(acq_time):
            reading.value = self.__control.read_attribute(self.attribute).value
        return meas.average

No framework burden: the only requirement is to have 'name' (string) and 'config' (dict) as constructor arguments

Some helper base classes are provided

from khoros.core import Actuator

class detector_cover(Actuator):
   def __init__(self, name, config):
       Actuator.__init__(self)

       self.wago = config["controller"]
       self.key_in = config['cover_state_in']
       self.key_out = config['cover_state_out']
       self.key_cmd = config['cover_cmd']

   def _set_in(self):
       self.wago.set(self.key_cmd, 0)

   def _set_out(self):
       self.wago.set(self.key_cmd, 1)

   def _is_in(self):
       return self.wago.get(self.key_in)

   def _is_out(self):
       return self.wago.get(self.key_out)

Khoros Sequences

from khoros import *
from khoros.setup_globals import *
import logging
import sys
import numpy as np

def quick_realign(slits={s1v:0.2}, cnt=i0, mot=thgt, npts=40, start=-0.2, stop=0.2):    
    def restore_slits(motor_pos = [(m, m.position()) for m in slits.iterkeys()):
        for m, pos in motor_pos:
            logging.info("Restoring '%s` to %f", m.name, pos)
            m.move(pos)

    def quick_realign_cleanup(transmission = attenuators.get_transmission()):
        fast_shutter.close()   
        safshut.close()
        attenuators.set_transmission(transmission)
       
    with cleanup(quick_realign_cleanup):
        with cleanup(restore_slits):
            for m, pos in slits.iteritems():
               m.move(pos, wait=False)        
        
            safshut.open()
            fast_shutter.open()
            attenuators.set_transmission(100)
            flux = cnt.read()
            logging.info("Flux before quick realign: %f", flux)
                
            dscan(mot, start, stop, npts, 0.1, cnt)
            data = last_scan_data()
            max_pos = data[np.argmax(data[:,1]),0]

            mot.move(max_pos)
        centrebeam()
        flux = cnt.read()
        logging.info("Flux after quick realign: %f", flux)
        

Khoros Shell

Web application, a la IPython Notebook (Jupyter)

Khoros Shell architecture

bottle + gevent

gipcgevent-cooperative child processes and inter-process communication

Khoros Shell architecture

Shell is part of the package, but it is an optional component

 

Nice proof-of-concept, with only 800 Javascript LOC (shell.js + synoptic.js),  700 Python LOC (shell.py + interpreter.py)

 

Currently in use in production on ID30B

Dependencies list

JavaScript: jQuery 2.1, jQueryUI, jQuery Layout, MouseTrap, Dygraph, CodeMirror

Python: bottle, gevent, gipc

 

Khoros integration within MXCuBE 2

Straightforward with dedicated Hardware Objects

 

One "Khoros" Hardware Object is used as a "footbridge" for others to get access to Khoros exported objects

from HardwareRepository.BaseHardwareObjects import HardwareObject
import os
import sys
import khoros

class Khoros(HardwareObject):
  def __init__(self, *args):
    HardwareObject.__init__(self, *args)

  def init(self, *args):
     setup_file = self.getProperty("setup_file")

     khoros.setup(setup_file, self.__dict__)

Future perspectives for MXCuBE

Replace Hardware Repository with Beacon ?

 

Replace MXCuBE Motor object (which doesn't really exist ?) to an Emotion object -at least with the same API-

 

MXCuBE 3 Web Edition could integrate a CLI

Questions ?