• Client communicates with server by sending messages (i.e. request)
• Server answers with messages that client can understand (i.e. response)
• Both kinds of messages need to have a specified format that both the server and the client implement
• How do we define the messages for encoding and decoding in a DRY and easily extensible way using a programming language of our choice?

Problem Definition

Naive Solution

class Message(object):
    def __init__(self, id, version=10, flags=0, operation=0x01):
        self.id = id
        self.version = version
        self.flags = flags
        self.operation = operation

    def encode(self):
        # ...

    def decode(self, message):
        # ...

• No matadata (not even types in dynamically typed languages)
• Messages have too much responsibility
• Tight coupling

The Dictionary Way

class Message(object):
    metadata = {
        "id": {"type": "uvarlong"},
        "version": {"type": "byte"},
        "flags": {"type": "uvarint", "since_version": 11},
        "operation": {"type": "byte"}
    }

    def __init__(self, id, version=12, flags=0, operation=0x01):
        self.id = id
        self.version = version
        self.flags = flags
        self.operation = operation

• Not pretty
• No basic type checking or validation, unless performed manually

The SQLAlchemy Way

import messenger as m

class Message(m.Message):
    id = m.Uvarlong()
    version = m.Byte(default=12)
    flags = m.Uvarint(default=0, since_version=11)
    operation = m.Byte(default=0x01)

• Class attributes carry metadata information, object attributes contain concrete values
• Basic type checking by default, validation can be done by m.Message super class
• Separation of concerns: validation, encoding/decoding, transmitting etc. can be done by different objects

Implementation of Datatypes

class DataType(object):
    def __init__(self, **kwargs):
        for key, value in kwargs.iteritems():
            setattr(self, key, value)

    @property
    def type(self):
        return type(self).__name__.lower()

class Byte(DataType):
    pass

class Uvarint(DataType):
    pass

class Uvarlong(DataType):
    pass

Implementation of Message Super Class

class Message(object):
    def __init__(self, **kwargs):
        # Filter out all defined fields that are private and that are not an
        # instance of class DataType
        m_fields = filter(
            lambda f: not f.startswith('__') and
            isinstance(getattr(self, f), DataType), dir(self))

        for f_name in m_fields:
            f_cls = getattr(self.__class__, f_name)
            # If someone used key-word arguments in the initializer,
            # let's pass them as the default values
            if f_name in kwargs:
                setattr(self, f_name, kwargs[f_name])
            # If there is a default available, use that
            elif hasattr(f_cls, 'default'):
                setattr(self, f_name, f_cls.default)
            # If all that fails, just initialize to None
            else:
                setattr(self, f_name, None)

    @property
    def cls(self):
        return self.__class__

What About Order?

class CreatedCounter(object):
    _count = 0

    @staticmethod
    def count():
        count = CreatedCounter._count
        CreatedCounter._count += 1
        return count


class DataType(object):
    def __init__(self, **kwargs):
        self._created = CreatedCounter.count()
        # ...

Python doesn't remember order of definiton, easy to fix:

class Message(object):
    def __init__(self, **kwargs):
        m_fields = filter(
            lambda f: not f.startswith('__') and
            isinstance(getattr(self, f), DataType), dir(self))

        self.fields = \
            sorted(m_fields, key=lambda fn: getattr(self, fn)._created)

        for f_name in self.fields:
            # ...

What's Next?

class Composite(DataType):
    pass

class List(DataType):
    pass

class Bytes(DataType):
    def __init__(self, size, **kwargs):
        super(Bytes, self).__init__(**kwargs)
        self.args.append(size)

class Varbytes(DataType):
    pass

class Ushort(DataType):
    pass

• More types, composite types, lists, types with arguments:

• Encoder/decoder implementation that interprets the matadata

Real World Example

class RequestHeader(m.Message):
    magic = m.Byte(default=0xA0)
    id = m.Uvarlong()
    version = m.Byte(default=25)
    op = m.Byte()
    cname = m.String(optional=True)
    flags = m.Uvarint(default=0)
    ci = m.Byte(default=ClientIntelligence.BASIC)
    t_id = m.Uvarint(default=0)

class Request(m.Message):
    header = m.Composite(default=RequestHeader)

    def __init__(self, **kwargs):
        super(Request, self).__init__(**kwargs)
        self.header.op = self.OP_CODE

class GetRequest(Request):
    OP_CODE = 0x03
    key = m.Varbytes()

• Taken from: https://github.com/VaclavDedik/infinispan-py

Easy to Add New Functionality

 @@ -229,6 +221,29 @@
      OP_CODE = 0x14
  
  
 +class StatsRequest(Request):
 +    OP_CODE = 0x15
 +
 +
 +class Stat(m.Message):
 +    name = m.String()
 +    value = m.String()
 +
 +
 +class StatsResponse(Response):
 +    OP_CODE = 0x16
 +    n = m.Uvarint()
 +    stats = m.List(of=Stat, size=lambda s: s.n)
 +
 +
  class ErrorResponse(Response):
      OP_CODE = 0x50
      error_message = m.String()

http://vaclavdedik.com

@VaclavDedik

Thanks for listening.

Questions?