Defining Messages For a Client/Server Protocol Implementation
- Client communicates with server by sending messages (i.e. request)
- Server answers with messages that client can understand (i.e. response)
- Both kinds of messages need to have a specified format that both the server and the client implement
- How do we define the messages for encoding and decoding in a DRY and easily extensible way using a programming language of our choice?
Problem Definition
Naive Solution
class Message(object):
def __init__(self, id, version=10, flags=0, operation=0x01):
self.id = id
self.version = version
self.flags = flags
self.operation = operation
def encode(self):
# ...
def decode(self, message):
# ...
- No matadata (not even types in dynamically typed languages)
- Messages have too much responsibility
- Tight coupling
The Dictionary Way
class Message(object):
metadata = {
"id": {"type": "uvarlong"},
"version": {"type": "byte"},
"flags": {"type": "uvarint", "since_version": 11},
"operation": {"type": "byte"}
}
def __init__(self, id, version=12, flags=0, operation=0x01):
self.id = id
self.version = version
self.flags = flags
self.operation = operation
- Not pretty
- No basic type checking or validation, unless performed manually
The SQLAlchemy Way
import messenger as m
class Message(m.Message):
id = m.Uvarlong()
version = m.Byte(default=12)
flags = m.Uvarint(default=0, since_version=11)
operation = m.Byte(default=0x01)
- Class attributes carry metadata information, object attributes contain concrete values
- Basic type checking by default, validation can be done by m.Message super class
- Separation of concerns: validation, encoding/decoding, transmitting etc. can be done by different objects
Implementation of Datatypes
class DataType(object):
def __init__(self, **kwargs):
for key, value in kwargs.iteritems():
setattr(self, key, value)
@property
def type(self):
return type(self).__name__.lower()
class Byte(DataType):
pass
class Uvarint(DataType):
pass
class Uvarlong(DataType):
pass
Implementation of Message Super Class
class Message(object):
def __init__(self, **kwargs):
# Filter out all defined fields that are private and that are not an
# instance of class DataType
m_fields = filter(
lambda f: not f.startswith('__') and
isinstance(getattr(self, f), DataType), dir(self))
for f_name in m_fields:
f_cls = getattr(self.__class__, f_name)
# If someone used key-word arguments in the initializer,
# let's pass them as the default values
if f_name in kwargs:
setattr(self, f_name, kwargs[f_name])
# If there is a default available, use that
elif hasattr(f_cls, 'default'):
setattr(self, f_name, f_cls.default)
# If all that fails, just initialize to None
else:
setattr(self, f_name, None)
@property
def cls(self):
return self.__class__
What About Order?
class CreatedCounter(object):
_count = 0
@staticmethod
def count():
count = CreatedCounter._count
CreatedCounter._count += 1
return count
class DataType(object):
def __init__(self, **kwargs):
self._created = CreatedCounter.count()
# ...
Python doesn't remember order of definiton, easy to fix:
class Message(object):
def __init__(self, **kwargs):
m_fields = filter(
lambda f: not f.startswith('__') and
isinstance(getattr(self, f), DataType), dir(self))
self.fields = \
sorted(m_fields, key=lambda fn: getattr(self, fn)._created)
for f_name in self.fields:
# ...
What's Next?
class Composite(DataType):
pass
class List(DataType):
pass
class Bytes(DataType):
def __init__(self, size, **kwargs):
super(Bytes, self).__init__(**kwargs)
self.args.append(size)
class Varbytes(DataType):
pass
class Ushort(DataType):
pass
- More types, composite types, lists, types with arguments:
- Encoder/decoder implementation that interprets the matadata
Real World Example
class RequestHeader(m.Message):
magic = m.Byte(default=0xA0)
id = m.Uvarlong()
version = m.Byte(default=25)
op = m.Byte()
cname = m.String(optional=True)
flags = m.Uvarint(default=0)
ci = m.Byte(default=ClientIntelligence.BASIC)
t_id = m.Uvarint(default=0)
class Request(m.Message):
header = m.Composite(default=RequestHeader)
def __init__(self, **kwargs):
super(Request, self).__init__(**kwargs)
self.header.op = self.OP_CODE
class GetRequest(Request):
OP_CODE = 0x03
key = m.Varbytes()
- Taken from: https://github.com/VaclavDedik/infinispan-py
Easy to Add New Functionality
@@ -229,6 +221,29 @@
OP_CODE = 0x14
+class StatsRequest(Request):
+ OP_CODE = 0x15
+
+
+class Stat(m.Message):
+ name = m.String()
+ value = m.String()
+
+
+class StatsResponse(Response):
+ OP_CODE = 0x16
+ n = m.Uvarint()
+ stats = m.List(of=Stat, size=lambda s: s.n)
+
+
class ErrorResponse(Response):
OP_CODE = 0x50
error_message = m.String()
@VaclavDedik
Thanks for listening.
Questions?
Defining Messages For a Client/Server Protocol Implementation
By Václav Dedík
Defining Messages For a Client/Server Protocol Implementation
- 422