Fosdem 02/01/2020

Vincent Perez

• Vincent Perez,  Software engineer
• ~ 4 years exp in Python
• Big fan of functional programming

The goals of this talk are to show:

• Monad is a simple (but powerful!) concept

• Monads can be leveraged in Python

Monads are known to be a scary concept, but..

Warning: Will rely more on examples than theory

# Motivating example

Pb: Timed functions

``````from functools import wraps
from time import sleep, time

# Decorator to time function execution
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return result, end - start
return wrapper``````

Pb: Timed functions

``````# let's define some functions

@time_it
def fast(x):
return x + 1

@time_it
def slow(x):
sleep(0.1)
return x + 1

@time_it
def slow2(x):
sleep(0.1)
return x + 2``````

How to chain these and get the total time?

Pb: Timed functions

``````# Method 1: the obvious way

x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2``````

How to chain these and get the total time?

Pb: Timed functions

What could be improved here?

• Need to repeat the logic of unpacking value + time
``````x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)``````

Pb: Timed functions

What could be improved ?

• Need to repeat summation logic
``total_time = time0 + time1 + time2``

Pb: Timed functions (with bind)

``````def bind(value_and_time, f):
"""
:param value_and_time: Tuple[T, float]
:param f: Callable[[T], Tuple[U, float]]
:rtype: Tuple[U, float]
"""
result, t = f(value_and_time[0])
return result, t + value_and_time[1]``````
``````

x2, t = bind(bind(fast(1), slow), slow2)``````

Pb: Timed functions (with bind)

``````def bind(value_and_time, f):
return result, t + value_and_time[1]

x2, t = bind(bind(fast(1), slow), slow2)``````
• value & time unboxing / time summation are encoded once in the bind function :)
• Nested binds can be a bit harder to read

Pb: Timed functions (with bind + object notation)

``````class TimedValue(object):

def __init__(self, value, time=0):
self.value = value
self.time = time

def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value,  new_time)``````

Pb: Timed functions (with bind + object notation)

``````class TimedValue(object):
...
def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value, new_time)

# Keep same functions as before, but change the decorator
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return TimedValue(result, end - start)
return wrapper``````

Pb: Timed functions (with bind + object notation)

``````timed_value = (
fast(1)
.bind(slow)
.bind(slow2)
)

value = timed_value.value
time = timed_value.time``````

Same as before, but chaining methods instead of nesting functions.

What did we just do ?

We just invented a monad (the TimedValue class)!

# But... what is it ?

• Monad is a general concept (with TimedValue being one instance of this concept)
• It can be seen as a design pattern to help us compose functions with "effects"
• Analogy: monadic value -> amplified / boxed value  / value with context (eg timed values, list of values...)

What is it good for?

• Make composition of functions easier

• Avoid repeating computational patterns

• Particularly useful when pipelining operations

Our friend the monad, formal introduction

• define a unit function (or constructor): plain value -> monadic value
• define a bind function: for applying a function to a monadic value
• unit and bind must respect some properties (not covered here)

``````class TimedValue(object):

def __init__(self, value, time=0):
self.value = value
self.time = time

@classmethod
def unit(cls, value):
return cls(value)``````

``````class TimedValue(object):

def __init__(self, value, time=0):
self.value = value
self.time = time

def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value,  new_time)``````

# Another example

``````user = props.user
friends = user.friends if user else None
first_friend = friends[0] if len(friends) > 0 else None
friends_of_first_friend = (
first_friend.friends if first_friend else None
)``````
• Repeating the if ... else ... none guard => can we abstract this away with a monad ?

• two kind of values: full or empty

• if an empty value  is encountered during a computation pipeline, just forward it
• This allows us not to check for `None` at every step

``````class Maybe(object):
def __init__(self, value):
self.value = value

@classmethod
def unit(cls, value):
return cls(value)``````
``````
def bind(self, f):
if self.value is None:
return self  # forward the empty value

result = f(self.value)
if isinstance(result, Maybe):
return result
else:
# note to monad experts: conflating bind and map
return Maybe.unit(result)``````

``````def first_value(values):
if len(values) > 0:
return values[0]
return None

friends_of_first_friends = (
Maybe.unit(props)
.bind(lambda props: props.user)
.bind(lambda user: user.friends)
.bind(first_value)
.bind(lambda first_friend: first_friend.friends)
)``````
• We can chain functions without None guards (done once in bind) :)
• Can we be more concise?
• Yes, by leveraging the dynamism of Python!

``````# instead of writing this:
Maybe.unit(obj).bind(lambda obj: obj.method())
# we would like to write
Maybe.unit(obj).method()
# we can probably hack something with __getattr__ ...``````

Text

``````def __getattr__(self, name):
field = getattr(self.value, name)
if not callable(field):
return self.bind(lambda _: field)
return lambda *args, **kwargs: self.bind(
lambda _: field(*args, **kwargs)
)

``````

``````def first_value(values):
if len(values) > 0:
return values[0]
return None

friends_of_first_friends = (
Maybe.unit(props)
.user
.friends
.bind(first_value)
.friends
)
``````

Let's reconsider our TimedValue example

``````x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2``````

What if we don't have a linear pipeline?  (eg the final value depends on x0)

``final_value = x0 * x2``

Let's reconsider our TimedValue example , monadic version

``````timed_value = (
fast(1)
.bind(
lambda x0: slow(x1)
.bind(slow2)
.bind(lambda x2: x0 * x2)
)
)

final_value = timed_value.value
time = timed_value.time``````

Can we have a friendlier syntax ?

What we would like to write is this:

``````timed_value = [
x0 * x2
for x0 in fast(1)
for x1 in slow(x1)
for x2 in slow2(x2)
]

final_value = timed_value.value
time = timed_value.time``````
• timed_value is a TimedValue! (not a list)
• for x0 in fast(1) reads as x0 = fast(1).value

• This intuition comes from the fact that lists are monads, and a comprehension can be written in terms of List.unit and List.bind
• Idea: generalize comprehensions to all monads
• But Python doesn't allow us to overload the meaning of comprehensions!
• Solution: ast transformations!

Python source code

AST

Python Bytecode

Scan + Parse

Compile

Interpret

NewAst

Ast Transform

Compile

CPython program execution and AST Transformation

Ast transformations with a decorator

``````def monad_comprehension(monad_cls):
"""
`unit` and `bind`
"""
def decorator(f):
"""
:param f: function in which we
"""
# ast transformations ...``````

Demo #1

``````# AST transformations wrapped into a function decorator

def f():
return [
x0 * x2
for x0 in fast(1)
for x1 in slow(x0)
for x2 in slow2(x1)
]
``````

Demo #2

``````# AST transformations wrapped into a function decorator

def f():
return [
(x + y)
for x in Maybe(5)
for y in Maybe(6)
]

# outputs `Just 11```````
• f returns a Maybe! (not a list)
• for x in Maybe(5) reads as
x = Maybe(5).value

Demo #3

``````# AST transformations wrapped into a function decorator

def f():
return [
(x + y)
for x in Maybe(None)
for y in Maybe(6)
]

# outputs the empty value``````

We can now easily handle non linear pipelines \o/

# Conclusion

Conclusion

Take aways

• Monad is a simple concept
• (Simple) monads are simple to implement
• Not as alien as one may think
• Deferred in twisted are a kind of monad
• Maybe monads equivalents are found in Rust, Java ...
• Promises: monad for async programming (Javascript)
• LINQ query syntax is very close to Monad comprehensions (C#)

Conclusion

References (Thanks to these guys):

• Dan Piponi:
• wesdyer
• Alexander Schepanovski
• http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/

Conclusion

Find the code on github:

# Thank you!

https://vincent-prz.github.io/

https://github.com/vincent-prz