# Pythonic monads in real life

PyConfr, 2018/10/7

• Vincent Perez, Developer @ Legalstart

• Python dev for (almost) 3 years

• Big fan of functional programming

The goals of this talk are to show:

• Monad is a simple (but powerful!) concept

• Monads can be leveraged in Python

Monads are known to be a scary concept, but..

# Introductory example

Pb: Timed functions

``````from functools import wraps
from time import sleep, time

# Decorator to time function execution
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return result, end - start
return wrapper``````

Pb: Timed functions

``````# let's define some functions

@time_it
def fast(x):
return x + 1

@time_it
def slow(x):
sleep(0.1)
return x + 1

@time_it
def slow2(x):
sleep(0.1)
return x + 2``````

How to chain these and get the total time?

Pb: Timed functions

``````# Method 1: the obvious way

x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2``````

How to chain these and get the total time?

Pb: Timed functions

What could be improved here?

• Need to repeat the logic of unpacking value + time
``````x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)``````

Pb: Timed functions

What's wrong here?

• Need to repeat summation logic
``total_time = time0 + time1 + time2``

Pb: Timed functions (with bind)

``````def bind(value_and_time, f):
"""
:param value_and_time: Tuple[T, float]
:param f: Callable[[T], Tuple[U, float]]
:rtype: Tuple[U, float]
"""
result, t = f(value_and_time[0])
return result, t + value_and_time[1]

``````
``````

x2, t = bind(bind(fast(1), slow), slow2)``````

Pb: Timed functions (with bind)

``````def bind(value_and_time, f):
"""
:param value_and_time: Tuple[T, float]
:param f: Callable[[T], Tuple[U, float]]
:rtype: Tuple[U, float]
"""
result, t = f(value_and_time[0])
return result, t + value_and_time[1]

x2, t = bind(bind(fast(1), slow), slow2)``````
• value & time unboxing / time summation are encoded once in the bind function :)
• Nested binds can be a bit harder to read

Pb: Timed functions (with bind + object notation)

``````class TimedValue(object):

def __init__(self, value, time=0):
self.value = value
self.time = time

def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value,  new_time)

``````

Pb: Timed functions (with bind + object notation)

``````class TimedValue(object):

...

def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value, new_time)

# Keep same functions as before, but change the decorator
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return TimedValue(result, end - start)
return wrapper``````

Pb: Timed functions (with bind + object notation)

``````timed_value = (
fast(1)
.bind(slow)
.bind(slow2)
)

value = timed_value.value
time = timed_value.time``````

Same as before, but chaining methods instead of nesting functions.

Pb: Timed functions (with bind + object notation)

But, why not create one big function doing all of this?

``````@time_it
def composed_function(x):
x0 = x + 1
sleep(0.1)
x1 = x0 + 1
sleep(0.1)
return x1 + 2``````
• Loss of time tracking for smaller functions
• We could do something else than summing in bind (eg averaging time, taking the max, etc..)

What did we just do ?

We just invented a monad (the TimedValue class)!

• The definition of a way to combine functions / abstractions (embodied by a bind function)

• Analogy: monadic value -> amplified / boxed value (eg timed values, list of values...)

What is it good for?

• Make composition of functions easier

• Avoid repeating computational patterns

• Particularly useful when pipelining operations

Our friend the monad, formal introduction

• define a type
• define unit: a -> M a

Our friend the monad, formal introduction: unit

``````class TimedValue(object):

def __init__(self, value, time=0):
self.value = value
self.time = time

@classmethod
def unit(cls, value):
return cls(value)``````

Our friend the monad, formal introduction

• define a type
• define unit: a -> M a
• define bind: M a -> (a -> M b) -> M b

Our friend the monad, formal introduction: bind

Normal composition:

Our friend the monad, formal introduction: bind

Composition with bind:

Our friend the monad, formal introduction

• define a type
• define unit: a -> M a
• define bind: M a -> (a -> M b) -> M b
• define map: (a -> b) -> M a -> M b

Our friend the monad, formal introduction: map

Our friend the monad, formal introduction: map

``````class TimedValue(object):

# here f is a function which returns a plain value
def map(self, f):
return TimedValue(f(self.value), self.time)

``````

# Another example

``````user = props.user
friends = user.friends if user else None
first_friend = friends[0] if len(friends) > 0 else None
friends_of_first_friend = first_friend.friends if first_friend else None
``````
• Repeating the if ... else ... none guard => can we abstract this away with a monad ?

• Put value in a box

• two kinds of box: full (real value) or empty (None)

• if an empty box is encountered during a computation pipeline, just forward the empty box

``````class Maybe(object):

def __init__(self, value):
self.value = value

@classmethod
def unit(cls, value):
return cls(value)

def bind(self, f):
if self.value is None:
return self  # forward the empty box
return f(self.value)

def map(self, f):
if self.value is None:
return self  # forward the empty box
new_value = f(self.value)
return Maybe.unit(new_value)
``````

``````def first_value(values):
if len(values) > 0:
return values[0]
return None

friends_of_first_friends = (
Maybe.unit(props)
.map(lambda props: props.user)
.map(lambda user: user.friends)
.map(first_value)
.map(lambda first_friend: first_friend.friends)
)``````
• we can chain functions without None guards (done once in map) :)
• Can we be more concise?

``````# Express attribute access as a function
def getattr_func(attr_name):
return lambda obj: getattr(obj, attr_name)

friends_of_first_friends = (
Maybe.unit(props)
.map(getattr_func('user'))
.map(getattr_func('friends'))
.map(first_value)
.map(getattr_func('friends'))
)
``````

Let's reconsider our TimedValue example

``````x0, time0 = fast(1)
x1, time1 = slow(x0)
value, time2 = slow2(x1)

total_time = time0 + time1 + time2``````

What if we don't have a linear pipeline?  (eg the final value depends on x0)

``final_value = x0 * value``

Let's reconsider our TimedValue example , monadic version

``````timed_value = (
fast(1)
.bind(slow)
.bind(slow2)
)

final_value = timed_value.value
time = timed_value.time``````

Let's reconsider our TimedValue example , monadic version

``````timed_value = (
fast(1)
.bind(
lambda x: slow(x)
.bind(slow2)
.map(lambda y: x * y)
)
)

final_value = timed_value.value
time = timed_value.time``````

Can we have a friendlier syntax ?

Little detour: Lists

``````def unit(x):
return [x]

def bind(l, f):
return [
y
for x in l
for y in f(x)
]``````

Lists can be seen as a monad

Lists can be seen as a monad

``````def unit(x):
return [x]

def bind(l, f):
return [
y
for x in l
for y in f(x)
]

# Example
def f(x):
return [-x, x]

bind([1, 2], f)  # outputs [-1, 1, -2, 2]
``````

Lists comprehensions

``````[
h(x, y, z)
for x in some_list
for y in f(x)
for z in g(x, y)
]``````
• Each "for" can use variables defined by a previous "for"
• the expression at the top can use all of them
• => context gradually augmented by each "for"
• => this syntax looks like a great candidate
• But we can't overload list comprehensions in Python ..

A list comprehension can be written in terms of unit and bind

``````# example 1: a simple list
l = [1, 2]

[x for x in l] == bind(l, lambda x: unit(x))
``````

A list comprehension can be written in terms of unit and bind

``````# example 1: a simple list
l = [1, 2]

[x for x in l] == bind(l, lambda x: unit(x))

# example 2: a list of lists
ll = [[1, 2], [3, 4]]

[2*y for x in ll for y in x] == \
bind(ll, lambda x: bind(x, lambda y: unit(2*y)))
``````

• A list comprehension can be expressed in terms of bind and unit from the list monad, and conversely
• I can now overload the behaviour of lists comprehensions by specifying arbitrary unit and bind functions
• But how ?

We can perform an ast transformation!

Original Code (list comprehension syntax)

We can then plug this transformation into a function decorator

-> ast

-> new ast

-> new code (bind expression)

Demo #1

``````# AST transformations wrapped into a function decorator

@awesome(Maybe)
def f():
return [
(x + y)
for x in Maybe(5)
for y in Maybe(6)
]

# outputs Maybe (11)``````
• The function returns a Maybe! (not a list)
• for x in Maybe(5) reads as
x = Maybe(5).value

Demo #1

``````# AST transformations wrapped into a function decorator

@awesome(Maybe)
def f():
return [
(x + y)
for x in Maybe(None)
for y in Maybe(6)
]

# outputs Empty``````

Demo #2

``````# AST transformations wrapped into a function decorator

@awesome(TimedValue)
def f():
return [
x0 * x2
for x0 in fast(1)
for x1 in slow(x0)
for x2 in slow2(x1)
]
``````
• The function returns a TimedValue! (not a list)
• for x0 in fast(1) reads as
x0 = fast(1).value

# Conclusion

Conclusion

Take aways

• Monad is a simple concept
• (Simple) monads are simple to implement
• Not as alien as one may think
• LINQ query syntax is very close to Monad comprehensions (C#)
• Promises: monad for async programming (Javascript)
• Bonus: Python provides a syntax for monadic computations (with list comprehensions)

Conclusion

Unexplored territory

• IO / Asynchronous programming (IO monad, promises, ...)
• Exceptions / Failure handling (Error, Either ..)
• ...

Conclusion

References (Thanks to these guys):

• Dan Piponi:
• wesdyer
• Alexander Schepanovski
• http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/

Conclusion

Illustrations:

Conclusion

Find the code on github:

• Online product to make incorporations (and other things) in France easier and cheaper
• We're Hiring!
https://www.legalstart.fr/corp/recrutement/

# Thank you!

A list comprehension can be written in terms of unit and bind

``````# General case: a list comprehension looks approximatively like this
# (ifs omitted)
[
selector
for id_1 in query_1
for id_2 in query_2
...
for id_n in query_n
]
``````

A list comprehension can be written in terms of unit and bind

``````# General case: a list comprehension looks approximatively like this
# (ifs omitted)
[
selector
for id_1 in query_1
for id_2 in query_2
...
for id_n in query_n
]

# We can rewrite this comprehension as:

bind(query_1,
lambda id_1: bind(query_2,
lambda id_2: bind(query_3,
...
lambda id_n: unit(selector)
)
)
...
)
``````

``````# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):

def visit_ListComp(self, node):
def build_call(generators, elt):
if generators == []:
return ast.Call(
func=ast.Name(id='__unit__'),
args=[node.elt],
keywords=[]
)
else:
first_generator, *rest = generators
return ast.Call(
func=ast.Name(id='__bind__'),
args=[
first_generator.iter,
ast.Lambda(
args=[first_generator.target.id],
body=build_call(rest, elt)
)
],
keywords=[]
)

return build_call(node.generators, node.elt)``````

``````# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):

def visit_ListComp(self, node):
def build_call(generators, elt):
if generators == []:
return ast.Call(
func=ast.Name(id='__unit__'),
args=[node.elt],
keywords=[]
)
else:
first_generator, *rest = generators
return ast.Call(
func=ast.Name(id='__bind__'),
args=[
first_generator.iter,
ast.Lambda(
args=[first_generator.target.id],
body=build_call(rest, elt)
)
],
keywords=[]
)

return build_call(node.generators, node.elt)``````

base case : unit call

lambda : recursive call

Bind call

``````def madness(monad_cls):
def decorator(f):
# Decompile the code
source = inspect.getsource(f)
tree = ast.parse(source)

# transform the ast
tree.body[0] = ComprehensionTransformer().visit(tree.body[0])

ast.fix_missing_locations(tree)

# Recompile it
code = compile(tree, '', 'exec')
globs = {
**f.__globals__,
}
context = {}

# exec the code (redefines the function)
exec(code, globs, context)
return context[f.__name__]
return decorator``````

By v-perez

• 1,946