Pythonic monads in real life
PyConfr, 2018/10/7
- Vincent Perez, Developer @ Legalstart
- Python dev for (almost) 3 years
- Big fan of functional programming
The goals of this talk are to show:
-
Monad is a simple (but powerful!) concept
- Monads can be leveraged in Python
Monads are known to be a scary concept, but..
Introductory example
Pb: Timed functions
from functools import wraps
from time import sleep, time
# Decorator to time function execution
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return result, end - start
return wrapperPb: Timed functions
# let's define some functions
@time_it
def fast(x):
return x + 1
@time_it
def slow(x):
sleep(0.1)
return x + 1
@time_it
def slow2(x):
sleep(0.1)
return x + 2How to chain these and get the total time?
Pb: Timed functions
# Method 1: the obvious way
x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)
total_time = time0 + time1 + time2How to chain these and get the total time?
Pb: Timed functions
What could be improved here?
- Need to repeat the logic of unpacking value + time
x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)Pb: Timed functions
What's wrong here?
- Need to repeat summation logic
total_time = time0 + time1 + time2Pb: Timed functions (with bind)
def bind(value_and_time, f):
"""
:param value_and_time: Tuple[T, float]
:param f: Callable[[T], Tuple[U, float]]
:rtype: Tuple[U, float]
"""
result, t = f(value_and_time[0])
return result, t + value_and_time[1]
x2, t = bind(bind(fast(1), slow), slow2)Pb: Timed functions (with bind)
def bind(value_and_time, f):
"""
:param value_and_time: Tuple[T, float]
:param f: Callable[[T], Tuple[U, float]]
:rtype: Tuple[U, float]
"""
result, t = f(value_and_time[0])
return result, t + value_and_time[1]
x2, t = bind(bind(fast(1), slow), slow2)- value & time unboxing / time summation are encoded once in the bind function :)
- Nested binds can be a bit harder to read
Pb: Timed functions (with bind + object notation)
class TimedValue(object):
def __init__(self, value, time=0):
self.value = value
self.time = time
def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value, new_time)
Pb: Timed functions (with bind + object notation)
class TimedValue(object):
...
def bind(self, f):
timed_value = f(self.value)
new_value = timed_value.value
new_time = self.time + timed_value.time
return TimedValue(new_value, new_time)
# Keep same functions as before, but change the decorator
def time_it(f):
@wraps(f)
def wrapper(*args, **kwargs):
start = time()
result = f(*args, **kwargs)
end = time()
return TimedValue(result, end - start)
return wrapperPb: Timed functions (with bind + object notation)
timed_value = (
fast(1)
.bind(slow)
.bind(slow2)
)
value = timed_value.value
time = timed_value.timeSame as before, but chaining methods instead of nesting functions.
Pb: Timed functions (with bind + object notation)
But, why not create one big function doing all of this?
@time_it
def composed_function(x):
x0 = x + 1
sleep(0.1)
x1 = x0 + 1
sleep(0.1)
return x1 + 2- Loss of time tracking for smaller functions
- We could do something else than summing in bind (eg averaging time, taking the max, etc..)
What did we just do ?
We just invented a monad (the TimedValue class)!
Monad formalization
Our friend the monad
What is a monad?
- The definition of a way to combine functions / abstractions (embodied by a bind function)
- Analogy: monadic value -> amplified / boxed value (eg timed values, list of values...)
Our friend the monad
What is it good for?
- Make composition of functions easier
- Avoid repeating computational patterns
- Particularly useful when pipelining operations
Our friend the monad, formal introduction
How to define a monad:
- define a type
- define unit: a -> M a
Our friend the monad, formal introduction: unit
class TimedValue(object):
def __init__(self, value, time=0):
self.value = value
self.time = time
@classmethod
def unit(cls, value):
return cls(value)Our friend the monad, formal introduction
How to define a monad:
- define a type
- define unit: a -> M a
- define bind: M a -> (a -> M b) -> M b
Our friend the monad, formal introduction: bind
Normal composition:
Our friend the monad, formal introduction: bind
Composition with bind:
Our friend the monad, formal introduction
How to define a monad:
- define a type
- define unit: a -> M a
- define bind: M a -> (a -> M b) -> M b
- define map: (a -> b) -> M a -> M b
Our friend the monad, formal introduction: map
Our friend the monad, formal introduction: map
class TimedValue(object):
# here f is a function which returns a plain value
def map(self, f):
return TimedValue(f(self.value), self.time)
Another example
Another example: the Maybe monad
user = props.user
friends = user.friends if user else None
first_friend = friends[0] if len(friends) > 0 else None
friends_of_first_friend = first_friend.friends if first_friend else None
- Repeating the if ... else ... none guard => can we abstract this away with a monad ?
Another example: the Maybe monad
Maybe Monad: rationale
- Put value in a box
- two kinds of box: full (real value) or empty (None)
- if an empty box is encountered during a computation pipeline, just forward the empty box
Another example: the Maybe monad
class Maybe(object):
def __init__(self, value):
self.value = value
@classmethod
def unit(cls, value):
return cls(value)
def bind(self, f):
if self.value is None:
return self # forward the empty box
return f(self.value)
def map(self, f):
if self.value is None:
return self # forward the empty box
new_value = f(self.value)
return Maybe.unit(new_value)
Another example: the Maybe monad
def first_value(values):
if len(values) > 0:
return values[0]
return None
friends_of_first_friends = (
Maybe.unit(props)
.map(lambda props: props.user)
.map(lambda user: user.friends)
.map(first_value)
.map(lambda first_friend: first_friend.friends)
)- we can chain functions without None guards (done once in map) :)
- Can we be more concise?
Another example: the Maybe monad
# Express attribute access as a function
def getattr_func(attr_name):
return lambda obj: getattr(obj, attr_name)
friends_of_first_friends = (
Maybe.unit(props)
.map(getattr_func('user'))
.map(getattr_func('friends'))
.map(first_value)
.map(getattr_func('friends'))
)
Comprehending Monads (Bonus)
Monad comprehensions
Let's reconsider our TimedValue example
x0, time0 = fast(1)
x1, time1 = slow(x0)
value, time2 = slow2(x1)
total_time = time0 + time1 + time2What if we don't have a linear pipeline? (eg the final value depends on x0)
final_value = x0 * valueMonad comprehensions
Let's reconsider our TimedValue example , monadic version
timed_value = (
fast(1)
.bind(slow)
.bind(slow2)
)
final_value = timed_value.value
time = timed_value.timeMonad comprehensions
Let's reconsider our TimedValue example , monadic version
timed_value = (
fast(1)
.bind(
lambda x: slow(x)
.bind(slow2)
.map(lambda y: x * y)
)
)
final_value = timed_value.value
time = timed_value.timeCan we have a friendlier syntax ?
Monad comprehensions
Little detour: Lists
def unit(x):
return [x]
def bind(l, f):
return [
y
for x in l
for y in f(x)
]Lists can be seen as a monad
Monad comprehensions
Lists can be seen as a monad
def unit(x):
return [x]
def bind(l, f):
return [
y
for x in l
for y in f(x)
]
# Example
def f(x):
return [-x, x]
bind([1, 2], f) # outputs [-1, 1, -2, 2]
Monad comprehensions
Lists comprehensions
[
h(x, y, z)
for x in some_list
for y in f(x)
for z in g(x, y)
]- Each "for" can use variables defined by a previous "for"
- the expression at the top can use all of them
- => context gradually augmented by each "for"
- => this syntax looks like a great candidate
- But we can't overload list comprehensions in Python ..
Monad comprehensions
A list comprehension can be written in terms of unit and bind
# example 1: a simple list
l = [1, 2]
[x for x in l] == bind(l, lambda x: unit(x))
Monad comprehensions
A list comprehension can be written in terms of unit and bind
# example 1: a simple list
l = [1, 2]
[x for x in l] == bind(l, lambda x: unit(x))
# example 2: a list of lists
ll = [[1, 2], [3, 4]]
[2*y for x in ll for y in x] == \
bind(ll, lambda x: bind(x, lambda y: unit(2*y)))
Monad comprehensions
- A list comprehension can be expressed in terms of bind and unit from the list monad, and conversely
- I can now overload the behaviour of lists comprehensions by specifying arbitrary unit and bind functions
- But how ?
Monad comprehensions
We can perform an ast transformation!
Original Code (list comprehension syntax)
We can then plug this transformation into a function decorator
-> ast
-> new ast
-> new code (bind expression)
Monad comprehensions
Demo #1
# AST transformations wrapped into a function decorator
@awesome(Maybe)
def f():
return [
(x + y)
for x in Maybe(5)
for y in Maybe(6)
]
# outputs Maybe (11)- The function returns a Maybe! (not a list)
-
for x in Maybe(5) reads as
x = Maybe(5).value
Monad comprehensions
Demo #1
# AST transformations wrapped into a function decorator
@awesome(Maybe)
def f():
return [
(x + y)
for x in Maybe(None)
for y in Maybe(6)
]
# outputs EmptyMonad comprehensions
Demo #2
# AST transformations wrapped into a function decorator
@awesome(TimedValue)
def f():
return [
x0 * x2
for x0 in fast(1)
for x1 in slow(x0)
for x2 in slow2(x1)
]
- The function returns a TimedValue! (not a list)
-
for x0 in fast(1) reads as
x0 = fast(1).value
Conclusion
Conclusion
Take aways
- Monad is a simple concept
- (Simple) monads are simple to implement
- Not as alien as one may think
- LINQ query syntax is very close to Monad comprehensions (C#)
- Promises: monad for async programming (Javascript)
- Bonus: Python provides a syntax for monadic computations (with list comprehensions)
Conclusion
Unexplored territory
- IO / Asynchronous programming (IO monad, promises, ...)
- Exceptions / Failure handling (Error, Either ..)
- ...
Conclusion
References (Thanks to these guys):
-
Dan Piponi:
- http://blog.sigfpe.com/2012/03/overloading-python-list-comprehension.html
- http://blog.sigfpe.com/2006/08/you-could-have-invented-monads-and.html
-
wesdyer
- https://blogs.msdn.microsoft.com/wesdyer/2008/01/10/the-marvels-of-monads/
-
Alexander Schepanovski
- http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/
Conclusion
Illustrations:
http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html
Conclusion
Find the code on github:
https://github.com/v-prz/monad_comprehension.py
- Online product to make incorporations (and other things) in France easier and cheaper
- We're Hiring!
https://www.legalstart.fr/corp/recrutement/
Thank you!
Monad comprehensions
A list comprehension can be written in terms of unit and bind
# General case: a list comprehension looks approximatively like this
# (ifs omitted)
[
selector
for id_1 in query_1
for id_2 in query_2
...
for id_n in query_n
]
Monad comprehensions
A list comprehension can be written in terms of unit and bind
# General case: a list comprehension looks approximatively like this
# (ifs omitted)
[
selector
for id_1 in query_1
for id_2 in query_2
...
for id_n in query_n
]
# We can rewrite this comprehension as:
bind(query_1,
lambda id_1: bind(query_2,
lambda id_2: bind(query_3,
...
lambda id_n: unit(selector)
)
)
...
)
Monad comprehensions
# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):
def visit_ListComp(self, node):
def build_call(generators, elt):
if generators == []:
return ast.Call(
func=ast.Name(id='__unit__'),
args=[node.elt],
keywords=[]
)
else:
first_generator, *rest = generators
return ast.Call(
func=ast.Name(id='__bind__'),
args=[
first_generator.iter,
ast.Lambda(
args=[first_generator.target.id],
body=build_call(rest, elt)
)
],
keywords=[]
)
return build_call(node.generators, node.elt)Monad comprehensions
# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):
def visit_ListComp(self, node):
def build_call(generators, elt):
if generators == []:
return ast.Call(
func=ast.Name(id='__unit__'),
args=[node.elt],
keywords=[]
)
else:
first_generator, *rest = generators
return ast.Call(
func=ast.Name(id='__bind__'),
args=[
first_generator.iter,
ast.Lambda(
args=[first_generator.target.id],
body=build_call(rest, elt)
)
],
keywords=[]
)
return build_call(node.generators, node.elt)base case : unit call
lambda : recursive call
Bind call
Monad comprehensions
def madness(monad_cls):
def decorator(f):
# Decompile the code
source = inspect.getsource(f)
tree = ast.parse(source)
# transform the ast
tree.body[0] = ComprehensionTransformer().visit(tree.body[0])
ast.fix_missing_locations(tree)
# Recompile it
code = compile(tree, '', 'exec')
globs = {
**f.__globals__,
'__unit__': monad_cls.unit,
'__bind__': monad_cls.bind,
}
context = {}
# exec the code (redefines the function)
exec(code, globs, context)
return context[f.__name__]
return decorator