Pythonic monads in real life

               PyConfr, 2018/10/7

  • Vincent Perez, Developer @ Legalstart
     
  • Python dev for (almost) 3 years
     
  • Big fan of functional programming
     

The goals of this talk are to show:

 

  • Monad is a simple (but powerful!) concept
     
  • Monads can be leveraged in Python

Monads are known to be a scary concept, but..

Introductory example

Pb: Timed functions

from functools import wraps
from time import sleep, time


# Decorator to time function execution
def time_it(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        start = time()
        result = f(*args, **kwargs)
        end = time()
        return result, end - start
    return wrapper

Pb: Timed functions

# let's define some functions

@time_it
def fast(x):
    return x + 1


@time_it
def slow(x):
    sleep(0.1)
    return x + 1


@time_it
def slow2(x):
    sleep(0.1)
    return x + 2

How to chain these and get the total time?

Pb: Timed functions

# Method 1: the obvious way

x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2

How to chain these and get the total time?

Pb: Timed functions

What could be improved here?

  • Need to repeat the logic of unpacking value + time
x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

Pb: Timed functions

What's wrong here?

  • Need to repeat summation logic
total_time = time0 + time1 + time2

Pb: Timed functions (with bind)

def bind(value_and_time, f):
    """
    :param value_and_time: Tuple[T, float]
    :param f: Callable[[T], Tuple[U, float]]
    :rtype: Tuple[U, float]
    """
    result, t = f(value_and_time[0])
    return result, t + value_and_time[1]




x2, t = bind(bind(fast(1), slow), slow2)

Pb: Timed functions (with bind)

def bind(value_and_time, f):
    """
    :param value_and_time: Tuple[T, float]
    :param f: Callable[[T], Tuple[U, float]]
    :rtype: Tuple[U, float]
    """
    result, t = f(value_and_time[0])
    return result, t + value_and_time[1]


x2, t = bind(bind(fast(1), slow), slow2)
  • value & time unboxing / time summation are encoded once in the bind function :) 
  • Nested binds can be a bit harder to read

 

Pb: Timed functions (with bind + object notation)

class TimedValue(object):

    def __init__(self, value, time=0):
        self.value = value
        self.time = time

    def bind(self, f):
        timed_value = f(self.value)
        new_value = timed_value.value
        new_time = self.time + timed_value.time
        return TimedValue(new_value,  new_time)

Pb: Timed functions (with bind + object notation)

class TimedValue(object):
    
    ...
     
    def bind(self, f):
        timed_value = f(self.value)
        new_value = timed_value.value
        new_time = self.time + timed_value.time
        return TimedValue(new_value, new_time)


# Keep same functions as before, but change the decorator
def time_it(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        start = time()
        result = f(*args, **kwargs)
        end = time()
        return TimedValue(result, end - start)
    return wrapper

Pb: Timed functions (with bind + object notation)

timed_value = (
    fast(1)
    .bind(slow)
    .bind(slow2)
)

value = timed_value.value
time = timed_value.time

Same as before, but chaining methods instead of nesting functions. 

Pb: Timed functions (with bind + object notation)

But, why not create one big function doing all of this?

@time_it
def composed_function(x):
    x0 = x + 1
    sleep(0.1)
    x1 = x0 + 1
    sleep(0.1)
    return x1 + 2
  • Loss of time tracking for smaller functions
  • We could do something else than summing in bind (eg averaging time, taking the max, etc..)

What did we just do ?

We just invented a monad (the TimedValue class)!

Monad formalization

Our friend the monad

What is a monad?

  • The definition of a way to combine functions / abstractions (embodied by a bind function)
     
  • Analogy: monadic value -> amplified / boxed value (eg timed values, list of values...)

Our friend the monad

What is it good for?

 

  • Make composition of functions easier
     
  • Avoid repeating computational patterns
     
  • Particularly useful when pipelining operations
     

Our friend the monad, formal introduction

How to define a monad:

 

  • define a type
  • define unit: a -> M a

Our friend the monad, formal introduction: unit

class TimedValue(object):

    def __init__(self, value, time=0):
        self.value = value
        self.time = time

    @classmethod
    def unit(cls, value):
        return cls(value)

Our friend the monad, formal introduction

How to define a monad:

 

  • define a type
  • define unit: a -> M a
  • define bind: M a -> (a -> M b) -> M b 

Our friend the monad, formal introduction: bind

Normal composition:

Our friend the monad, formal introduction: bind

Composition with bind:

Our friend the monad, formal introduction

How to define a monad:

 

  • define a type
  • define unit: a -> M a
  • define bind: M a -> (a -> M b) -> M b 
  • define map​: (a -> b) -> M a -> M b

Our friend the monad, formal introduction: map

Our friend the monad, formal introduction: map

class TimedValue(object):

    # here f is a function which returns a plain value
    def map(self, f):
        return TimedValue(f(self.value), self.time)

Another example

Another example: the Maybe monad

user = props.user
friends = user.friends if user else None
first_friend = friends[0] if len(friends) > 0 else None
friends_of_first_friend = first_friend.friends if first_friend else None
  • Repeating the if ... else ... none guard => can we abstract this away with a monad ?

Another example: the Maybe monad

Maybe Monad: rationale

  • Put value in a box
     
  • two kinds of box: full (real value) or empty (None)
     
  • if an empty box is encountered during a computation pipeline, just forward the empty box 

Another example: the Maybe monad

class Maybe(object):

    def __init__(self, value):
        self.value = value

    @classmethod
    def unit(cls, value):
        return cls(value)

    def bind(self, f):
        if self.value is None:
            return self  # forward the empty box
        return f(self.value)

    def map(self, f):
        if self.value is None:
            return self  # forward the empty box
        new_value = f(self.value)
        return Maybe.unit(new_value)

Another example: the Maybe monad

def first_value(values):
    if len(values) > 0:
        return values[0]
    return None

friends_of_first_friends = (
    Maybe.unit(props)
    .map(lambda props: props.user)
    .map(lambda user: user.friends)
    .map(first_value)
    .map(lambda first_friend: first_friend.friends)
)
  • we can chain functions without None guards (done once in map) :)
  • Can we be more concise?

Another example: the Maybe monad

# Express attribute access as a function
def getattr_func(attr_name):
    return lambda obj: getattr(obj, attr_name)

friends_of_first_friends = (
    Maybe.unit(props)
    .map(getattr_func('user'))
    .map(getattr_func('friends'))
    .map(first_value)
    .map(getattr_func('friends'))
)

Comprehending Monads (Bonus)

Monad comprehensions

Let's reconsider our TimedValue example 

x0, time0 = fast(1)
x1, time1 = slow(x0)
value, time2 = slow2(x1)

total_time = time0 + time1 + time2

What if we don't have a linear pipeline?  (eg the final value depends on x0)

final_value = x0 * value

Monad comprehensions

Let's reconsider our TimedValue example , monadic version

timed_value = (
    fast(1)
    .bind(slow)
    .bind(slow2)
)

final_value = timed_value.value
time = timed_value.time

Monad comprehensions

Let's reconsider our TimedValue example , monadic version

timed_value = (
    fast(1)
    .bind(
        lambda x: slow(x)
        .bind(slow2)
        .map(lambda y: x * y)
    )
)

final_value = timed_value.value
time = timed_value.time

Can we have a friendlier syntax ?

Monad comprehensions

Little detour: Lists

def unit(x):
    return [x]


def bind(l, f):
    return [
        y
        for x in l
        for y in f(x)
    ]

Lists can be seen as a monad

Monad comprehensions

Lists can be seen as a monad

def unit(x):
    return [x]


def bind(l, f):
    return [
        y
        for x in l
        for y in f(x)
    ]

# Example
def f(x):
    return [-x, x]


bind([1, 2], f)  # outputs [-1, 1, -2, 2]

Monad comprehensions

Lists comprehensions

[   
    h(x, y, z)
    for x in some_list
    for y in f(x)
    for z in g(x, y)
]
  • Each "for" can use variables defined by a previous "for"
  • the expression at the top can use all of them
  • => context gradually augmented by each "for"
  • => this syntax looks like a great candidate
  • But we can't overload list comprehensions in Python ..

Monad comprehensions

A list comprehension can be written in terms of unit and bind

# example 1: a simple list
l = [1, 2]

[x for x in l] == bind(l, lambda x: unit(x))

Monad comprehensions

A list comprehension can be written in terms of unit and bind

# example 1: a simple list
l = [1, 2]

[x for x in l] == bind(l, lambda x: unit(x))


# example 2: a list of lists
ll = [[1, 2], [3, 4]]

[2*y for x in ll for y in x] == \
    bind(ll, lambda x: bind(x, lambda y: unit(2*y)))

Monad comprehensions

 

 

  • A list comprehension can be expressed in terms of bind and unit from the list monad, and conversely
  • I can now overload the behaviour of lists comprehensions by specifying arbitrary unit and bind functions
  • But how ?

Monad comprehensions

We can perform an ast transformation!

 

 

Original Code (list comprehension syntax)

 

We can then plug this transformation into a function decorator

-> ast

-> new ast

-> new code (bind expression)

Monad comprehensions

Demo #1

 

# AST transformations wrapped into a function decorator

@awesome(Maybe)
def f():
    return [
        (x + y)
        for x in Maybe(5)
        for y in Maybe(6)
    ]

# outputs Maybe (11)
  • The function returns a Maybe! (not a list)
  • for x in Maybe(5) reads as
    x = Maybe(5).value

Monad comprehensions

Demo #1

 

# AST transformations wrapped into a function decorator

@awesome(Maybe)
def f():
    return [
        (x + y)
        for x in Maybe(None)
        for y in Maybe(6)
    ]

# outputs Empty

Monad comprehensions

Demo #2

 

# AST transformations wrapped into a function decorator

@awesome(TimedValue)
def f():
    return [
        x0 * x2
        for x0 in fast(1)
        for x1 in slow(x0)
        for x2 in slow2(x1)
    ]
  • The function returns a TimedValue! (not a list)
  • for x0 in fast(1) reads as
    x0 = fast(1).value

Conclusion

Conclusion

Take aways

  • Monad is a simple concept
  • (Simple) monads are simple to implement
  • Not as alien as one may think
    • LINQ query syntax is very close to Monad comprehensions (C#)
    • Promises: monad for async programming (Javascript)
  • Bonus: Python provides a syntax for monadic computations (with list comprehensions)

Conclusion

Unexplored territory

  • IO / Asynchronous programming (IO monad, promises, ...)
  • Exceptions / Failure handling (Error, Either ..)
  • ...

Conclusion

References (Thanks to these guys):

  • Dan Piponi:
    • http://blog.sigfpe.com/2012/03/overloading-python-list-comprehension.html
    • http://blog.sigfpe.com/2006/08/you-could-have-invented-monads-and.html
  • wesdyer
    • https://blogs.msdn.microsoft.com/wesdyer/2008/01/10/the-marvels-of-monads/
  • Alexander Schepanovski
    • http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/

 

Conclusion

Illustrations:

http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html

Conclusion

Find the code on github:

https://github.com/v-prz/monad_comprehension.py

  • Online product to make incorporations (and other things) in France easier and cheaper  
  • We're Hiring!
    https://www.legalstart.fr/corp/recrutement/

Thank you!

Monad comprehensions

A list comprehension can be written in terms of unit and bind

# General case: a list comprehension looks approximatively like this 
# (ifs omitted)
[
    selector
    for id_1 in query_1
    for id_2 in query_2
    ...
    for id_n in query_n
]

Monad comprehensions

A list comprehension can be written in terms of unit and bind

# General case: a list comprehension looks approximatively like this 
# (ifs omitted)
[
    selector
    for id_1 in query_1
    for id_2 in query_2
    ...
    for id_n in query_n
]

# We can rewrite this comprehension as:

bind(query_1,
     lambda id_1: bind(query_2,
        lambda id_2: bind(query_3,
        ...
            lambda id_n: unit(selector)
            )
        )
        ...
)

Monad comprehensions

# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):    
    
    def visit_ListComp(self, node):
        def build_call(generators, elt):
            if generators == []:
                return ast.Call(
                    func=ast.Name(id='__unit__'),
                    args=[node.elt],
                    keywords=[]
                )
            else:
                first_generator, *rest = generators
                return ast.Call(
                    func=ast.Name(id='__bind__'),
                    args=[
                        first_generator.iter,
                        ast.Lambda(
                            args=[first_generator.target.id],                                
                            body=build_call(rest, elt)
                        )
                    ],
                    keywords=[]
                )

        return build_call(node.generators, node.elt)

Monad comprehensions

# Use of a node transformer
# simplified code
class ComprehensionTransformer(ast.NodeTransformer):    
    
    def visit_ListComp(self, node):
        def build_call(generators, elt):
            if generators == []:
                return ast.Call(
                    func=ast.Name(id='__unit__'),
                    args=[node.elt],
                    keywords=[]
                )
            else:
                first_generator, *rest = generators
                return ast.Call(
                    func=ast.Name(id='__bind__'),
                    args=[
                        first_generator.iter,
                        ast.Lambda(
                            args=[first_generator.target.id],                                
                            body=build_call(rest, elt)
                        )
                    ],
                    keywords=[]
                )

        return build_call(node.generators, node.elt)

base case : unit call

lambda : recursive call

Bind call

Monad comprehensions

def madness(monad_cls):
    def decorator(f):
        # Decompile the code
        source = inspect.getsource(f)
        tree = ast.parse(source)

        # transform the ast
        tree.body[0] = ComprehensionTransformer().visit(tree.body[0])

        ast.fix_missing_locations(tree)

        # Recompile it
        code = compile(tree, '', 'exec')
        globs = {
            **f.__globals__,
            '__unit__': monad_cls.unit,
            '__bind__': monad_cls.bind,
        }
        context = {}

        # exec the code (redefines the function)
        exec(code, globs, context)
        return context[f.__name__]
    return decorator
Made with Slides.com