Pythonic monads

Fosdem 02/01/2020

Vincent Perez

  • Vincent Perez,  Software engineer
  • ~ 4 years exp in Python
  • Big fan of functional programming

The goals of this talk are to show:

 

  • Monad is a simple (but powerful!) concept
     
  • Monads can be leveraged in Python

 

Monads are known to be a scary concept, but..

Warning: Will rely more on examples than theory

Motivating example

Pb: Timed functions

from functools import wraps
from time import sleep, time


# Decorator to time function execution
def time_it(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        start = time()
        result = f(*args, **kwargs)
        end = time()
        return result, end - start
    return wrapper

Pb: Timed functions

# let's define some functions

@time_it
def fast(x):
    return x + 1


@time_it
def slow(x):
    sleep(0.1)
    return x + 1


@time_it
def slow2(x):
    sleep(0.1)
    return x + 2

How to chain these and get the total time?

Pb: Timed functions

# Method 1: the obvious way

x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2

How to chain these and get the total time?

Pb: Timed functions

What could be improved here?

  • Need to repeat the logic of unpacking value + time
x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

Pb: Timed functions

What could be improved ?

  • Need to repeat summation logic
total_time = time0 + time1 + time2

Pb: Timed functions (with bind)

def bind(value_and_time, f):
    """
    :param value_and_time: Tuple[T, float]
    :param f: Callable[[T], Tuple[U, float]]
    :rtype: Tuple[U, float]
    """
    result, t = f(value_and_time[0])
    return result, t + value_and_time[1]


x2, t = bind(bind(fast(1), slow), slow2)

Pb: Timed functions (with bind)

def bind(value_and_time, f):
    return result, t + value_and_time[1]


x2, t = bind(bind(fast(1), slow), slow2)
  • value & time unboxing / time summation are encoded once in the bind function :) 
  • Nested binds can be a bit harder to read

 

Pb: Timed functions (with bind + object notation)

class TimedValue(object):

    def __init__(self, value, time=0):
        self.value = value
        self.time = time

    def bind(self, f):
        timed_value = f(self.value)
        new_value = timed_value.value
        new_time = self.time + timed_value.time
        return TimedValue(new_value,  new_time)

Pb: Timed functions (with bind + object notation)

class TimedValue(object):    
    ...     
    def bind(self, f):
        timed_value = f(self.value)
        new_value = timed_value.value
        new_time = self.time + timed_value.time
        return TimedValue(new_value, new_time)

# Keep same functions as before, but change the decorator
def time_it(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        start = time()
        result = f(*args, **kwargs)
        end = time()
        return TimedValue(result, end - start)
    return wrapper

Pb: Timed functions (with bind + object notation)

timed_value = (
    fast(1)
    .bind(slow)
    .bind(slow2)
)

value = timed_value.value
time = timed_value.time

Same as before, but chaining methods instead of nesting functions. 

What did we just do ?

We just invented a monad (the TimedValue class)!

But... what is it ?

Our friend the monad

What is a monad?

  • Monad is a general concept (with TimedValue being one instance of this concept)
  • It can be seen as a design pattern to help us compose functions with "effects"
  • Analogy: monadic value -> amplified / boxed value  / value with context (eg timed values, list of values...)

Our friend the monad

What is it good for?

 

  • Make composition of functions easier
     
  • Avoid repeating computational patterns
     
  • Particularly useful when pipelining operations
     

Our friend the monad, formal introduction

How to define a monad:

 

  • define a unit function (or constructor): plain value -> monadic value
  • define a bind function: for applying a function to a monadic value
  • unit and bind must respect some properties (not covered here)

Our friend the monad

class TimedValue(object):

    def __init__(self, value, time=0):
        self.value = value
        self.time = time

    @classmethod
    def unit(cls, value):
        return cls(value)

Our friend the monad

class TimedValue(object):

    def __init__(self, value, time=0):
        self.value = value
        self.time = time

    def bind(self, f):
        timed_value = f(self.value)
        new_value = timed_value.value
        new_time = self.time + timed_value.time
        return TimedValue(new_value,  new_time)

Another example

Another example: the Maybe monad

user = props.user
friends = user.friends if user else None
first_friend = friends[0] if len(friends) > 0 else None
friends_of_first_friend = (
  first_friend.friends if first_friend else None
)
  • Repeating the if ... else ... none guard => can we abstract this away with a monad ?

Another example: the Maybe monad

Maybe Monad: rationale

 

  • two kind of values: full or empty
     
  • if an empty value  is encountered during a computation pipeline, just forward it
  • This allows us not to check for `None` at every step

 

Another example: the Maybe monad

class Maybe(object):
    def __init__(self, value):
        self.value = value

    @classmethod
    def unit(cls, value):
        return cls(value)

     def bind(self, f):
        if self.value is None:
            return self  # forward the empty value          
        
        result = f(self.value)        
        if isinstance(result, Maybe):
            return result
        else:
            # note to monad experts: conflating bind and map
            return Maybe.unit(result)

Another example: the Maybe monad

def first_value(values):
    if len(values) > 0:
        return values[0]
    return None

friends_of_first_friends = (
    Maybe.unit(props)
    .bind(lambda props: props.user)
    .bind(lambda user: user.friends)
    .bind(first_value)
    .bind(lambda first_friend: first_friend.friends)
)
  • We can chain functions without None guards (done once in bind) :)
  • Can we be more concise?
  • Yes, by leveraging the dynamism of Python!

Another example: the Maybe monad

# instead of writing this:
Maybe.unit(obj).bind(lambda obj: obj.method())
# we would like to write
Maybe.unit(obj).method()
# we can probably hack something with __getattr__ ...

Another example: the Maybe monad

Text

def __getattr__(self, name):
        field = getattr(self.value, name)
        if not callable(field):
            return self.bind(lambda _: field)
        return lambda *args, **kwargs: self.bind(
            lambda _: field(*args, **kwargs)
        )

Another example: the Maybe monad

def first_value(values):
    if len(values) > 0:
        return values[0]
    return None


friends_of_first_friends = (
    Maybe.unit(props)
    .user
    .friends
    .bind(first_value)
    .friends
)

Comprehending Monads (Experimental)

Monad comprehensions

Let's reconsider our TimedValue example 

x0, time0 = fast(1)
x1, time1 = slow(x0)
x2, time2 = slow2(x1)

total_time = time0 + time1 + time2

What if we don't have a linear pipeline?  (eg the final value depends on x0)

final_value = x0 * x2

Monad comprehensions

Let's reconsider our TimedValue example , monadic version

timed_value = (
    fast(1)
    .bind(
        lambda x0: slow(x1)
        .bind(slow2)
        .bind(lambda x2: x0 * x2)
    )
)

final_value = timed_value.value
time = timed_value.time

Can we have a friendlier syntax ?

Monad comprehensions

What we would like to write is this:

timed_value = [
  x0 * x2
  for x0 in fast(1)
  for x1 in slow(x1)
  for x2 in slow2(x2)
]

final_value = timed_value.value
time = timed_value.time
  • timed_value is a TimedValue! (not a list)
  • for x0 in fast(1) reads as x0 = fast(1).value

Monad comprehensions

  • This intuition comes from the fact that lists are monads, and a comprehension can be written in terms of List.unit and List.bind
  • Idea: generalize comprehensions to all monads
  • But Python doesn't allow us to overload the meaning of comprehensions!
  • Solution: ast transformations!

Monad comprehensions

Python source code

AST

Python Bytecode

Scan + Parse

Compile

Interpret

NewAst

Ast Transform

Compile

CPython program execution and AST Transformation

Monad comprehensions

Ast transformations with a decorator

def monad_comprehension(monad_cls):
	"""  
    :param monad_cls: A class implementing
    `unit` and `bind`
    """
    def decorator(f):
        """
        :param f: function in which we
        want to overload list comprehensions
        """
        # ast transformations ...

Monad comprehensions

Demo #1

 

# AST transformations wrapped into a function decorator

@monad_comprehension(TimedValue)
def f():
    return [
        x0 * x2
        for x0 in fast(1)
        for x1 in slow(x0)
        for x2 in slow2(x1)
    ]

Monad comprehensions

Demo #2

 

# AST transformations wrapped into a function decorator

@monad_comprehension(Maybe)
def f():
    return [
        (x + y)
        for x in Maybe(5)
        for y in Maybe(6)
    ]

# outputs `Just 11`
  • f returns a Maybe! (not a list)
  • for x in Maybe(5) reads as
    x = Maybe(5).value

Monad comprehensions

Demo #3

 

# AST transformations wrapped into a function decorator

@monad_comprehension(Maybe)
def f():
    return [
        (x + y)
        for x in Maybe(None)
        for y in Maybe(6)
    ]

# outputs the empty value

We can now easily handle non linear pipelines \o/

Conclusion

Conclusion

Take aways

  • Monad is a simple concept
  • (Simple) monads are simple to implement
  • Not as alien as one may think
    • Deferred in twisted are a kind of monad
    • Maybe monads equivalents are found in Rust, Java ...
    • Promises: monad for async programming (Javascript)
    • LINQ query syntax is very close to Monad comprehensions (C#)

Conclusion

References (Thanks to these guys):

  • Dan Piponi:
    • http://blog.sigfpe.com/2012/03/overloading-python-list-comprehension.html
    • http://blog.sigfpe.com/2006/08/you-could-have-invented-monads-and.html
  • wesdyer
    • https://blogs.msdn.microsoft.com/wesdyer/2008/01/10/the-marvels-of-monads/
  • Alexander Schepanovski
    • http://hackflow.com/blog/2015/03/29/metaprogramming-beyond-decency/

 

Conclusion

Find the code on github:

https://github.com/v-prz/monad_comprehension.py

Thank you!

https://vincent-prz.github.io/

https://github.com/vincent-prz

 

Pythonic monads

By v-perez

Made with Slides.com

Pythonic monads

  • 1,006

More from v-perez