Generators, Comprehensions and Iterable data structures

Kalil de Lima - Python Meetup

https://github.com/kaozdl

kalil.de.lima@fing.edu.uy

kalil@rootstrap.com

What's an iterable?

An iterable is an object that represents a collection of data. Such collection should be able to provide his elements one at a time.

Implements the __iter__() method
Implements the __next__() method
Can be looped over with for ... in ... :
A list can be generated from such object

Properties

A quick example

class MyNumbers:
  def __iter__(self):
    self.a = 1
    return self

  def __next__(self):
    if self.a <= 20:
      x = self.a
      self.a += 1
      return x
    else:
      raise StopIteration

myclass = MyNumbers()
myiter = iter(myclass)

#Now we can iterate with this
for x in myiter:
  print(x)

#Or with
for x in MyNumbers():
  print(x)

Generators

A generator is a function with a yield statement.

Unlike normal functions, the values of the local variables are not destroyed after the function yields, preserving the state of execution.

This defines an iterator with the following advantages:

Generators

Implements __next__ and __iter__ automatically
Iteration stop is taken care of automatically
Iteration related exceptions are handled
Nicer syntax in most usual cases
Does not require to define a class

Some examples

#All the even natural numbers
def all_even():
    n = 0
    while True:
        yield n
        n += 2

#Circular lists
def circular(start, end):
    current = start
    while True:
        yield current
        if current == end:
           current = start
        else:
           current += 1

Generator Expressions

Quick way of defining simple generators
Does not need to declare a function
Returns an iterable so they can be looped over immediately

Some Examples

#A nice alternative to map
squares = (x*x for x in values)

#Useful for filtering
even_squares = (x*x for x in values if x % 2 == 0)

#Can be iterated right away
for address in (a in addresses if 'wes' in a):
    print(address)

#Can be fed to builtins right away
{ x: x*x for x in values }

Why should i use all this?

Reduces memory footprint
Improves readability for simple situations
Allows quick filtering due to if clause
Similar to map for quick transformations
Easy to read notation
Accepted as arguments for most of the iterable builtins like dict, str and list
Looks Cool

When should I use this?

Whenever you need to process elements one by one
When you can't load a whole list of elements in memory for processing
When you need lazy loading
When an element in the iterator generates the next
When you only want to work with a subset of an iterable that you already have

When shouldn't I?

When you need to iterate back and forth
When you need to go several times trough the same iterable
When you can solve it with a list
When you can't solve it with one line

A quick Bonus

Even tough is not released yet, Python 3.8 introduces assignment expressions, which can be used inside generator expressions to improve the readability and memoize function calls

#Now
some_list = [
    generate_expensive_element(value)
    for value in some_iterable 
        if meets_condition(generate_expensive_element(value))
    ]


#In python 3.8
some_list = [
    (elem := generate_expensive_element(value)) #Element gets memoized
    for value in some_iterable 
        if meets_condition(elem)
    ]

Thanks for listening!

All code examples are available at:

https://github.com/kaozdl/python_meetup