Ward Wiz:
Using Python to look up civic data

Problem:

Need to look up city ward information for many Minneapolis street addresses.

Existing tool: Ward Finder Address Search

Disadvantage: must enter one address at a time. What if you have hundreds of addresses?

Solution: Ward Wiz

Google Cloud application
Uses Python 2.7 and Webapp2
User pastes a list of Minneapolis street addresses into the textbox.
User enters his or her email address and clicks Submit.
Application sends requests to the City of Minneapolis web site in the background.
Application sends email to user when results are complete.

Code for looking up city wards (wards.py)

import urllib
import urllib2
import re
from google.appengine.api import memcache

URL = "http://apps.ci.minneapolis.mn.us/AddressPortalApp/Search/SearchPOST?AppID=WardFinderApp"

pattern = re.compile(r"/ward([0-9]+)/")

# Look up the ward for a Minneapolis street address.

def get_ward(street_address):
    street_address = normalize(street_address)
    ward = memcache.get(street_address)
    if ward is not None:
        return ward
    
    values = {'Address': street_address}
    data = urllib.urlencode(values)
    ward = 'NA'
    req = urllib2.Request(URL, data)
    
    try:    
        response = urllib2.urlopen(req)
        url = response.geturl()     
    except:
        return 'NA'
        
    match = re.search(pattern, url)
    if match:
        ward = match.group(1)
    memcache.set(street_address, ward)
    return ward
    
def normalize(street):
    street = street.strip().upper()
    for stopword in (' MINNEAPOLIS', ' MPLS', ' APT ', ' APT.', ' ROOM ', 
                     ' UNIT ', '#', ' NO '): 
        if stopword in street:
            index = street.index(stopword)
            street = street[:index].strip()
    return street

Main application code (main.py)

import webapp2
import urllib
from google.appengine.api import mail
from google.appengine.api import taskqueue
import wards

WEB_FORM = """\
<html>
  <head>
    <title>Ward Wiz</title>
  </head>
  <body>
    <h1>Ward Wiz</h1>
    <p>This service looks up the wards for Minneapolis street addresses,
       and returns the results by email. </p>
    <p>Results may take several minutes to arrive, so please be patient.</p>
    <p><strong>Tips:</strong> Include the house number, street name, and direction.<p>
    <p>Do not include apartment number, city, state, or zip code.</p> 
    <p>This service uses the 
    <a href="http://apps.ci.minneapolis.mn.us/AddressPortalApp/?AppID=WardFinderApp">City 
    of Minneapolis ward finder application</a>.</p>
    <form action="/enqueue" method="post">
      <div>Email: <input type="text" name="user"></div>
      <br>
      Enter Minneapolis street addresses (one per line) <br>
      <div><textarea name="streets" rows="20" cols="60" placeholder="2799 1st Ave SE"></textarea></div>
      <div><input type="submit" value="submit"></div>
    </form>
  </body>
</html>
"""

class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.write(WEB_FORM)


class Enqueue(webapp2.RequestHandler):
    def post(self):
        streets = self.request.get('streets')
        user = self.request.get('user')
        taskqueue.add(params={'user': user, 'streets': streets})
        self.redirect('/')


class TaskRunner(webapp2.RequestHandler):
    def post(self):
        user_address = self.request.get('user')
        streets = self.request.get('streets')
        if user_address and streets:
            streets = streets.split('\n')
            output = []
            for street in streets:
                street = street.strip()
                if street:
                    ward = wards.get_ward(street)
                    output.append("%s,%s" % (street, ward))
            sender_address = "dradcliffe@gmail.com"
            subject = "Your ward information"
            body = "\n".join(output)
            mail.send_mail(sender_address, user_address, subject, body)


application = webapp2.WSGIApplication([
    ('/', MainPage),
    ('/enqueue', Enqueue),
    ('/_ah/queue/default', TaskRunner)
], debug=False)

application: ward-wiz 
version: 1
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /favicon\.ico
  static_files: static/favicon.ico
  upload: static/favicon\.ico
- url: /.*
  script: main.application

  
libraries:
- name: webapp2
  version: latest

Configuration file: app.yaml

https://github.com/Radcliffe/WardFinder

Alternative method

Look up latitude and longitude for each address. (Geocoding)
Compare the latitudes and longitudes with the ward boundaries.
Why? Doing the math ourselves is faster and more reliable than sending hundreds of POST requests.

Geocoding in Python is Easy

import geocoder

address = "1501 Hennepin Ave E, Minneapolis MN"
g = geocoder.google(address)
latitude, longitude = g.latlng

But how do we find the ward boundaries? Open Data Minneapolis!

Ward boundary data

Ward boundaries are available in GeoJSON format.
Each ward is represented by a polygon.
A polygon is a list of [lon, lat] pairs
Crossing test: A point is inside a polygon if and only if a ray from that point crosses the polygon an odd number of times.

Ward finder using GeoJSON