Ward Wiz:
Using Python to look up civic data
Problem:
Need to look up city ward information for many Minneapolis street addresses.
Existing tool: Ward Finder Address Search
Disadvantage: must enter one address at a time. What if you have hundreds of addresses?
Solution: Ward Wiz
- Google Cloud application
- Uses Python 2.7 and Webapp2
- User pastes a list of Minneapolis street addresses into the textbox.
- User enters his or her email address and clicks Submit.
- Application sends requests to the City of Minneapolis web site in the background.
- Application sends email to user when results are complete.
Code for looking up city wards (wards.py)
import urllib
import urllib2
import re
from google.appengine.api import memcache
URL = "http://apps.ci.minneapolis.mn.us/AddressPortalApp/Search/SearchPOST?AppID=WardFinderApp"
pattern = re.compile(r"/ward([0-9]+)/")
# Look up the ward for a Minneapolis street address.
def get_ward(street_address):
street_address = normalize(street_address)
ward = memcache.get(street_address)
if ward is not None:
return ward
values = {'Address': street_address}
data = urllib.urlencode(values)
ward = 'NA'
req = urllib2.Request(URL, data)
try:
response = urllib2.urlopen(req)
url = response.geturl()
except:
return 'NA'
match = re.search(pattern, url)
if match:
ward = match.group(1)
memcache.set(street_address, ward)
return ward
def normalize(street):
street = street.strip().upper()
for stopword in (' MINNEAPOLIS', ' MPLS', ' APT ', ' APT.', ' ROOM ',
' UNIT ', '#', ' NO '):
if stopword in street:
index = street.index(stopword)
street = street[:index].strip()
return street
Main application code (main.py)
import webapp2
import urllib
from google.appengine.api import mail
from google.appengine.api import taskqueue
import wards
WEB_FORM = """\
<html>
<head>
<title>Ward Wiz</title>
</head>
<body>
<h1>Ward Wiz</h1>
<p>This service looks up the wards for Minneapolis street addresses,
and returns the results by email. </p>
<p>Results may take several minutes to arrive, so please be patient.</p>
<p><strong>Tips:</strong> Include the house number, street name, and direction.<p>
<p>Do not include apartment number, city, state, or zip code.</p>
<p>This service uses the
<a href="http://apps.ci.minneapolis.mn.us/AddressPortalApp/?AppID=WardFinderApp">City
of Minneapolis ward finder application</a>.</p>
<form action="/enqueue" method="post">
<div>Email: <input type="text" name="user"></div>
<br>
Enter Minneapolis street addresses (one per line) <br>
<div><textarea name="streets" rows="20" cols="60" placeholder="2799 1st Ave SE"></textarea></div>
<div><input type="submit" value="submit"></div>
</form>
</body>
</html>
"""
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.write(WEB_FORM)
class Enqueue(webapp2.RequestHandler):
def post(self):
streets = self.request.get('streets')
user = self.request.get('user')
taskqueue.add(params={'user': user, 'streets': streets})
self.redirect('/')
class TaskRunner(webapp2.RequestHandler):
def post(self):
user_address = self.request.get('user')
streets = self.request.get('streets')
if user_address and streets:
streets = streets.split('\n')
output = []
for street in streets:
street = street.strip()
if street:
ward = wards.get_ward(street)
output.append("%s,%s" % (street, ward))
sender_address = "dradcliffe@gmail.com"
subject = "Your ward information"
body = "\n".join(output)
mail.send_mail(sender_address, user_address, subject, body)
application = webapp2.WSGIApplication([
('/', MainPage),
('/enqueue', Enqueue),
('/_ah/queue/default', TaskRunner)
], debug=False)application: ward-wiz
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /favicon\.ico
static_files: static/favicon.ico
upload: static/favicon\.ico
- url: /.*
script: main.application
libraries:
- name: webapp2
version: latest
Configuration file: app.yaml
Alternative method
- Look up latitude and longitude for each address. (Geocoding)
- Compare the latitudes and longitudes with the ward boundaries.
- Why? Doing the math ourselves is faster and more reliable than sending hundreds of POST requests.
Geocoding in Python is Easy
import geocoder
address = "1501 Hennepin Ave E, Minneapolis MN"
g = geocoder.google(address)
latitude, longitude = g.latlngBut how do we find the ward boundaries? Open Data Minneapolis!
Ward boundary data
- Ward boundaries are available in GeoJSON format.
- Each ward is represented by a polygon.
- A polygon is a list of [lon, lat] pairs
- Crossing test: A point is inside a polygon if and only if a ray from that point crosses the polygon an odd number of times.
Ward finder using GeoJSON
import json
def load_polygons(filename="City_Council_Wards.json"):
polygons = {}
with open(filename) as f:
wards = json.load(f)
for feature in wards['features']:
ward = feature["properties"]["BDNUM"]
coordinates = feature["geometry"]["coordinates"][0]
polygons[ward] = coordinates
return polygons
def point_in_polygon(x, y, poly):
p = [(x1 - x, y1 - y) for x1, y1 in poly]
x2, y2 = p[0]
crossings = 0
for i in xrange(1, len(p)):
x1, y1 = x2, y2
x2, y2 = p[i]
s = sgn(y1)
if s != sgn(y2) and (x1>=0 or x2>=0) and sgn(x2*y1-x1*y2) == s:
crossings += 1
return crossings % 2 == 1
def get_ward_by_lng_lat(lng, lat, polygons):
for ward, poly in polygons.iteritems():
if point_in_polygon(lng, lat, poly):
return ward
return 'NA'
def sgn(x):
if x < 0:
return -1
return 1Looking up City Wards with Python & Google App Engine
By David Radcliffe
