Sync Norwich

HELLO!

I'm a web developer

By day:

freelance for clients

By night:

make maps, hack on fun stuff

http://anna.ps

HOW I GOT INTO MAPS

DOMESDAY BOOK

DOMESDAY BOOK

SHEEP, COWS AND EELS

The original overambitious Government IT project...

starting out in Norfolk

DOMESDAY INFRASTRUCTURE

GeoDjango + PostGIS database back-end

+

Google Maps / OpenStreetMap front-end

HOUSES PRICES / TRAIN TIMES

THE PROCESS

A. Train times

1. Get list of all train stations in England/Wales/Scotland, with postcode and TLC

2. Scrape traintimes.org.uk for direct peak-time trains

3. Take average of 3 train times

B. House prices

4. Convert postcode to lat/lng

5. Run Nestoria API search on lat/lng: mean price of 3-bed within 2km in last 6 months

C. Graph

6. Graph house price against train times

7. Add moving average line, analyse the results

HOUSE PRICES / TRAIN TIMES

A TALK OF TWO HALVES

1. A brief overview of the landscape

A TALK OF TWO HALVES

2. So, you've got some awesome data... what now?

PART 1: MODERN MAPS

Insanely brief background: maps on the web
Hot topics:

Tools
- Awesome map-making tools - Mapbox, CartoDB
- Map superpowers with D3.js
- WebGL for million-point maps in the browser
Data
- Big OSM data, and big data tools
- Governments releasing open geographic data
- Real-time data

EARLY 1990s: FIRST WEB MAPS

1993: Xerox PARC Map Viewer

Change scale, projections, add marks

1990s/2000S: TILED MAPS IN HTML, Flash, JAvaSCRIPT

Address lookup, aerial photos, map display, routing

1996-2006: Multimap

2005: GOOGLE MAPS -
MAPS 2.0

Slidable, zoomable maps using Ajax

2004-: OPEN STREET MAP

Open map data: the Wikipedia of maps

MODERN MAPS:
AWESOME MAPPING TOOLS

TileMill, Mapbox

MODERN MAPS:
D3.JS

D3: Visualization LIBRARIES

Before D3

BEFORE D3: CHARTING LIBS

Convenient, but tricky to extend or debug

var plot1 = $.jqplot('chart1', [[3,7,9,1,4,6,8,2,5]]);

BEFORE D3: Charting LIBS

Hard to debug using standard web tools

D3.jS: Bind data to the dom

var dataset = [ 5, 10, 13, 19, 21, 25, ...];

var rectangles = 
    svg.selectAll("rect").data(dataset)

rectangles.enter().append("rect")
.attr("width", 24)
.attr("height", function(d) {
    return d * 4;
});

WITH D3: DEBUG IN BROWSER

The D3 Difference

1. Solves the fundamental problem of data visualisation: manipulate the DOM with data

2. Web standards: style with CSS, interact with JS events, debug with DevTools

3. Fast, simple, efficient

4. Sexy animations and transitions

5. Modular code encourages plugins

MODERN MAPS WITH D3

MODERN MAPS:
WEBGL & MILLION-POINT maps

MODErn MAPS: OSM DATA

Map of OSM contributors in Europe

MODERN MAPS:
USING OSM DATA

walks.io - automatic walk generator

MODERN MAPS:
USING OSM DATA

automatically finds walks within a set travel time
from OSM footpaths, stations and scenic-ness data

MODERN MAPS:
OPEN REAL-TIME DATA

http://citymapper.com

MODERN MAPS:
OSM + WEBGL

ViziCities: OSM data in the browser

MODERN MAPS:
OPEN DATA + WEbGL

http://vizicities.com

PART 2:
SO, YOU'VE GOT SOME DATA...

You've got a spreadsheet of some awesome data, and you want to put it onto a map.

What now?

1. Geocoding your data

2. Exploring your data

3. Displaying your data

GEOCODING YOUR DATA

Geographic fields you may have:

Latitude & longitudes
Postal codes
ONS codes
Address strings, town names
Administrative districts - names or codes

Geocoding = geographic fields -> coordinates

LATLNGS

You're very lucky!

Points
Lines
Polygons

... you can start exploring and displaying right away

GOOGLE FUSION TABLES

Launched in 2009 for large data tables

Unbeatable ease for getting started

GOOGLE FUSION TABLES

Sort and filter data, view results on map

GOOGLE FUSION TABLES

Heat maps out of the box

Chloropleths (intensity maps) - merge with boundaries

CARTODB

Google Fusion Tables +++

Hosted geospatial database with PostGIS

Free only up to 25MB

CARTODB

Hexbins and advanced styling,

customisable info-boxes, choice of map backgrounds

CARTODB

Dynamic time maps with Torque

TILEMILL & MAPBOX

TileMill desktop software for Mac and Windows:

Import datasets, design your own maps -

again, no programming skills needed

POSTCODES

Full postcodes should "just work"
in Google Fusion Tables, CartoDB.

But beware postcode accuracy!

NATIONAL GRID (OSGB36)

Coordinate TF0325

find grid TF, 3km east

and 25km north

To get latlng, use UK Postcode API:

http://www.uk-postcodes.com/api.php

ADDRESS STRINGS

King's Centre, King Street, Norwich
King Street, Norwich
Norwich

Can use Google Maps or CartoDB (Nokia) to geocode

Probably won't be 100% accurate

ADMINISTRATIVE DISTRICTS

If your file has long strings like 'E09000001'...

congratulations, those are UK districts!

You'll need to find the boundaries.

HOW?????

uk districts

These can be:

Administrative - councils, districts
Electoral - constituencies, wards
Census - output areas
Health - here be dragons!
Postal - ye olde postcode
Other - police, Europe...

See: A Beginner's Guide to Geography, by the ONS

ADMINISTRATIVE

Boring but crucial.
Get boundaries from ONS (open data).

ONS ADMIN CODES

Woohoo! The most exciting part of our talk!

ONS changed all its data codes in 2011.

From hierarchical short codes:

12 - Cambridgeshire
12UD - Fenland... etc

To nine-letter codes, all starting with E0:

City of London - E09000001

The first two numbers now tell you the type of area.

Your data may have either!

MAPPING ONS CODES

MapIt!

http://mapit.mysociety.org/area/E08000025.html

http://mapit.mysociety.org/area/OOCN.html

gns <-> ons codes

MAPPING ADMIN BOUNDARIES

Can be done with Fusion Tables or CartoDB with a little effort, but for no effort at all... OpenHeatMap

INSTANT SHAREABLE MAPS

Note time slider!

Open HEAT MAP

What OpenHeatMap understands in the UK:

Admin regions:

Council names
Council codes (old codes only)
County names
Region names

Electoral regions:

Constituency names

Constituency codes (old codes only)

CENSUS: OUTPUT AREAS

Designed to be static
Designed to be socially homogenous
OAs are about 150 households, LSOAs = 4-5 OAs
Census/social data typically available at LSOA level

OUTPUT AREAS...

... are cool

OUTPUT AREAS

Get boundaries from the ONS
Shapefile is about 150MB
You can make it smaller with ogr2ogr

POSTCODE -> ADMIN AREA?

MapIt!

BOUNDARY AREAS?

MapIt!

MAKING MAPS WITH D3

... is something to do
after you've explored the data

MAKING MAPS WITH D3

...is fun, but not something
you want to learn in a 20-minute talk

MAKING MAPS WITH D3

Recommendations for starting mapping:

Scott Murray, Interactive Data Visualisation for the Web

Mike Bostock's Let's Make a Map

ARE WE DONE?

Nope!

DISPLAYING YOUR MAP

Projection
Colours
How to lie with maps...

DISPLAYING - PROJECTIONS

Is Greenland really bigger than Australia?

Projection slides from a talk by Dimi Sztanko (@dimi)

DISPLAYING - PROJECTIONS

Nope!

Projection slides from a talk by Dimi Sztanko (@dimi)

MERCATOR PROJECTION

Cylindrical projection from 1569
Great for navigating with a compass...
Horrible for rendering the whole world
"Politically incorrect"

Projection slides from a talk by Dimi Sztanko (@dimi)

WHICH PROJECTION TO USE?

Hundreds to choose from...

Projection slides from a talk by Dimi Sztanko (@dimi)

NO "RIGHT" ANSWER

There is no silver bullet
Every projection has distortions
Depends what you are showing
Depends what do you want to say
Depends on your taste
Ultimate super detailed guide with explanations: http://bit.ly/1iycTul

Projection slides from a talk by Dimi Sztanko (@dimi)

PROJECTIONS FOR COUNTRIES

If your country is elongated north-south...

use transverse mercator

Projection slides from a talk by Dimi Sztanko (@dimi)

PROJECTIONS FOR COUNTRIES

Projection slides from a talk by Dimi Sztanko (@dimi)

PROJECTIONS FOR COUNTRIES

Projection slides from a talk by Dimi Sztanko (@dimi)

PROJECTIONS FOR DISTANCE FROM A SINGLE POINT

Projection slides from a talk by Dimi Sztanko (@dimi)

(USE FOR YOUR MISSILE MAPS)

DISPLAYING - COLOURS

1. Identify the number of data classes

2. Set the nature of your data:

qualitative
diverging
sequential

3. Choose options:

colour-blind safe
print safe

4. Pick a colour scheme

DISPLAYING - MORE...

WHERE TO GET HELP - GEO

http://gis.stackexchange.com

WHERE TO GET HELP - DATA

http://opendata.stackexchange.org

ANy QUESTIONS?

Slides made with:
http://slid.es

Projection slides by Dimi Sztanko:

http://twitter.com/sztanko

Me:
http://anna.ps

http://twitter.com/darkgreener

DRINKS!

PS: Work for mySociety.org (MaPit):

http://mysocietyltd.theresumator.com

web developers + events officer

VISUALISING

GEOGRAPHIC

DATA

Sync Norwich

More from annaps