VISUALISING

GEOGRAPHIC

DATA

FOR THE WEB

Anna Powell-Smith

HELLO!


I'm a web developer

By day: 
freelance for clients

By night: 
make maps, hack on fun stuff

http://anna.ps

HOW I GOT INTO MAPS


DOMESDAY BOOK


DOMESDAY BOOK


DOMESDAY BOOK


SHEEP, COWS AND EELS


The original overambitious Government IT project...
starting out in Norfolk 


DOMESDAY INFRASTRUCTURE


GeoDjango + PostGIS database back-end
+ 
Google Maps / OpenStreetMap front-end

HOUSES PRICES / TRAIN TIMES


THE PROCESS

A. Train times

1. Get list of all train stations in England/Wales/Scotland, with postcode and TLC

2. Scrape traintimes.org.uk for direct peak-time trains
3. Take average of 3 train times 
B. House prices
4. Convert postcode to lat/lng
5. Run Nestoria API search on lat/lng: mean price of 3-bed within 2km in last 6 months
C. Graph
6. Graph house price against train times
7. Add moving average line, analyse the results

HOUSE PRICES / TRAIN TIMES

A TALK OF TWO HALVES

1. A brief overview of the landscape

A TALK OF TWO HALVES

2. So, you've got some awesome data... what now?

PART 1: MODERN MAPS


  • Insanely brief background: maps on the web
  • Hot topics:
    • Tools
      • Awesome map-making tools - Mapbox, CartoDB
      • Map superpowers with D3.js
      • WebGL for million-point maps in the browser
      Data
      • Big OSM data, and big data tools
      • Governments releasing open geographic data
      • Real-time data 

EARLY 1990s: FIRST WEB MAPS


1993: Xerox PARC Map Viewer
Change scale, projections, add marks

1990s/2000S: TILED MAPS IN HTML, Flash, JAvaSCRIPT


Address lookup, aerial photos, map display, routing
1996-2006: Multimap

2005: GOOGLE MAPS - 
 MAPS 2.0


Slidable, zoomable maps using Ajax

2004-: OPEN STREET MAP


Open map data: the Wikipedia of maps

MODERN MAPS:
AWESOME MAPPING TOOLS


TileMill, Mapbox

MODERN MAPS: 
D3.JS 


D3: Visualization LIBRARIES


Before D3

BEFORE D3: CHARTING LIBS

Convenient, but tricky to extend or debug
var plot1 = $.jqplot('chart1', [[3,7,9,1,4,6,8,2,5]]);

 

BEFORE D3: Charting LIBS

Hard to debug using standard web tools

D3.jS: Bind data to the dom

var dataset = [ 5, 10, 13, 19, 21, 25, ...];

var rectangles = 
    svg.selectAll("rect").data(dataset)

rectangles.enter().append("rect")
.attr("width", 24)
.attr("height", function(d) {
    return d * 4;
});

WITH D3: DEBUG IN BROWSER


The D3 Difference

1. Solves the fundamental problem of data visualisation: manipulate the DOM with data
2. Web standards: style with CSS, interact with JS events, debug with DevTools
3. Fast, simple, efficient
4. Sexy animations and transitions
5. Modular code encourages plugins


MODERN MAPS WITH D3


MODERN MAPS: 
WEBGL & MILLION-POINT maps


MODErn MAPS: OSM DATA


Map of OSM contributors in Europe

MODERN MAPS: 
USING OSM DATA


walks.io - automatic walk generator

MODERN MAPS: 
USING OSM DATA


automatically finds walks within a set travel time
from OSM footpaths, stations and scenic-ness data

MODERN MAPS: 
OPEN REAL-TIME DATA


http://citymapper.com

MODERN MAPS:
OSM + WEBGL


ViziCities: OSM data in the browser

MODERN MAPS: 
OPEN DATA + WEbGL


http://vizicities.com

PART 2:
SO, YOU'VE GOT SOME DATA...


You've got a spreadsheet of some awesome data, and you want to put it onto a map. 

What now?

1. Geocoding your data
2. Exploring your data
3. Displaying your data

GEOCODING YOUR DATA

Geographic fields you may have:

  1. Latitude & longitudes 
  2. Postal codes
  3. ONS codes 
  4. Address strings, town names
  5. Administrative districts - names or codes

Geocoding = geographic fields -> coordinates

LATLNGS


You're very lucky!

  • Points 
  • Lines
  • Polygons

... you can start exploring and displaying right away

GOOGLE FUSION TABLES

Launched in 2009 for large data tables
Unbeatable ease for getting started

GOOGLE FUSION TABLES


Sort and filter data, view results on map

GOOGLE FUSION TABLES


Heat maps out of the box

Chloropleths (intensity maps) - merge with boundaries

CARTODB


Google Fusion Tables +++
Hosted geospatial database with PostGIS
Free only up to 25MB

CARTODB


Hexbins and advanced styling, 
customisable info-boxes, choice of map backgrounds

CARTODB


Dynamic time maps with Torque

TILEMILL & MAPBOX


TileMill desktop software for Mac and Windows:
Import datasets, design your own maps - 
again, no programming skills needed

POSTCODES


Full postcodes should "just work"
in Google Fusion Tables, CartoDB.
But beware postcode accuracy!

NATIONAL GRID (OSGB36)


Coordinate TF0325
find grid TF, 3km east 
and 25km north


To get latlng, use UK Postcode API:
http://www.uk-postcodes.com/api.php 

ADDRESS STRINGS

  • King's Centre, King Street, Norwich
  • King Street, Norwich
  • Norwich

Can use Google Maps or CartoDB (Nokia) to geocode
Probably won't be 100% accurate

ADMINISTRATIVE DISTRICTS

If your file has long strings like 'E09000001'...
congratulations, those are UK districts!

You'll need to find the boundaries. 
HOW?????

uk districts

These can be: 

  • Administrative - councils, districts
  • Electoral - constituencies, wards
  • Census - output areas
  • Health - here be dragons!
  • Postal - ye olde postcode
  • Other - police, Europe... 

See: A Beginner's Guide to Geography, by the ONS

ADMINISTRATIVE


Boring but crucial.
Get boundaries from ONS (open data).

ONS ADMIN CODES

Woohoo! The most exciting part of our talk!
ONS changed all its data codes in 2011. 

From hierarchical short codes:
  • 12 - Cambridgeshire
  • 12UD - Fenland... etc

To nine-letter codes, all starting with E0: 
  • City of London - E09000001
The first two numbers now tell you the type of area. 

Your data may have either!

MAPPING ONS CODES


MapIt!
http://mapit.mysociety.org/area/E08000025.html
http://mapit.mysociety.org/area/OOCN.html
gns <-> ons codes

MAPPING ADMIN BOUNDARIES


Can be done with Fusion Tables or CartoDB with a little effort, but for no effort at all... OpenHeatMap

INSTANT SHAREABLE MAPS


Note time slider!

Open HEAT MAP

What OpenHeatMap understands in the UK:

Admin regions:
  • Council names
  • Council codes (old codes only)
  • County names
  • Region names

Electoral regions: 
  • Constituency names
  • Constituency codes (old codes only)

    • CENSUS: OUTPUT AREAS

    • Designed to be static
    • Designed to be socially homogenous
    • OAs are about 150 households, LSOAs = 4-5 OAs
    • Census/social data typically available at LSOA level

    OUTPUT AREAS...

    ... are cool

    OUTPUT AREAS


    • Get boundaries from the ONS
    • Shapefile is about 150MB
    • You can make it smaller with ogr2ogr 

    POSTCODE -> ADMIN AREA?


    MapIt!

    BOUNDARY AREAS?


    MapIt!

    MAKING MAPS WITH D3

    ... is something to do
    after you've explored the data

    MAKING MAPS WITH D3

    ...is fun, but not something
    you want to learn in a 20-minute talk

    MAKING MAPS WITH D3


    Recommendations for starting mapping: 
    Scott Murray, Interactive Data Visualisation for the Web
    Mike Bostock's Let's Make a Map

    ARE WE DONE?


    Nope!

    DISPLAYING YOUR MAP


    • Projection
    • Colours
    • How to lie with maps...

    DISPLAYING - PROJECTIONS


    Is Greenland really bigger than Australia?
    Projection slides from a talk by Dimi Sztanko (@dimi)

    DISPLAYING - PROJECTIONS


    Nope! 

    Projection slides from a talk by Dimi Sztanko (@dimi)

    MERCATOR PROJECTION


    • Cylindrical projection from 1569
    • Great for navigating with a compass...
    • Horrible for rendering the whole world
    • "Politically incorrect"

    Projection slides from a talk by Dimi Sztanko (@dimi)

    WHICH PROJECTION TO USE?


    Hundreds to choose from...

    Projection slides from a talk by Dimi Sztanko (@dimi)

    NO "RIGHT" ANSWER

    • There is no silver bullet
    • Every projection has distortions
    • Depends what you are showing
    • Depends what do you want to say
    • Depends on your taste
    • Ultimate super detailed guide with  explanations: http://bit.ly/1iycTul 



    Projection slides from a talk by Dimi Sztanko (@dimi)

    PROJECTIONS FOR COUNTRIES

    If your country is elongated north-south...
    use transverse mercator

    Projection slides from a talk by Dimi Sztanko (@dimi)

    PROJECTIONS FOR COUNTRIES


    Projection slides from a talk by Dimi Sztanko (@dimi)

    PROJECTIONS FOR COUNTRIES


    Projection slides from a talk by Dimi Sztanko (@dimi)

    PROJECTIONS FOR DISTANCE FROM A SINGLE POINT

    Projection slides from a talk by Dimi Sztanko (@dimi)

    (USE FOR YOUR MISSILE MAPS)


    DISPLAYING - COLOURS

    DISPLAYING - COLOURS

    1. Identify the number of data classes

    2. Set the nature of your data:
    • qualitative
    • diverging
    • sequential

    3. Choose options:
    • colour-blind safe
    • print safe

    4. Pick a colour scheme

    DISPLAYING - MORE...


    WHERE TO GET HELP - GEO


    http://gis.stackexchange.com

    WHERE TO GET HELP - DATA


    http://opendata.stackexchange.org

    ANy QUESTIONS?


    Slides made with:
    http://slid.es
    Projection slides by Dimi Sztanko:
    http://twitter.com/sztanko

    Me:
    http://anna.ps
    http://twitter.com/darkgreener

    DRINKS!


    PS: Work for mySociety.org (MaPit): 

    http://mysocietyltd.theresumator.com

    web developers + events officer
    Made with Slides.com