Cooking with Open Source tools


KC CSO Real-time Viewer

June 4, 2014

Peter Keum 
peter.keum@kingcounty.gov /mapitpk.com
King County Wastewater Treatment Division


King County WTD GIS Specialist

what is open source software?

"Software distributed with three basic conditions"

"Free to Share"

"Free to Modify"

"Free to Share Modifications"


cooking with open source tools



  • Background Information
  • Data Viewer (beta)
  • Work Flow
  • Description of Tools

BackGround Information

King County Wastewater Treatment Division


Mission: 

  • Protect Public Health 
  • Enhance Environment by Treating and Reclaiming water, Recycling Solids and Generating Energy

WTD Facts by numbers


  • 1.5 million people
  • 420 square miles
  • 391 miles sewer pipeline 
  • 47 Pump Stations
  • 38 CSO Outfalls
  • 25 Regulator Stations
  • 3 Regional Treatment Plants
  • 2 Local Treatment Plants


Combined sewer overflow (CSO)

Stormwater + Sewer = CSO


KC CSO Outfalls


Federally (EPA) mandated to show the CSO events and locations.

King County:  38 CSO outfalls 
City of Seattle:  ~ 90 CSO outfalls 
15% of Service area is combined sewer
 
1950 :    20 Billion Gallons 
1980:    Average 2.3 Billion Gallons
2005:   Reduced to 1 Billion Gallons

Current Real-time CSO viewer


  • Locations
  • Current Status 10min interval
  • Static basemap (no slippy map)

(Beta) Real-time CSO viewer


  • Slippy map - zoom/pan/identify features 
  • Detail information for user

Workflow 

E T L - Extract, Transform, Load

E  T  L

Extract  Transform  Load


1. Extract - python to take csv status file from FTP site.

2. Tranform - python to generate geoJSON file.

3. Load - "push" newly created geoJSON file into GitHub repository

tools


  • Python
  • GitHub
  • geoJSON



python



Open source scripting  language that is extremely flexible and powerful. 

Natural language that is human-readable expression like  Ruby, Visual Basic and Java

Guido van Rossum 

Developed in 1989


 

PYthon



  • Web-based code sharing and publishing service using Git
  • Launched on 2008 
  • 13 million repositories, largest code host

  • Free accounts for open source projects
  • Paid plan for private repositories

git

Code version control system that manages and stores code projects. 

Open source project initally started 2005 by Linus for Linux

GIthub


Three important concepts:
"forking"
"pull request"
"merge"

github


"Forking"

Copying a repository from one user's account to another.

You make copy and work as your own. 

github


"pull request"

Sending a notification to the original owner that you've made some changes that owner might be interested in incorporating 


"merge"
Process of incorporating modification into one's own project 

geojson

  • Data format, format finalized in June 2008
  • JSON (JavaScript Object Notation) + Geospatial data
  • More compact and easily loaded in JavaScript than XML  
  • "Fat Free" XML

 {
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "type": "Point",
        "coordinates": [
          -122.33103632926941,
          47.59887827375554
        ]
      }
    }
  ]
}

GeoJSON

GeoJSON tools

Geojson.io : Web-Editor for GeoJSON


GeoJSONLint: GeoJSON validator

OpenStreetmap

Github basemap


  • Wikipedia of world map
  • Freely editable dataset of the globe
  • Easy to edit/modify data onto OSM
  • Emphasis on local knowledge
  • It is NOT 
    • Rendered map
    • Proprietary
    • Static
    • Controlled by authority

project status


  • Live Prototype
  • Initial Testing is promising
  • Migrate python script to enterprise server
  • Internal Staff Usage Test in near future

Summary 


  • Background Information - WTD
  • CSO outfalls
  • Beta Real-time CSO status viewer
  • Extract Transform Load
  • Various Tools 
    • Python - scripting language
    • GitHub - web service based on Git
    • geoJSON - data format type

CUGOS

Cascadia Users of Geospatial Open Source

Formed in 2007


Meet 3rd Wednesday at ClipCard
Downtown Seattle

International FOSS4G 2014 
Portland, OR







CSO Real-Time Status Viewer





THank you!




questions / comments?


KCGIS User Group Meeting

By Peter Keum

KCGIS User Group Meeting

Presentation for KCGIS User group meeting June 4, 2014

  • 2,175
Loading comments...

More from Peter Keum