[Open] Data
+
San Diego
Maksim Pecherskiy
Performance & Analytics Department
City of San Diego
@MrMaksimize
Who Is This Guy?
- Chief Data Officer, City of San Diego
- Performance & Analytics Department
- Lots of years doing software engineering
- Worked in Puerto Rico through Code for America
- Saw how powerful data can be when used inside government
Why Open Data is
important to the City
- Provide high quality public service
- Work in partnership with all our communities to achieve safe and livable neighborhoods
- Create and sustain a resilient, economically prosperous city
Opening data ties in directly with each of our Strategic Goals and allows us to monitor our progress.
Why [Open] Data is important
As a Resident
- Look at the city budget
- Pull in a calendar of events into my phone
- Avoid construction in my commute
- See when my street or sidewalk will be fixed
As a City Employee
- Access data from other departments
- Be transparent
- Reduce time responding to PRA
Benefits of Open Data
- Improve service delivery without increasing resources
- Facilitate intra-departmental data sharing
- Build and integrate city data into applications
- Provide most up-to-date and accurate data to consumers
- Provide city data to power new businesses and startups
Efficiency
Empowerment
Economic Development
Q: But I'm a Nerd,
Why Should I Care?
A: Because You're Lazy,
but you want to build
cool things
This?
- Download PDF
- Download Tabula
- Download Java
- Extract Page #5
- Run Tabula
- Select and Extract Table Data
- Import into Excel
- Fix headers
- Paste into viz tool
- Visualize
Total Time: 1 Hour
Or This?
Open Data Policy
- Draws on other existing policies.
-
Defines terms, making sure data meets "open criteria"
-
Assigns responsibilities to Chief Data Officer and to City Departments
-
Sets timeline
-
Includes reporting requirements to Mayor and Council
-
Comes with an Implementation Plan
-
They're both on the internet! (http://sdgo.io/sdpad)
Where We Are Today
- Issued inventory guidelines in March
- Designated information coordinators
- Acquired base knowledge of City information
- Identified individuals working with specific sets of data
- Completed preliminary inventory
AKA WTH Have You Been Doing?
About the city
- 11,000 Employees
- 35 Departments
- A ton of diversity in data stores
How We Got Here
- Databases
- Department spreadsheets
- Shared drives
- Online apps
1. Identify data sources
How We Got Here
What are all the single datasets
you can pull from the data sources?
2. Identify all datasets
How We Got Here
The individual in charge of the datasets – the Data Steward – answers questions
about the data and completes a catalog.
3. Complete dataset catalog
The Backend
We minimized in-person meetings and automated as much as we could, relying heavily on technology.
The Backend
Internal dashboards and metrics tracked
each department's progress.
What We Found
- 2000+ Datasets
- 800 Datasources
- Lots of low hanging fruit
Observations
- Amazing talent among our City workforce
- Potential for internal efficiencies
- Need to align open data program with technology initiatives
- Smart communication is key
Community Stuff
Where We're NOT Going
Where We're Going
- Timely
- Well-Described
- Reliable
- Complete
- Used
Where We're Going
Find
Prioritize
Describe
Clean / Transform
Evaluate
Publish
Update
- Value
- Security
- Quality
- Readiness
Where We're Going
Prioritize
Components
- Base metadata in inventory
- Metadata Schema
- Each Dataset
- Conform to Federal Open Standards
Where We're Going
Describe
- How is the data collected?
- Are there more reliable sources?
- Can we merge the sources together?
- Is the data of high quality?
- Are there gaps in the data that prevent analysis?
- Is there Personally Identifiable Information (PII) in text fields?
Where We're Going
Evaluate
- Systematically Remove PII
- Mold to standard or tidy data
- Combine multiple sources
- Make data useful
Where We're Going
Clean / Transform
Where We're Going
Publish and Update
Where We're Going
Publish and Update
- Business Intelligence
- Analytics
- PRA Cut-Down Time
- Citizen Engagement
- Communications
USE
Today
Manual
Find
Prioritize
Describe
Clean / Transform
Evaluate
Publish
Update
Automatic
Use
Tomorrow
Manual
Find
Prioritize
Describe
Clean / Transform
Evaluate
Publish
Update
Automatic
Use
An Example
Other Random + Cool Things
Splunk
Docker
Streetlights
See this
presentation Online!
Presentation:
http://sdgo.io/sdodgen1
I Thirst For More!
http://tiny.cc/odreport
Open San Diego
http://www.meetup.com/open-san-diego
The Vision
Get the proper stakeholders with the right skills,
involved in a timely manner,
equipped with the appropriate technology and accurate data
to facilitate good decisions
and innovative solutions for our residents.
But I Want 2 Hack Now!
Crime
- SDPD:
- Arjis:
- UCR:
- Transportation Injury Mapping System:
But I Want 2 Hack Now!
Disaster Response
- Fire dispatch:
- Cal Fire:
- County emergency map:
- SDGE outage map:
But I Want 2 Hack Now!
Water & Maps
- District-level usage:
- Local reservoir levels:
- SANGIS
- http://rdw.sandag.org/Account/Login