Flip the Script
Avoiding the common pitfalls of GIS scripting
Daniel Silk | Senior Location Data Analyst
This work is licensed under a Creative Commons Attribution 4.0 International License.
Data
Geospatial
LINZ Data Service
Geodetic
Land
Survey and Titles
Valuation
Topographic
Sea
Hydrographic Charts
Notices to Mariners
Regulatory
Overseas Investment
Electoral Support
Place Names
Crown Property
Crown Land, River and Lake Bed
High Country Pastoral Leases
A Short History Lesson..
2004
Started using PostgreSQL / PostGIS
- Free with great community support
- QGIS 0.1 for visualisation
A Short History Lesson..
2008
Internal application development
- Cadastral adjustment blocks
PostGIS data store
A Short History Lesson..
2014
QGIS 2.6 deployed via Citrix environment
- accessible to every LINZ staff member
A Short History Lesson..
2014
More internal application development
- Place names (gazetteer)
A Short History Lesson..
2016
More internal application development
- Address points
- Roads
Contributions
Advanced selection tools
WFS provider rewrite
Delimited text plugin rewrite
WMTS rendering improvements
Various plugins
Why
Scripting?
Topography
- produce Topo50 / Topo250 map series
Increased Focus on Data
- historically, data a by-product of map production
- now key national datasets available for re-use
New Tools
- existing map data captured with 1Spatial software
Capture User Stories
User Story
- As a [role] I want to [feature] so that [reason]
simple description of a requirement
- As a [topographic data analyst] I want to [identify all unnamed roads under 300m] so that [I can check that those roads are suitable for a Topo50 map]
Acceptance Criteria
captures intent
- I can choose which layer to run the script against
- All roads in the output are unnamed
- All roads in the output are shorter than 300m
- The output is provided as a geopackage
- The output retains all of the attributes of the input
shapefile
Start with a Test Dataset
Large Sample
Focused Test Dataset
Test Descriptions
Desired Result
Use Version Control
Common Trap?
- Spatial professionals often end up scripting something
- But no version control?
Benefits
- Visible, enables collaboration
- Manage production vs dev versions
- Easily compare any code state - easy debugging
Training
- Heaps of free courses online
- So worth paying for structured training if possible
User Story Held Close to the Code
Keep Track of Changes
Semantic Versioning
semver.org
- Maintain a version number somewhere in your code
- Format: MAJOR.MINOR.PATCH
Release Branches
- Branch for each minor release
- release-0.2
- release-1.0
Changelog
keepachangelog.com
- Maintain a list of notable changes for each version
- Split into subsections e.g. Added / Changed / Fixed
- Maintain an Unreleased section too!
Demo Frequently
Demo
- Check against acceptance criteria
- Frequent demos allow for fast fails
Write Tests
Testing
- Acceptance criteria are a good place to start
Script Assistant
- Running tests in QGIS got tedious
- Enter the Script Assistant plugin
Use Continuous Integration
Automated Testing
- What are other people in the QGIS community doing?
QGIS Testing Environment Docker
github.com/boundlessgeo/qgis-testing-environment-docker
- Allows plugin tests to be automated to run in a real instance of QGIS
- Support for QGIS 2.14 / 2.18 / 2.99
Keep Up with QGIS
Visual Changelogs
qgis.org
QGIS Enhancement Proposals
github.com/qgis/QGIS-Enhancement-Proposals
Thanks for listening!
slides
slides.com/dwsilk/flip-the-script
other resources
github.com/linz/qgis-scriptassistant-plugin
github.com/boundlessgeo/qgis-testing-environment-docker
contact
@dwsilk on The Spatial Community (Slack channel)
spatialcommunity.org
Thanks for listening!
slides
slides.com/dwsilk/flip-the-script
other resources
github.com/linz/qgis-scriptassistant-plugin
github.com/boundlessgeo/qgis-testing-environment-docker
contact
@dwsilk on The Spatial Community (Slack channel)
spatialcommunity.org
This work is licensed under a Creative Commons Attribution 4.0 International License.
Flip the Script
By Daniel Silk
Flip the Script
Avoiding the common pitfalls of GIS scripting
