Hands on with GDAL/OGR

• .5hr: Setup & Installation
• 2hr: Instruction & Group Exercises

This workshop is adapted and updated from http://slides.sarasafavi.com/gdal/#/ 

which is cc-by-nc 4.0| © Sasha Hart & Sara Safavi, 2017

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

What you'll do today

• Lots of typing-what-you-see
• Probably a bit of Googling
• Forget at least 50% by tonight   (that's ok!)
• Leave here 100% more familiar with GDAL/OGR

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

What is GDAL/OGR?

Geospatial Data Abstraction Library for translating & processing raster & vector geospatial data.

• Free & open source
• Used by your favorite desktop GIS
• Has a command line interface

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

What kinds of data?

GDAL's superpower is that it can read or write pretty much any spatial file you throw at it.

As of April 2016, GDAL can translate 142 raster formats, and 84 vector formats!

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

A note about the name:

"GDAL/OGR" refers to the combined project with both raster & vector tools

• GDAL: raster
• OGR: vector

Today, most refer to the combined project as "GDAL"

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Installation & Setup

https://mapscaping.com/installing-gdal-for-beginners/

If you want to use Conda:

https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html

file:///Users/maples/Downloads/conda-cheatsheet.pdf

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Windows

Recommended route:
OSGEO4W Installer
http://trac.osgeo.org/osgeo4w

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Linux

• Ubuntu: GDAL 1.11 in Default (Universe)
• Debian: GDAL 1.10 in Stable (Jessie)
• Fedora/CentOS/RHEL: EPEL 7 has GDAL 1.11

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Mac

Recommended route:
https://medium.com/@vascofernandes_13322/how-to-install-gdal-on-macos-6a76fb5e24a4

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

GDAL for Python in CONDA

Recommended route: https://opensourceoptions.com/how-to-install-gdal-with-anaconda/

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Getting Started

GDAL/OGR tools are accessed on the command line
(i.e., in the terminal or shell)

Note for Windows Users:
Use “OSGeo4W Shell”

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Verify installation by typing ogrinfo & hitting enter.
If all went well, you should see:

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Check point

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Exploring Data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Demo Data

https://drive.google.com/file/d/18c1L3nt3FTcvd2YLm5HwXzfiulAgs0en/view?usp=sharing

Download & unzip
Move data to a "sandbox" directory

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Check out the data visually

Use your favorite GIS desktop application to view data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Demo vector data: Austin-area zipcodes & pools

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Demo raster data: Austin-area DEM & (clipped) DOQQ

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Explore data via the command line

Open your terminal and navigate to your working directory

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

OGR for vector data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

From your working directory, try this:
ogrinfo pools/pools.shp

INFO: Open of `pools/pools.shp'
      using driver `ESRI Shapefile' successful.
1: pools (3D Polygon)
	

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Now try this:
ogrinfo -so zipcodes/zipcodes.shp zipcodes

INFO: Open of `zipcodes/zipcodes.shp'
      using driver `ESRI Shapefile' successful.

Layer name: zipcodes
Geometry: Polygon
Feature Count: 80
Extent: (-98.589809, 29.669673) - (-97.184067, 31.037805)
Layer SRS WKT:
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_84",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
ZIPCODE: String (12.0)
NAME: String (30.0)
CREATED_DA: Date (10.0)
CREATED_BY: String (50.0)
MODIFIED_D: Date (10.0)
MODIFIED_B: String (50.0)
SHAPE_AREA: Real (19.11)
SHAPE_LEN: Real (19.11)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

GDAL for raster data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Try this:
gdalinfo dem10m/dem10m.dem

Driver: USGSDEM/USGS Optional ASCII DEM (and CDED)
Files: dem10m/dem10m.dem
       dem10m/dem10m.dem.aux.xml
Size is 3600, 3600
Coordinate System is:
GEOGCS["NAD83",
    DATUM["North_American_Datum_1983",
        SPHEROID["GRS 1980",6378137,298.257222101,
            AUTHORITY["EPSG","7019"]],
        TOWGS84[0,0,0,0,0,0,0],
        AUTHORITY["EPSG","6269"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9108"]],
    AUTHORITY["EPSG","4269"]]
Origin = (-97.999999407110110,31.000000503864648)
Pixel Size = (0.000277777777778,-0.000277777777778)
Metadata:
  AREA_OR_POINT=Point
Corner Coordinates:
Upper Left  ( -97.9999994,  31.0000005) ( 98d 0' 0.00"W, 31d 0' 0.00"N)
Lower Left  ( -97.9999994,  30.0000005) ( 98d 0' 0.00"W, 30d 0' 0.00"N)
Upper Right ( -96.9999994,  31.0000005) ( 97d 0' 0.00"W, 31d 0' 0.00"N)
Lower Right ( -96.9999994,  30.0000005) ( 97d 0' 0.00"W, 30d 0' 0.00"N)
Center      ( -97.4999994,  30.5000005) ( 97d30' 0.00"W, 30d30' 0.00"N)
Band 1 Block=3600x3600 Type=Float32, ColorInterp=Undefined
  Min=79.900 Max=398.900
  Minimum=79.900, Maximum=398.900, Mean=193.121, StdDev=63.589
  NoData Value=-32767
  Unit Type: m
  Metadata:
    STATISTICS_MAXIMUM=398.89999389648
    STATISTICS_MEAN=193.12105660542
    STATISTICS_MINIMUM=79.900001525879
	STATISTICS_STDDEV=63.588582941289
	

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Converting Data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Exporting original data into new formats

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Vector I/O Formats

ogr2ogr --formats

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Raster I/O Formats

gdal_translate --formats

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

shapefile → geojson

let's make a webmap-friendly version
of our zipcodes shapefile:
ogr2ogr -f geojson zipcodes.geojson zipcodes/zipcodes.shp

NOTE: This will generate Warnings - that's ok!

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Now try:
ogrinfo zipcodes.geojson
 
and:
ogrinfo -so zipcodes.geojson zipcodes

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

GeoTIFF → georeferenced PNG

gdal_translate -of png doqq.tif converted_doqq.png

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Now try:
gdalinfo converted_doqq.png

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Reprojecting Data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Side note: EPSG Codes
spatialreference.org (great resource!)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Let's make everything WGS84!

ogr2ogr -t_srs EPSG:4326 reprojected_pools.shp pools/pools.shp
 
gdalwarp -t_srs EPSG:4326 doqq.tif reprojected_doqq.tif

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Questions so far?

• ogrinfo
• gdalinfo
• ogr2ogr
• gdal_translate
• gdalwarp

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Querying Data

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Asking questions about your data layers

OGR allows you to use a dialect of SQL ("OGR SQL") to query layers

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Q: "How many swimming pools are in Austin?"

ogrinfo pools/pools.shp -sql "SELECT COUNT(*) FROM pools"

A: 51,073

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Q: "What is the first pool feature?"

ogrinfo -q pools/pools.shp -sql "SELECT * FROM pools WHERE fid = 1"

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Q: "How many above ground pools in Austin?"

ogrinfo -q pools/pools.shp -sql "SELECT COUNT(*) FROM pools WHERE FEATURE = 'Above ground'"

A: 21,550

Saving query results to a new file

Similar to how you might save a "query view" in other GIS software, you can use OGR to create a new file based on a SQL query

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Create a shapefile containing only the zipcode boundaries in Austin city limits"

ogr2ogr austin_zips.shp zipcodes/zipcodes.shp -sql "SELECT * FROM zipcodes WHERE name = 'Austin'"

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Create a shapefile containing small zipcodes"

ogr2ogr under_10mil.shp zipcodes/zipcodes.shp -sql "SELECT * FROM zipcodes WHERE SHAPE_AREA <10000000"

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

under_10mil.shp (tiny, purple shape)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Create a geoJSON with only zipcode 78756's boundary"

ogr2ogr -f geojson 78756.geojson zipcodes/zipcodes.shp -sql "SELECT * FROM zipcodes WHERE ZIPCODE = '78756'"

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Break

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Clipping to create new data

You can clip a target data layer to a given clipping layer to create a new layer

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Create a new layer with only the pools in zipcode 78756"

ogr2ogr -clipsrc 78756.geojson 78756_pools.shp reprojected_pools.shp

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

78756_pools.shp (shown over zipcodes)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Show only the imagery in zipcode 78756"

gdalwarp -cutline 78756.geojson doqq.tif 78756_doqq.tif

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

78756_doqq.tif (shown over zipcodes)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Clip a DEM to a bounding box"

gdal_translate -srcwin 0 0 1000 1000 -of USGSDEM dem10m/dem10m.dem clipped_dem.dem

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

clipped_dem.dem (shown with original DEM)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Fun with DEMs

Now that our DEM is clipped to a more managable size, let's see what it can do

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Generate a hillshade from our DEM"

gdaldem hillshade -of png clipped_dem.dem hillshade.png

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

hillshade.png

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Make a hillshade Using a z-factor"

gdaldem hillshade -of png -az 135 clipped_dem.dem hillshade135.png

https://gdal.org/programs/gdaldem.html

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

hillshade135.png

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Make a color relief"

gdaldem color-relief clipped_dem.dem colorramp.txt colorrelief.tif

Format of colorramp.txt:
(value) (R) (G) (B)

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

colorrelief.tif

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

"Create contour lines from a DEM"

gdal_contour -a elev -i 20 clipped_dem.dem contours.shp

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

contours.shp

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Questions?

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Thank you!!!

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Things to Try on Your Own

• Reproject a dataset using a PROJ4 string or a .prj file
• Create an index on the zipcodes shapefile
• Merge two datasets together
• Resize a raster

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

Useful Links

https://github.com/dwtkns/gdal-cheat-sheet

https://gdal.org/programs/index.html
https://gdal.org/user/ogr_sql_dialect.html
http://www.gdal.org

These Slides are at: https://slides.com/staceymaples/handsonwithgdal

MORE Useful Links

1. A Gentle Introduction to GDAL
2. Map Projections & gdalwarp
3. Geodesy & Local Map Projections
4. Working with Satellite Data
5. Shaded Relief

These Slides are at: https://slides.com/staceymaples/handsonwithgdal