It Takes a Village to Find a Village:
Solving complex problems with diverse geospatial tools
Stace Maples
Stanford Geospatial Center
https://link.springer.com/article/10.1186/s41018-018-0030-y
Hannah:
"I can see the settlements in Google Earth, but I know that those images are old and those villages are long gone,.."
Me:
"You have come to the right place!"
Where should we look?
Obtaining and Preparing Imagery
~5000 sqkm
200gb+
(before pansharpening)
ArcMap crashed after 9 hrs processing
With the Geospatial Data Abstraction Library
(or, GDAL)
http://www.gdal.org/
So I wrote a python script...
#Import modules
import glob, os, gdal, fnmatch
#Set base directory to location of script
base_directory = os.path.abspath('')
#Make a list of Multispectral directories with 'MUL' in the path
multi_directory_list = glob.glob(os.path.join(base_directory, '*MUL'))
#Make Panchromatic directory [pan_directory] path by replacing 'MUL' with 'PAN'
for multi_directory in multi_directory_list:
pan_directory = multi_directory.replace('MUL','PAN')
print(multi_directory)
print(pan_directory)
#Make a list of TIF files in the multi_directory
multi_tiffs_list = fnmatch.filter((os.listdir(multi_directory)), '*.TIF')
for multi_tiff_name in multi_tiffs_list:
print(multi_tiff_name)
#Get the matching panchromatic image for the current multispectral by searching for the RowCol
multi_rowcol_wildcard = '*'+multi_tiff_name[19:24]+'*'
#print(multi_rowcol)
pan_tiffs_list = fnmatch.filter((os.listdir(pan_directory)), '*.TIF')
pan_tiff_name = fnmatch.filter(pan_tiffs_list, multi_rowcol_wildcard)
#print(pan_tiff_name[0])
#Pan-sharpen the multispectral using it's corresponding panchromatic image using JPEG compression and internal tiles
print('pansharpen')
os.system('python gdal_pansharpen.py -co NBITS=12 -co COMPRESS=JPEG -co JPEG_QUALITY=50 -co PHOTOMETRIC=YCBCR -co TILED=YES ' + os.path.join(pan_directory,pan_tiff_name[0]) + ' ' + os.path.join(multi_directory,multi_tiff_name) + ',band=7 ' + os.path.join(multi_directory,multi_tiff_name) + ',band=5 ' + os.path.join(multi_directory,multi_tiff_name) + ',band=3 ' + base_directory + '/sharp_' + multi_tiff_name)
#Add overviews for resulting tiffs
os.system('gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL -r average ' + base_directory + '/sharp_' + multi_tiff_name + ' 2 4 8 16')
#Create a Virtual Raster Mosaic of VRT type from all resulting sharpened TIFF images
print('create VRT mosaic from SHARP images')
os.system('gdalbuildvrt sharpmosaic.vrt *.TIF')
#Haven't finished figuring out how to get a decently stretched image out of this part. More to do!
print('MBtiles out')
print('gdal_translate sharpmosaic.vrt sharpmosaic.mbtiles -of MBTILES')
print('gdaladdo -r average sharpmosaic.mbtiles 2 4 8 16')
https://github.com/mapninja/DGF_parser
Which took 45 minutes to process the imagery
http://www.gdal.org/
Reconnaissance
HIVE
13440 x 5400 x .53m pixels
=
20 sq km
Typical Monitor
1440 x 900 x .53m pixels
=
.7 sq km
Or,...
250 shift+arrow pans vs. 7000+
~ 6 hrs
Outcomes
- Completed 2 weeks from the DGF imagery capture date
- ~6hrs for the recon
- 5,000+ sq km
- 225 settlements
-
We surveyed 347 women of reproductive age & 824 of their children 15 years of age and younger
-
572 of whom were under five.
- QGIS
- GDAL
- Python
- ArcMap
- ArcGIS Online
- Humanitarian OpenStreetMap Task Manager
- OpenStreetmap.org
- CartoDB
- Garmin
- DigitalGlobe Foundation
- HIVE @ICME Stanford
- David Rumsey Map Center
Machine Learning?
-
Daily 3m Planet Imagery
-
Excellent OSM-Based Training Data
-
Cheap GPUs and ML Cloud Infrastructure...
-
Working with DescartesLabs to automate the process
Relationships were as important in this project as technology...
DigitalGlobe Foundation
ICME
David Rumsey Map Center
Humanitarian OpenStreetMap Team
OSM Community
Stanford spends a great deal of energy, time and money developing it's
"Brand"
And part of my job is to leverage that brand for all I can to provide resources and connections for my Stanford Researchers
As a community and culture, you have a globally recognized brand.
A brand that others want to associate with.
Build it
Protect it
Leverage it
in your relationships with vendors, stakeholders and partners
Title Text
The geospatial data science and mapping tools we are working with, this week, are part of that "Macroscope"
"I suppose it is tempting,if the only tool you have is a hammer, to treat everything as if it were a nail."
Abraham Maslow - "The Psychology of Science, 1966"
Don't be this...
...be this.
gis.stanford.edu
github.com/stanfordgeospatialcenter
@mapninja
stacemaples@stanford.edu
slides.com/staceymaples/imw2018
NZ IMW 2018
By Stace Maples
NZ IMW 2018
- 578