It Takes a Village to Find a Village:

Solving complex problems with diverse geospatial tools

 

Stace Maples

Stanford Geospatial Center

https://link.springer.com/article/10.1186/s41018-018-0030-y

Hannah:

"I can see the settlements in Google Earth, but I know that those images are old and those villages are long gone,.."

Me:

"You have come to the right place!"

Where should we look?

Obtaining and Preparing Imagery

~5000 sqkm
200gb+

(before pansharpening)

ArcMap crashed after 9 hrs processing

With the Geospatial Data Abstraction Library

(or, GDAL)

http://www.gdal.org/

So I wrote a python script...

#Import modules
import glob, os, gdal, fnmatch

#Set base directory to location of script
base_directory = os.path.abspath('')

#Make a list of Multispectral directories with 'MUL' in the path
multi_directory_list = glob.glob(os.path.join(base_directory, '*MUL'))

#Make Panchromatic directory [pan_directory] path by replacing 'MUL' with 'PAN'
for multi_directory in multi_directory_list:
    pan_directory = multi_directory.replace('MUL','PAN')
    print(multi_directory)
    print(pan_directory)
    #Make a list of TIF files in the multi_directory
    multi_tiffs_list = fnmatch.filter((os.listdir(multi_directory)), '*.TIF')
    for multi_tiff_name in multi_tiffs_list:
        print(multi_tiff_name)
#Get the matching panchromatic image for the current multispectral by searching for the RowCol
        multi_rowcol_wildcard = '*'+multi_tiff_name[19:24]+'*'
        #print(multi_rowcol)
        pan_tiffs_list = fnmatch.filter((os.listdir(pan_directory)), '*.TIF')
        pan_tiff_name = fnmatch.filter(pan_tiffs_list, multi_rowcol_wildcard)
        #print(pan_tiff_name[0])
#Pan-sharpen the multispectral using it's corresponding panchromatic image using JPEG compression and internal tiles
        print('pansharpen')
        os.system('python gdal_pansharpen.py -co NBITS=12 -co COMPRESS=JPEG -co JPEG_QUALITY=50 -co PHOTOMETRIC=YCBCR -co TILED=YES ' + os.path.join(pan_directory,pan_tiff_name[0]) + ' ' + os.path.join(multi_directory,multi_tiff_name) + ',band=7 ' + os.path.join(multi_directory,multi_tiff_name) + ',band=5 ' + os.path.join(multi_directory,multi_tiff_name) + ',band=3 ' + base_directory + '/sharp_' + multi_tiff_name)
#Add overviews for resulting tiffs
        os.system('gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL -r average ' + base_directory + '/sharp_' + multi_tiff_name + ' 2 4 8 16')
#Create a Virtual Raster Mosaic of VRT type from all resulting sharpened TIFF images
print('create VRT mosaic from SHARP images')
os.system('gdalbuildvrt sharpmosaic.vrt *.TIF')
#Haven't finished figuring out how to get a decently stretched image out of this part. More to do!
print('MBtiles out')
print('gdal_translate sharpmosaic.vrt sharpmosaic.mbtiles -of MBTILES')
print('gdaladdo -r average sharpmosaic.mbtiles 2 4 8 16')

https://github.com/mapninja/DGF_parser

Which took 45 minutes to process the imagery

http://www.gdal.org/

Reconnaissance

HIVE

13440 x 5400 x .53m pixels
=
20 sq km

 

Typical Monitor

1440 x 900 x .53m pixels
=
.7 sq km

 

Or,...
250 shift+arrow pans vs. 7000+

 

~ 6 hrs

Outcomes

  • Completed 2 weeks from the DGF imagery capture date
  • ~6hrs for the recon
  • 5,000+ sq km
  • 225 settlements

 

  • We surveyed 347 women of reproductive age & 824 of their children 15 years of age and younger

  • 572 of whom were under five.

  • QGIS
  • GDAL
  • Python
  • ArcMap
  • ArcGIS Online
  • Humanitarian OpenStreetMap Task Manager
  • OpenStreetmap.org
  • CartoDB
  • Garmin
  • DigitalGlobe Foundation
  • HIVE @ICME Stanford
  • David Rumsey Map Center

Machine Learning?

  • Daily 3m Planet Imagery

  • Excellent OSM-Based Training Data

  • Cheap GPUs and ML Cloud Infrastructure...

  • Working with DescartesLabs to automate the process
     

Relationships were as important in this project as technology...
 

DigitalGlobe Foundation

ICME

David Rumsey Map Center

Humanitarian OpenStreetMap Team

OSM Community

Stanford spends a great deal of energy, time and money developing it's
"Brand"

 

And part of my job is to leverage that brand for all I can to provide resources and connections for my Stanford Researchers

 

As a community and culture, you have a globally recognized brand.

A brand that others want to associate with.

Build it
Protect it
Leverage it

in your relationships with vendors, stakeholders and partners

Title Text

The geospatial data science and mapping tools we are working with, this week, are part of that "Macroscope"

"I suppose it is tempting,if the only tool you have is a hammer, to treat everything as if it were a nail."

Abraham Maslow - "The Psychology of Science, 1966"

Don't be this...

...be this.

gis.stanford.edu
github.com/stanfordgeospatialcenter

 

@mapninja

stacemaples@stanford.edu

 

slides.com/staceymaples/imw2018

NZ IMW 2018

By Stace Maples