Kim Sundeen
GIS Developer and all-around map nerd. I develop user-focused and client solutions to help automate, analyze, and strategize data, mapping, and workflow needs.
Simplifying Metadata Editing for the Non-Programmer:
Semi-automating metadata documenting using Python's docx Word module
Kim Sundeen
GIS Programmer Analyst, MN Power, MN
work: 218-355-2836
218-355-2836
work: ksundeen [at] mnpower.com
personal: kim.h.sundeen [at] gmail.com
@KSundeen
Northern MN GIS Meeting, April 28th, 2016
Overview
- Importance of Metadata
- Changing our Thinking Towards Metadata
- Playing Catch-up in Documenting Metadata
-
Using Python for Metadata Import/Export
- Configuring modules
- Interactive Demo
Importance of Metadata?
Isn't it obvious?
- knowledge transfer, archiving historical records, record methods, standardize definitions, establish contacts, establish legal limitations for data distribution...etc
- FGDC: https://www.fgdc.gov/metadata
What are obstacles to simplifying process?
Changing the Culture of Metadata Editing
from FGDC's "Institutionalize Metadata Before It Institutionalizes You" (2005)
| Obstacle | Potential Solution |
|---|---|
| Metadata standards are too extensive & difficult to implement | Create metadata templates |
| Metadata production requires time & other resources |
Identify benefits to individuals who create metadata: 1. Reduced data management workload; 2. Better knowledge of distribution/sharing limitations; 3. Performance measures |
| Few tangible benefits & incentives to produce metadata. | Technical staff know the data, but management establishes new policies/procedures & allocates resources for policies. "Therefore, those inspired to integrate metadata production must champion an effort to educate administrators." |
How to Play Catch-up with Metadata?
Philosophical discussions of new & future policies aside...
let's simplify how to document EXISTING GIS layers.
Python Modules Demo
arcpy: http://pro.arcgis.com/en/pro-app/arcpy/get-started/what-is-arcpy-.htm
python-docx: http://python-docx.readthedocs.io/en/latest/index.html
arcpy_metadata: https://github.com/ucd-cws/arcpy_metadata
openpyxl: http://openpyxl.readthedocs.io/en/default/
- FYI on installing pip to easily install python packages in the "correct" place https://pip.pypa.io/en/stable/installing/
Tips on Installing Python Modules
- FYI on installing pip (Python Install Package) to easily install python packages in the "correct" place https://pip.pypa.io/en/stable/installing/
-
Use Windows cmd to navigate to directory where pip was installed.
-
Then follow instructions to install python packages...
-
Sometimes doesn't work; try easy install method
>pip install python-docx
>easy_install python-docx
Thinking Outside the Esri Metadata Box
-
How about creating a Word doc formatted using your existing metadata?
-
How about allowing GIS users to update Word doc/Excel doc instead of any metadata editor?
-
How about importing updates of Word doc/Excel doc metadata back into Esri's metadata editor?
PseudoCode to Automate
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
-
2) Export existing metadata into MS Word document for reviewing, updating, & entering new metadata into the Word doc
Provides printed version of metadata
-
3) (Future Addition) - take the newly updated MS Word / Excel file metadata & import directly into Esri's metadata editor.
Offers simple way for multiple users to review & correct metadata.
Automates import into metadata editor fields.
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
Iterating through FC Fields
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
Iterating through Subtypes
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
Iterating through Domains
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
Getting FC Field Value Counts
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
Initial csv output
PseudoCode to Automate (cont.)
1) Export attribute, domain, subtype, counts, field metadata for review in Excel
then just DATA ->Text to Columns -> Delimited -> Separate by ';'
PseudoCode to Automate (cont.)
-
2) Export existing metadata into MS Word document for reviewing, updating, & entering new metadata into the Word doc
Provides printed version of metadata
Formatting output Word Objects
PseudoCode to Automate (cont.)
-
2) Export existing metadata into MS Word document for reviewing, updating, & entering new metadata into the Word doc
Provides printed version of metadata
Creating Bullet Points for GIS Dataset Names
PseudoCode to Automate (cont.)
-
3a) (Future Addition) take the newly updated MS Word doc's metadata and import it directly into Esri's metadata editor.
Offers simple way for multiple users to review & correct metadata.
Automates import into metadata editor fields.
3b) (Future Addition) or take updated information from Excel & import into metadata editor.
PseudoCode to Automate (cont.)
3a) (Future Addition) take the newly updated MS Word doc's metadata and import it directly into Esri's metadata editor.
Using python's arcpy_metadata module
PseudoCode to Automate (cont.)
3b) (Future Addition) or take updated information from Excel & import into metadata editor.
Using python's openpyxl module
Simplifying Metadata Editing for the Non-Programmer: Semi-automating metadata documenting using Python's docx Word Module
By Kim Sundeen
Simplifying Metadata Editing for the Non-Programmer: Semi-automating metadata documenting using Python's docx Word Module
We all mean well when saying we're planning to add metadata to our GIS databases and layers. However, meaning well doesn't always go far enough. We need a more efficient and practical solution to help GIS staff enter the descriptions, definitions, contacts, dates, and all other relevant metadata pieces of information consistently. This presentation showcases a solution using two python modules (docx and arcpy_metadata) to 1) export existing metadata into a MS Word document for reviewing, updating, and entering new metadata into the the Word doc; and 2) take the newly updated MS Word doc's metadata and import it directly into Esri's metadata editor. This solution is two-fold: 1) it provides a printed version of your metadata and 2) it offers a simple way for multiple users to review and correct metadata for automated import into the metadata editor fields.
