Title Text

#TheStreetsOfWomen

Jessica Sena

@jsenag
github.com/jessisena

Computer Engineer Posgraduate UX

#TheStreetsOfWomen

http://geochicasosm.github.io/lascallesdelasmujeres

Technical Process

2. Process data

https://blog.mapbox.com/mapping-female-versus-male-street-names-b4654c1e00d5

3. Generate the map

1. Extract data

1. Extract data

GOAL:
To create a file with ONLY the STREETS of a city

Tools:
Tile Calculator - OSM QA Tiles - TileReduce - Overpass API

1. Extract data

http://osmlab.github.io/osm-qa-tiles/

http://tools.geofabrik.de/

1. Extract data

Implements MapReduce

(parallel computing)

Distributed spatial analysis

JavaScript + Mbtiles

https://github.com/mapbox/tile-reduce

1. Extract data

Manages tiles
Manages partial results + the final one

"GEO" version of

"MapReduce"

reduce.js

Exects operations and analisys

map.js

Tile

resultt

1. Extract data

BBOX definition and MBTILES path

reduce.js

const opts = {
    bbox: [-84.12, 9.9, -84.02, 9.96],
    log: true,
    zoom: 12,
    sources: [
      {
        name: 'osm',
        mbtiles: path.join(__dirname, 'data/latest.planet.mbtiles')      
      }
    ],
    maxWorkers: 4,
    map: __dirname+'/map.js'
};

.on('reduce', function(result, tile){
  num++;
  finalGeojson.features = finalGeojson.features.concat(result.features) ;
})

.on('end', function(error){
    (...)
});

Generate the final Geojson including only streets

1. Extract data

module.exports = function(tileLayers, tile, writeData, done){

  const osmRoads = cleanGeoms(normalize(flatten(clip(tileLayers.osm.osm, tile))));
  done(null, osmRoads);

};

STEP 1:    Clip    +      flatten + normalize

Turf.js

@mapbox/Tilebelt

map.js

geojson-flatten

geojson-normalize

1. Extract data

turfjs.org

PASO 2: CleanGeoms

LineString
MultiLineString

Polygon

?

module.exports = function(tileLayers, tile, writeData, done){

  const osmRoads = cleanGeoms(normalize(flatten(clip(tileLayers.osm.osm, tile))));
  done(null, osmRoads);

};

map.js

1. Extract data

PASO 2: CleanGeoms

Polygon

module.exports = function(tileLayers, tile, writeData, done){

  const osmRoads = cleanGeoms(normalize(flatten(clip(tileLayers.osm.osm, tile))));
  done(null, osmRoads);

};

map.js

FIXED!

function isValidSquare(geom) {

  return (
    geom.geometry.type === 'Polygon' &&
    geom.properties.highway &&
    geom.properties.highway === 'pedestrian'&&
    geom.properties.area &&
    geom.properties.area === 'yes'
    && geom.properties.name !== undefined
  );

}

1. Extract data

STEP 2: CleanGeoms

module.exports = function(tileLayers, tile, writeData, done){

  var osmRoads = cleanGeoms(normalize(flatten(clip(tileLayers.osm.osm, tile))));
  done(null, osmRoads);

};

map.js

1. Extract data

PASO 2: CleanGeoms

 geom.properties = {
   name: geom.properties.name,
   id: geom.properties['@id'],
   wikipedia_link: '',
   gender: 'unknown',
   scale: ''
 };

module.exports = function(tileLayers, tile, writeData, done){

  var osmRoads = cleanGeoms(normalize(flatten(clip(tileLayers.osm.osm, tile))));
  done(null, osmRoads);

};

map.js

1. Extract data

SOLUTION: 
CLIP!

Streets out

of boundaries

1. Extract data

CLIP:

https://www.npmjs.com/package/query-overpass

OSM ID

Geojson Boundary

flatten 
+ 
booleanContains

1. Extract data

Streets

~~out~~ inside boundaries

1. Extract data

"Clean" name for the classification

¿How?

1. Extract data

const filterList = ['Paseo','Río', 'Avenida', 'Hacienda', 'Puerto', 'Callejón', 'Calle', 'Calzada', 'Camino', 'Av.','Paso', 'Cañada', 'Minas', 'Cerrada', 
    'Puebla', 'Principal', 'Central','Primera', 'Segunda', 'Portón', 'Lateral', 'Calz.', 'Corrido', 'Casa', 'Villa', 'Mejía', 
    'Vía', 'Via', 'Real', 'Isla', 'Avendida', 'Marisma', 'Rada', 'Raudal', 'Ribera', 'Embocadura', 'Cataratas', 'Médanos', 
    'Mirador', 'Av', 'Jardín',  'A.', 'Circuito','Gral.', 'Rincón', 'Calz', 'Rinconada', 'Periférico', 'Cda', 'Jardin', 
    'C.', 'Callejon', 'Colegio', 'Valle', 'avenida', 'camino', 'calle', 'Calle', 'Rotonda', 'Parqueo', 'Parque', 'entrada', 
    'Entrada', 'sendero', 'Sendero', 'Pasaje', 'pasaje', 'Puerto', 'Ciudad', 'Puente', 'Boulevard', 'Agrosuperior', 'Bodegas', 
    'Autobanco', 'SkyTrace', 'Plaza', 'Motel', 'C/', 'Rotonda', 'Drive', 'Residencial', 'Automac', 
    'Auto', 'Transcersal', 'Inter', 'Pasillo', 'Centro', 'Caminito', 'Arandas', 'Proveedores', 'Cajero', 'Zona', 'Primer', 'Res.'
  ];

Dictionary

of street prefixes

languages!

1. Extract data

FINAL RESULT:

nombreciudad_streets.geojson
list.csv

2. Manage the data

GOALS:

Streets classification FEMALE/MALE
Associate Wikipedia links
Manual review

SOLUTION 1:

2. Manage the data

2.1. Classification F/M

Use limitations

Slow (1request/name)

Parse "Name + Surname"

API

SOLUTION 2:

CSV list (source INE)

+25.000 classified names

Local 
Dictionary

github.com/marcboquet/spanish-names

2. Manage the data

2.1. Classification F/M

Load the dictionaries in memory

Take the "clean name" column in CSV

Jacint (Verdaguer)

Does it appear in the women's dictionary?

Does it appear in the men's dictionary?

NO

YES

is a WOMEN

YES

is a MAN

NO

Discard the street

new Set();

2. Manage the data

SOLUTION 1:

2.2. Wikipedia Links

Slow (request/street)

Complex results parse

MediaWiki API

2. Manage the data

SOLUTION 2:

Wikidata
Wikiquote
Wikinoticas
Wikisource

68.000 resources

2. Manage the data

2.2. Wikipedia Links

68.000 resources

https://fusejs.io/

2. Manage the data

2.2. Wikipedia Links

SOLUTION 2:

2.3. Manual check

Streets named after a person

F/M Classification

Wikipedia links

2. Manage the data

nombreciudad_streets.geojson

How?

Generate FINAL GEOSJON

2. Manage the data

const streeMap = new Map();

(...)

streeMap.set(splitLine[0], {
    url: url,
    gender: splitLine[4],
    scale: ""
});

var finalGeojson = {
    "type": "FeatureCollection",
    "features": []
};

for (var key in geojson.features) {

 if(streeMap.has(geojson.features[key].properties.name)){        
   (...)    
    finalGeojson.features.push(geojson.features[key]);
 }
}

2. Manage the data

Generate FINAL GEOSJON

Generata

STATS

if(!tratadosList.has(geojson.features[key].properties.name)){

  tratadosList.add(geojson.features[key].properties.name);

  if (objValues.url !== ('' && null && undefined) && objValues.gender.toLowerCase() === FEMALE){
    stats.numLink++;
    stats.numFemale++;
  }else if (objValues.gender.toLowerCase() === FEMALE){
    stats.numFemale++;
    stats.numNoLink++;                 
    noLinkList += `\n${geojson.features[key].properties.name}`;                                                             
  }else{
    stats.numMale++;
  }                    
}

2. Manage the data

FINAL GEOSJON Generated!

2. Manage the data

2. Manage the data*

(*known errors)

Filtering by HIGHWAY
~~Filtering by LINESTRING~~ (squares)
1street <> 1name
Incomplete DB of names
Classification criteria

3. Visualize the data

    this.map = new mapboxgl.Map({
      container: 'map',
      style,
      center: initCenter, // [lng, lat]
      zoom: 1,
    });

Create map object (MyMap.js)

    map.addLayer({
      id: `${sourcename}-line`,
      type: 'line',
      source: {
        type: 'geojson',
        data: geojson,
      },
      layout: {
        'line-join': 'round',
        'line-cap': 'round',
      },
      paint: {
        'line-blur': ['case', ['==', ['get', 'wikipedia_link'], ''], 4, 1],
        'line-color': [
          'case',
          ['==', ['get', 'gender'], FEMALE],
          '#ffca3a',
          '#00B99E',
        ],
        'line-width': [
          'case',
          ['==', ['get', 'gender'], FEMALE],
          widthFemale,
          widthMale,
        ],
      },
    });

Add line layer:

3. Visualize the data

    map.addLayer({
      id: `${sourcename}-fill`,
      type: 'fill',
      source: {
        type: 'geojson',
        data: geojson,
      },
      layout: {},
      paint: {
        'fill-color': ['case', ['==', ['get', 'gender'], FEMALE], '#ffca3a', '#00B99E'],
        'fill-opacity': ['case', ['==', ['get', 'wikipedia_link'], ''], 0.2, 0.6],
      },
      filter: ['==', '$type', 'Polygon'],
    });

Add polygon layer:

3. Visualize the data

How to add a new city?

Copy scripts results in data
Add city in src/Constants.js

Scripts results files

3. Visualize the data

So far...

In the most of the cities, the ratio woman-man is around 15%-85%

Collaborative Project

github.com/geochicasosm/

Intrucciones en README.md

Still a lot of room

for improvement!

4. Opportunities

Prepare the project to accept other languages

Dictionary of streets prefixes
Dictionary of names
Wikipedia links
...

4. Opportunities

Improve de criteria to remove or keep streets:

4. Opportunities

Define a women's taxonomy:

Which women do we have in our streets? (religious, scientists, artists?)
How is the classification used in Wikidata? Can we use it?
Check the one used in Barcelona

*Add the data to the map!

4. Opportunities

Analyze the streets assignation:

In which streets are the women assigned? main ones, secondary ones? new neighborhoods?
Why this happens? Is it a new trend?
Define street's taxonomy or use the one in OSM?

4. Opportunities

Revert data:

Prepare an automated way to revert the data in OSM

Prepare an automated way to revert the data in Wikidata

4. Opportunities

How to properly scale the project:

Visulization level (improve the menu, how to show small areas...)

Data loading level (vector tiles?)

4. Opportunities

Improve data analysis:

Prepare external page with more charts? (comparisson between cities, categories, ways...)

What else could we extrapolate from the data?

4. Opportunities

Find a better way to find the wikipedia links:

The current process has a lot of errors, improve it or create a new one.

4. Opportunities

Define a template for a social campaign to change the street names:

What steps should we need to follow
Who should we need to contact
How would we organize it and how many people would we need?
What else do would we need in the project (data related)
...

4. Opportunities

PUT YOUR IDEA HERE

https://geochicasosm.github.io/lascallesdelasmujeres/
https://github.com/geochicasosm

@geochicasosm

www.geochicas.org

#TheStreetsOfWomen MIT 2020

By Jessica Sena

#TheStreetsOfWomen MIT 2020

#LasCallesDeLasMujeres is a project of 'GeoChicasOSM', emerged during the 8M of 2018. The talk explains the technical process carried out for its realisation, from the collection and data processing, to the elaboration of the map for its visualisation, emphasising the technical problems and limitations found, and how they have been solved.

Jessica Sena

Software engineer. Design and development of web and mobile applications

Title Text

Jessica Sena

#TheStreetsOfWomen

Technical Process

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

turfjs.org

?

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

1. Extract data

¿How?

1. Extract data

Dictionary

1. Extract data

1. Extract data

1. Extract data

2. Manage the data

2. Manage the data

2.1. Classification F/M

2. Manage the data

2.1. Classification F/M

2. Manage the data

2.2. Wikipedia Links

2. Manage the data

2. Manage the data

2.2. Wikipedia Links

2. Manage the data

2.2. Wikipedia Links

2.3. Manual check

2. Manage the data

2. Manage the data

2. Manage the data

2. Manage the data

FINAL GEOSJON Generated!

2. Manage the data

2. Manage the data*

3. Visualize the data

3. Visualize the data

3. Visualize the data

3. Visualize the data

3. Visualize the data

So far...

In the most of the cities, the ratio woman-man is around 15%-85%

Collaborative Project

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

4. Opportunities

#TheStreetsOfWomen MIT 2020

More from Jessica Sena