Wikipedia and

Wikidata access with Python

Toni Hermoso Pulido

@toniher

Amical Wikimedia

Wiki

Wikipedia

Wikimedia

MediaWiki

Wiki CMS

Built mostly in:

• PHP
• JavaScript

https://www.mediawiki.org

Working with Docker

https://docker.com/

Getting started with Docker

Docker images used

https://github.com/toniher/docker-SemanticMediaWiki

https://github.com/toniher/docker-debian-python-mediawiki

We build the one above following README instructions, modifying Bash scripts and Dockerfile if desired

We can build this one as well. Alternately we can reuse it from:

https://hub.docker.com/r/toniher/debian-python-mediawiki/

However you need to mount example scripts provided in the repo above

Bots

The hidden ones

https://www.mediawiki.org/wiki/Manual:Bots

MediaWiki API

Accessing MediaWiki programatically

https://ca.wikipedia.org/w/api.php

https://www.mediawiki.org/wiki/API:Main_page

MediaWiki API clients / libraries

• Python (Mwclient, Pywikibot)

• Perl (MediaWiki::API, MediaWiki::Bot)

• JavaScript (nodemw)

• etc.

https://www.mediawiki.org/wiki/API:Client_code

Mwclient

A MediaWiki API Python client library

http://mwclient.readthedocs.io/en/master/

RESTBase

A sophisticated Wikipedia REST API

https://ca.wikipedia.org/api/rest_v1/

Wikidata

Wikidata: Namespaces

• «Wiki ones»: User, Wikidata, Category, etc.
• Wikibase ones: Item, Property (both entities)

Special:AllPages

Wikidata: Concepts

• Labels
• Descriptions
• Aliases
• Sitelinks
• Statements

Wikidata Glossary

Wikidata: Concepts

Reelin

Wikidata: Statements

https://www.wikidata.org/wiki/Q3454165

RDF

(Resource Description Framework)

Wikidata API access

MediaWiki API access: Like any other MediaWiki.

https://wikidata.org/w/api.php

Access via JSON (or other) exports

SPARQL endpoint access

Wikidata API clients

• Wikidata Client Library for Python
• Wikidata Toolkit (Java)
• WikidataR (R)
• Wikidata-SDK (Javascript)
• Catmandu Wikidata (Perl)

Linked (Open) Data

http://lod-cloud.net/

SPARQL: Endpoints and interfaces

Wikidata SPARQL inteface

query.wikidata.org

Interface to multiple endpoints

Yasgui

Example query:

• Children of Spanish politicians (both in Wikidata)

SPARQL: API Access

Example for Wikidata: https://query.wikidata.org/sparql

SPARQL client in Python

SPARQLWrapper

Wikimedia Hackathon 2018

https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2018

#wmhack

18-20 May