Survey Report

Working with Open Data

Click on the right arrow to continue

In June 2019 I distributed a survey to companies that develop applications from open data in the European Union.

 

I found these companies on the EU Open Data Portal, European Data Portal, Open Data Incubator Europe (ODINE) and EU Datathon websites.

 

The survey’s goal was to understand more about their process and the issues they face.

 

Out of the 180 companies contacted, 36 completed the survey. This is what I found.

The majority of companies didn't receive external funding

Question: Have you received any funding for your project? If yes, who funded you?

No

Funding

Private

Funding

European

Authorities

National

Authorities

22

8

6

4

Many different types of applications were developed

Political Activity / Public Administration Transparency

Business Opportunities, Economics and Finance

Environment and Climate Change

Data Services / API

Legal

Consumer Product Safety

Data Storytelling

Food Consumption

Internet of Things Search Engines

Language

Pharmaceutical

Web Archive

10

8

5

3

3

1

1

1

1

1

1

1

Click on the down arrow to see how companies found the data

Google is used a lot when searching for data

18 of the 36 companies surveyed already knew where to find all the data they needed. 13 out of the remaining 18 companies used Google to search for data.

Existing Knowledge

Google Search

EU Open Data Portal

Asked Stakeholders, Experts, PA

European Data Portal

National Public Data Portals

EU Datathon Workshops

Ecosia Search

Academic Survey

21

13

6

Question: Did you know how to find the data? How did you find it?

3

3

3

2

1

1

Many different data sources were used

Click on the down arrow to see the data sources

More than 50 different data sources were mentioned by the 36 companies surveyed.

The majority of the companies used at least two data sources for their application.

 

 

National / Regional Open Data Sources [mentioned 11 times], Eurostat [7], European Central Bank [5], EUR-Lex [4], European Commission [3], European Environment Agency [3], European Parliament [3], TED [3], Transparency Register [3], Copernicus [2], CORDIS [2], EMODnet [2], Organisation for Economic Co-operation and Development [2], The World Bank [2], Wikidata [2], Alpha Vantage, CEPII, CRED EM-DAT (Emergency Events Database), Court of Justice of the EU, DGT - Translation Memory, European Chemicals Agency, European Medicines Agency, European Food Safety Authority, European Investment Bank, European Systemic Risk Board, Financial Transparency System, IATI Registry, Integrated Climate Data Center, NASA, National Oceanic and Atmospheric Administration, Our World in Data, Publications Office of the EU, Twitter, UN Comtrade, US Government, World Health Organization

Opinions about the difficulty of finding data varied...

Question: How difficult / easy was it to find the data you were looking for?

Difficult

Somewhat

Difficult

Very

Difficult

Average

Somewhat

Easy

Easy

Very

Easy

3

5

4

6

6

6

6

... and so did opinions about the quality of data

Question: How would you rate the quality of the data you found?

Bad

Somewhat

Bad

Very

Bad

Average

Somewhat

Good

Good

Very

Good

0

2

6

7

6

6

9

A large amount of work went into data preparation

Full processing pipeline, AI systems for data unification, Data discovery systems, Data cleaning, Data transformation, Data extrapolation, Data normalisation, Website data collection, Document transformation program, Machine-learning-processing, Editor for files and link sources, ETL data process, Data refresh, Data gathering and process, Text corpus processing, Missing data identification, Data review, Data extraction from XML, PDF parser to "read" documents and collect data in machine-readable format, Data manipulation, Data retrieval, Data analysis, Language detection, Data augmentation, Data access, Semantic descriptor linking, Streamlining of data input from multiple sources, Duplicate data removal, Data parsing, Raw data to RDF translation, Knowledge graph store, Data scraping, Data structuring, Data indexing, Data validation, Data conversion, Fuzzy string matching

This "word cloud" summarises the activities mentioned by the respondents

Difficulties were encountered when dealing with data

Unstructured data to be turned into structured data, Different kinds of data to be joined, Inconsistent data, Incomplete data, Errors in the data, Data to be turned into relevant metrics, Poorly standardised data, Reliability over time, Consistency of updates rate, Data standards and conventions changing over time, No commonality, Data to be related with geospatial structures, Fragmented data, Very different structures in each data source, Large amounts of data, Lack of metadata, Confusing metadata, No (or weak) response from data providers when suggesting improvements, Difficult to find the right (type of) data in the crammed EU Open Data Portal and other EU portals, Lack of documentation, Lack of recent data, Difficulty in combining different data sources due to inconsistencies / different granularity / different time periods covered, Difficult access to data due to lack of open data formats, Expert-level knowledge required to understand what data is available, Data to be parsed due to different standards used by data providers, Aggregating different levels of granularity, Conversion between wide and long format, Dealing with APIs, Open text fields with unstructured information, Discrepancies in data, Dealing with many different formats / protocols, Finding the data, Public Authorities not providing data although legally required to, Need of data not available with an open license

Python is king

Companies used a vast array of technologies to develop their applications, with Python being the most common.

Python [mentioned 18 times], R [8], d3.js [4], Node.js [4], PHP [4], PostgreSQL [4], Vue.js [4], Flask (Python) [3], Leaflet [3], MySQL [3], Shiny (R) [3], AngularJS [2], C [2], C# [2], C++ [2], Elasticsearch [2], ggplot2 [2], Go (Golang) [2], Java [2], MongoDB [2], NLTK (Python) [2], pandaSDMX (Python) [2], Scikit-learn (Python) [2], .NET, Amazon Aurora, Amazon Cognito, Apache Commons, AWS Lambda, Bash, Beautiful Soup (Python), CakePHP, Crossfilter (JS), Crypto (Node.js), data.table (R), dc.js, DigitalOcean, Django (Python), dplyr (R), Drupal, FNA Platform, Google Cloud Services, GraphQL, Heritrix, Hugo, JSON Hyper-Schema, Luigi (Python), lxml (Python), MariaDB, Microsoft SQL Server, Mono, Motu Client, Neo4j, NoSQL Database, NumPy (Python), OpenStreetMap, package countrycode (R), package haven (R), pdf2html, PostGIS, PostgreSQL, Power BI, QGIS, React, React Native, Redis, RocksDB, Ruby on Rails, Spring Framework, Spyder (Python), SQLite, SSDB, Tableau, Tesseract OCR, Windows Server, XGBoost

Many companies don't promote their work

Question: What are you currently doing (or have done) to promote your project to end users?

Networking / Contacting Stakeholders / Events

Nothing / Minimal Activity

Social Media

SEO

Traditional Media

Google Ads / PPC

Blog Posts

Not much, this is only an experiment

11

11

10

7

5

3

2

2

The number of end users is low for several companies

Note: these are the answers received from 16 companies. The other 20 companies surveyed either didn't know or didn't have data yet.

Question: How many end users interact with your project every month on average?

100 - 200 - 300 - 500 - 600 - 1,500 - 2,000 - 3,000 - 4,000 - 8,000 - 10,000 - 15,000 - 20,000 - 300,000 - 700,000 - 1,000,000

To conclude...

Quality and reliability of the data / Need to structure the data [mentioned 7 times], Understanding the market / Customer needs and behaviors [5], Lack of funding / Only spare time dedicated to the project [4], Deadlines [3], Large volume of data [3], Project's scope definition [2], Expertise in web design needed to be built from scratch, Building the ETL process, Clueless management, Creating an open source community, Data visualisation, Developing the database, Dispersed team, Interacting with APIs, Interpreting the data, Managing different languages, Marketing, Project maintenance, Reliability of PDF parsers, Resistance to public use of the data, Setting a mapserver's WCS supporting time dimension, Visual design and front-end

Question: What were the main difficulties / pain points you encountered when developing your project (technical issues, management issues, etc.)?

Understanding the processes and issues faced by companies working with open data is the start of the journey. Identifying what can be done to help them is the next step.

 

Do you have any suggestions?

What's next?

Please get in touch

hello@dvkwc.org