Open Data Quality Assessment

A systematic literature review

Presented by: Catie Sahadath

Supervisor: Dr. Mary Cavanagh

EBC 8101 | 4 April 2015

Overview

• Introduction
• Methodology
• Results
• Conclusions
• Takeaways
• Discussion

Introduction

Image: https://teachlikeyoumemeit.wordpress.com/

RQ1: What features, elements, and characteristics are evaluated in the extant literature outlining frameworks for assessing the quality of open government data sets?

Methodology

Image: https://www.slideshare.net/lsaghafi/research-methods-in-education-and-
education-technology-prof-lili-saghafi-conference-on-education-media-design-and-technology

Search Concepts

Open
Government Data

Quality

Framework

AND

(government NEAR open NEAR data)

((quality OR assess* OR evaluat* OR metric OR measur* OR benchmark OR barrier OR imped* OR usability)

AND

(framework OR model* OR standard OR method* OR indicator))

OR “ISO 25012” OR “ISO25012” OR “Data quality model” OR “OURdata index”)

Databases

Fields:

• Titles
• Keyword (author supplied, publisher supplied)
• Subject headings
• Abstract

Open First Methodology

Order of preference for database search interface:

1. Open search application programming interface (API)
    
2. Discriminatory API (E.g. requires registration, API key)
    
3. Database graphical user interface (GUI) command line search
    
4. Database GUI advanced search
    
5. Database GUI basic search

Exclusion Criteria:

E1: Included articles must address all three of open, government, and data

E2: Included articles must assess/evaluate datasets (not portals, websites, policies)

Eligibility / Inclusion

Selected works must explicitly list elements used in data assessment.

Selected works may be peer-reviewed articles, grey literature, or conference publications.

Literature Selection (PRISMA, 2009)

Total search results: 3 111

Total results after de-duplication: 2 771

Total results after E1 exclusions: 553

Total results after E2 exclusions: 78

Total eligible results: 29

Identification

Screening

Eligibility

Included

Data extraction

• Qualitative data extraction
• Vocabulary based on ISO/IEC 25012:2008
  Software engineering - Software product Quality Requirements (SQuaRE) - Data quality model

Recoverability

Efficiency

Confidentiality

Portability

Availability

Currentness

Compliance

Completeness

Accuracy

Traceability

Understandability

Credibility

Consistency

Precision

Accessibility

Results

Image: http://www.psychologywizard.net/inferential-statistics-ao1-ao2.html

RQ1: What features, elements, and characteristics are evaluated in the extant literature outlining frameworks for assessing the quality of open government data sets?

Recoverability (0)

Efficiency (0)

Confidentiality (0)

Portability (25)

Availability (21)

Currentness (20)

Compliance (18)

Completeness (16)

Accuracy (11)

Traceability (9)

Understandability (8)

Credibility (5)

Consistency (3)

Precision (3)

Accessibility (2)

n=29

Conclusions

• Language for quality assessment is not standardized
• Emphasis on:
  • Linked data (portability)
  • Non-proprietary data formats
  • Freely downloadable

Future directions

• How do these assessment criteria compare to the requirements for academic research data?
• Second paper outlining Open First Methodology

Key Takeaways

• Address biases to numeric data
• Now know the time commitment requirement for the Open First Methodology; plan accordingly
• Contradiction between committing to open methodology and using proprietary statistics software

Discussion

Image: https://memegenerator.net/instance/73910849/hank-hill-do-you-want-it-done-quick-or-do-you-want-it-done-right

Image credits:

Introduction: https://teachlikeyoumemeit.wordpress.com/

Methodology: https://www.slideshare.net/lsaghafi/research-methods-in-education-and-education-technology-prof-lili-saghafi-conference-on-education-media-design-and-technology

Results: http://www.psychologywizard.net/inferential-statistics-ao1-ao2.html

Discussion: https://memegenerator.net/instance/73910849/hank-hill-do-you-want-it-done-quick-or-do-you-want-it-done-right

References:

International Standards Organization. (2008). Software engineering - Software product Quality Requirements (SQuaRE) - Data quality model. ISO/IEC 25012:2008.

PRISMA . (2009). PRISMA Flow Diagram. Web. Accessed 2018-03-14. http://prisma-statement.org/documents/PRISMA%202009%20flow%20diagram.pdf

SLR Bibliography:

https://metacate.github.io/SLRbibliography/Presentation.html