Tomasz Ducin

23rd April 2016, Szczecin

JSON Schema

validate, generate, mock & automate

Tomasz Ducin

3rd November 2015, Warsaw

JSON taki albo owaki...

#169

JSON Schema

czyli kontrolowanie struktur z

Controlling Communication Structures:

JSON Schema

Tomasz Ducin

18th October 2015, Ossa/Rawa Mazowiecka 

JSON Schema

Tomasz Ducin

31th August 2015, Warsaw

#50

Tomek Ducin

JavaScript, Java, Python

senior software consultant @ Cybercom Poland

trainer @ Bottega IT Solutions

blah, blah, blah...

@tomasz_ducin

ducin

ducin.it

Agenda

  1. JSON
  2. schemas
  3. JSON Schema
  4. JSON Schema - Tool Stack

Douglas Crockford

creator of JSON

JSON:

brought to you by

Java Script Object Notation

  • defining object literals: {} and []
  • their values might be:
    • string (enclosed with ")
    • number (double type)
    • constant: false, true, null
  • allows unlimited nesting
  • flexible: will accept everything

Data Serialization Format

JS

N

YAML

used by

 RFC 7159 vs ECMA-404

QUIZ - is this a valid JSON?

  • {}
  • []
  • [1,2,3]
  • {a:1, b:2}
  • {'a':1, 'b':2}
  • {"a":1, "b":2}
  • [{1:2}, []]
  • [true, false, undefined]
  • ["WJUG hell yeah!"]
  • "WJUG halleluyah!"
  • true
  • Number(true)

solution architect

JavaScript is dynamically-typed

so is JSON

no compile-time check

let's modify JSON response

but why?

oh, because so!

FE

BE

What is more important?

$

<?xml version="1.0" encoding="UTF-8"?>

<shiporder orderid="889923"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="shiporder.xsd">
  <orderperson>John Smith</orderperson>
  <shipto>
    <name>Ola Nordmann</name>
    <address>Langgt 23</address>
    <city>4000 Stavanger</city>
    <country>Norway</country>
  </shipto>
  <item>
    <title>Empire Burlesque</title>
    <note>Special Edition</note>
    <quantity>1</quantity>
    <price>10.90</price>
  </item>
  <item>
    <title>Hide your heart</title>
    <quantity>1</quantity>
    <price>9.90</price>
  </item>
</shiporder>
<xs:element name="shipto">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="name" type="xs:string"/>
      <xs:element name="address" type="xs:string"/>
      <xs:element name="city" type="xs:string"/>
      <xs:element name="country" type="xs:string"/>
    </xs:sequence>
  </xs:complexType>
</xs:element>
<?xml version="1.0" encoding="UTF-8" ?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:element name="shiporder">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="orderperson" type="xs:string"/>
      <xs:element name="shipto">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="name" type="xs:string"/>
            <xs:element name="address" type="xs:string"/>
            <xs:element name="city" type="xs:string"/>
            <xs:element name="country" type="xs:string"/>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
      <xs:element name="item" maxOccurs="unbounded">
        <xs:complexType>
          <xs:sequence>
            <xs:element name="title" type="xs:string"/>
            <xs:element name="note" type="xs:string" minOccurs="0"/>
            <xs:element name="quantity" type="xs:positiveInteger"/>
            <xs:element name="price" type="xs:decimal"/>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
    </xs:sequence>
    <xs:attribute name="orderid" type="xs:string" use="required"/>
  </xs:complexType>
</xs:element>

</xs:schema>

communication between components relies on rigid structures

what if the structure changes from one side?

that's where schema comes in

API

Application Programming Interface

contract agreement - no changes forcing

I made 1500 100 900 apps and had no problems with JSON

where to use schemas?

  • api communication

  • configuration

  • internal app definitions

JSON Schema

  • XSD for XML

  • JSON Schema for JSON

JSON Schema

http://json-schema.org/

  • a specification

  • JSON used to define JSON

  • a meta-definition

{
  "type": "boolean"
}
{
  "type": "integer",
  "minimum": 100,
  "maximum": 200,
  "multipleOf": 13,
  "exclusiveMinimum": true
}
{
  "type": "string"
}
true
"WJUG"
169
{
  "type": "array",
  "minItems": 5,
  "items": {
    "enum": [
      "green",
      "blue",
      "yellow"
    ]
  }
}
[
  "green",
  "blue",
  "green",
  "green",
  "yellow",
  "green"
]
{
  "title": "Example Schema",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string"
    },
    "lastName": {
      "type": "string"
    },
    "age": {
      "description": "Age in years",
      "type": "integer",
      "minimum": 0
    }
  },
  "additionalProperties": false,
  "required": ["firstName", "lastName"]
}
{
  "firstName": "John",
  "lastName": "Lennon",
  "age": 40
}
{
  "firstName": "Paul",
  "lastName": "McCartney"
}
{
  ...
  "properties": {
    "storage": {
      "type": "object",
      "oneOf": [
        { "$ref": "#/definitions/diskDevice" },
        { "$ref": "#/definitions/diskUUID" },
        { "$ref": "#/definitions/nfs" },
        { "$ref": "#/definitions/tmpfs" }
      ]
    }
  }
  ...
}
$ref, oneOf, allOf, anyOf

JSON Schema Tool Stack

JSON Schema-based software

http://json-schema.org/implementations.html
  • Validators (32)
  • Schema generation (5)
  • Data generation (4)
  • Data parsing (2)
  • UI generation (3)
  • Hyper-schema handling (2)
  • Documentation generation (3)
  • Editors (2)
  • Compatibility (1)
  • Other (2)

JSON Schema online generator

  http://jsonschema.net/

JSON Schema online validator

  http://jsonschemalint.com/

JSON Schema
UI generator

  http://jeremydorn.com/json-editor/

https://github.com/jdorn/json-editor

Python validator

jsonschema

https://pypi.python.org/pypi/jsonschema

  • 2.6+, 3*
  • small and extensible
  • Draft 3  and  Draft 4 full support
  • Lazy validation - iterating through  all validation errors.
  • contextual information on what exactly failed (atomic value vs atomic schema)

https://github.com/Julian/jsonschema 

>>> from jsonschema import validate
>>> schema = {
...     "type" : "object",
...     "properties" : {
...         "price" : {"type" : "number"},
...         "name" : {"type" : "string"},
...     },
... }
>>> # SUCCESS: nothing happens :)
>>> validate({"name" : "Eggs",
  "price" : 34.99}, schema)
>>> # ERROR
>>> validate({"name" : "Eggs",
  "price" : "Invalid"}, schema)
Traceback (most recent call last):
    ...
ValidationError: 'Invalid' is not
  of type 'number'

Python DSL

jsl

https://pypi.python.org/pypi/jsl

  • Python DSL for defining JSON schemas
  • Python classes dumped to dictionaries

https://github.com/aromanovich/jsl

>>> class User(jsl.Document):
...     id = jsl.StringField(required=True)
...     login = jsl.StringField(required=True,
...         min_length=3, max_length=20)
>>> User.get_schema(ordered=True) # or False
{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "additionalProperties": false,
    "properties": {
        "id": {"type": "string"},
        "login": {
            "type": "string",
            "minLength": 3,
            "maxLength": 20
        }
    },
    "required": ["id", "login"]
}

Java validator

  http://json-schema-validator.herokuapp.com/

https://github.com/
  fge/json-schema-validator

online demo

<dependency>
    <groupId>com.github.fge</groupId>
    <artifactId>json-schema-validator</artifactId>
    <version>...</version>
</dependency>
import com.fasterxml.jackson.databind.JsonNode;
import com.github.fge.jsonschema.core.report.ProcessingReport;
import com.github.fge.jsonschema.main.JsonSchema;
import com.github.fge.jsonschema.main.JsonSchemaFactory;
...
final JsonNode fstabSchema = Utils.loadResource("my-schema.json");
final JsonNode good = Utils.loadResource("json-good.json");
final JsonNode bad = Utils.loadResource("json-bad.json");

final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
final JsonSchema schema = factory.getJsonSchema(fstabSchema);

ProcessingReport report;

report = schema.validate(good);
System.out.println(report);

report = schema.validate(bad);
System.out.println(report);

code example

more usage examples:
https://github.com/fge/json-schema-validator/tree/master/
src/main/java/com/github/fge/jsonschema/examples

Schema generator
from Java code

https://github.com/FasterXML/
    jackson-module-jsonSchema​

< Faster:XML />

Jackson JSON Schema Module

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-annotations</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.module</groupId>
    <artifactId>jackson-module-jsonSchema</artifactId>
    <version>...</version>
<dependency>
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
...

class POJO... {
       @JsonIgnore
       private String name;

       @JsonProperty
       public String getName() {
               return name;
       }

       @JsonIgnore
       public void setName(String name) {
               this.name = name;
       }

       ...
}

code example

       ...
       @JsonIgnore
       private CustomType type;

       @JsonProperty
       public CustomType getType() {
               return type;
       }

       @JsonIgnore
       public void setType(CustomType type) {
               this.type = type;
       }
{
  "type" : "array",
  "items" : {
    "type" : "object",
    "id" : "urn:jsonschema:mypackage:model:POJOClass",
    "properties" : {
      "name" : {
        "type" : "string"
      },
      "type" : {
        "type" : "string",
        "enum" : [
          "COMPANY_INFO",
          "CUSTOMER_INFO",
          "TRADER_INFO",
          ...
        ]
      }
    }
  }
}
public enum CustomType {

        COMPANY_INFO("tcmpi"),
        CUSTOMER_INFO("tcsti"),
        TRADER_INFO("ttrdi"),
        ...
...
}

API Contracting

  • stored in repository
  • available in CI
  • used by many
    automates
  • URLs
  • HTTP methods
  • HTTP status codes
  • HTTP content-type
  • HTTP headers
  • content itself 
/songs
  get
  post
  /{songId}
    get
    /file-content
      get
      post
/artists
  get
  post
    /{artistId}
      get
      /albums
        get
/albums
  get
  post
    /{albumId}
      get
      /songs
        get
/songs:
  description: Collection of available songs in Jukebox
  get:
    description: Get a list of songs based on the song title.
    queryParameters:
      songTitle:
        description: "The title of the song to search (...)"
        required: true
        minLength: 3
        type: string
        example: "Get L"
[...]

Contract

Example

[...]
    responses:
      200:
        body:
          application/json:
            example: |
              "songs": [
                  {
                    "songId": "550e8400-e29b-41d4-a716-446655440000",
                    "songTitle": "Get Lucky"
                  },
                  {
                    "songId": "550e8400-e29b-41d4-a716-446655440111",
                    "songTitle": "Loose yourself to dance"
                  },
                  {
                    "songId": "550e8400-e29b-41d4-a716-446655440222",
                    "songTitle": "Gio sorgio by Moroder"
                  }
                  ]

API Contracting Tools

  • documentation generators
  • mock data
  • stub servers
  • test generators & runners

Contracting

Tool Stacks

API Blueprint

Swagger

RAML:
RESTful API Modelling Language 

[...]
    responses:
      200:
        body:
          application/json:
            schema: |
              {
                "type": "object",
                "$schema": "http://json-schema.org/draft-03/schema",
                "id": "http://jsonschema.net",
                "required": true,
                "properties": {
                  "songTitle": {
                    "type": "string",
                    "required": true
                  },
                  "albumId": {
                    "type": "string",
                    "required": true,
                    "minLength": 36,
                    "maxLength": 36
                  }
                }
              }

Swagger & RAML
use JSON Schema

API Contracting

vs

Consumer-Driven Contracts

imbues providers with insight into their consumer obligations, and focuses service evolution around the delivery of the key business functionality demanded by consumers.

- Martin Fowler (here)

mock your API

fake data
online generator

  http://www.json-generator.com/

node.js-based

fake data generator

https://github.com/
   json-schema-faker/
   json-schema-faker

json-schema-faker

http://json-schema-faker.js.org

fake data generator grunt plugin

https://github.com/
   json-schema-faker/
   grunt-jsonschema-faker

grunt-jsonschema-faker

  • it's not that trendy YET!!!
  • meta-definition
  • validation base
  • fake data generator
  • Java had it, and so does the rest
  • enterprise will ❤ it!

THX!

@tomasz_ducin

4  Q&A

tomasz@ducin.it