Adam Retter

adam@evolvedbinary.com
 

XQuery / eXist-db Bootcamp for Humanists
CIHAM and HiSOMA
University of Lyon @ Online
2021-03-22


@adamretter

Using REST

with eXist-db

About Me

  • Director and CTO of Evolved Binary

    • XQuery / XSLT / Schema / RelaxNG

    • Scala / Java / C++ / Rust*

    • Concurrency and Scalability

  • Creator of FusionDB multi-model database (5 yrs.)

  • Contributor to Facebook's RocksDB (5 yrs.)

  • Core contributor to eXist-db XML Database (15 yrs.)

  • Founder of EXQuery, and creator of RESTXQ

  • Was a W3C XQuery WG Invited expert

  • Me: www.adamretter.org.uk

FusionDB, Elemental, and eXist-db

  • What are these, and how are they related?

    • FusionDB

      • Uses eXist-db components - 100% eXist-db API Compatible

      • Improved Storage, ACID Transactions, Instant Backup, etc.

      • Target - Larger projects and organisations

      • Source Available - Free For Dev. / Requires Licenses for Production Use

    • Elemental

      • Hard-fork of eXist-db - 100% eXist-db API Compatible (Release Q2 2021)

      • New Features and Fixes. Community Roadmap driven.

      • Target - Smaller projects and users

      • Source Available - Completely Free to Use

    • eXist-db

      • No one owns it

      • Open Source - Completely Free to Use

What we will cover today...

  • What is REST?

  • eXist-db's REST Server

  • Using eXist-db's REST API

  • How to design your own REST API

  • Plain REST vs. controller.xq vs. RESTXQ

What is REST?

  • REST - Representational State Transfer

    • ~ Architectural Styles and the Design of Network-based Software Architectures. Fielding, Roy. 2000.

    • A set of Principles, Properties, and Constraints

  • Helps guide us to build well-designed Web Applications

  • Focused on the use of HTTP

  • RESTful Applications are applications that adhere to REST principles

    • Not unusual for applications to be partially RESTful

    • You likely use RESTful applications every day via your Web Browser

REST - Architectural Constraints

  • Client / Server Architecture

    • Separate the UI concerns from the Storage concerns

    • The Web does this! i.e. Web Browser and Web Server

  • Statelessness

    • Every request is isolated and can stand alone

  • Cacheability

  • Layered System

  • Code on Demand

  • Uniform Interface

    • URIs, HTTP Headers, and HyperMedia

REST - Statelessness

  • Refers to Extrinsic State (Application State)

  • Specifically, NOT keeping Application State at the Server

    • Client maintains its own application state instead

    • Sends what is needed in each request

  • Reduces load on the Server

    • No Session needed per-user

    • Reduces conditional computation

  • Horizontal Scaling

    • Any server can process any request

Keeping Application State at the Server

This breaks Statelessness for REST!

Establish Session

Keeping Application State at the Client

Optional

REST - Uniform Interface

  • URI

    • One per resource (or collection of resources)

      • All apples (or the only apple): /apple

      • A specific apple: /apple/pink-lady

  • Manipulation

    • Verbs = HTTP Methods: GET, HEAD, PUT, POST, PATCH*, DELETE

  • Self-Descriptive Messages

    • Media Type

    • HTTP Headers: Content-Type, Accept

  • Hypermedia (HATEOS)

    • Response should include links to further resources

    • e.g. HTML

REST - HTTP Methods

  • GET

    • Retrieve a (representation of a) Resource(s)

  • HEAD

    • Get but with no representation (body), just metadata (headers)!

  • PUT

    • Store/Replace a Resource at URI

  • POST

    • Store a Resource as subordinate of URI

    • Update a Resource (consider PATCH instead)

  • DELETE

    • Delete a Resource(s)

REST - HTTP Headers for Media Types

  • Content Type

    • Indicates the Internet Media Type (Mime Type) of the Request/Response Body

    • e.g. Response from GET https://www.google.com:



    • Client POST/PUT/PATCH - tells the server the Media Type of the Resource it is sending

    • Server - tells the client the Media Type of the Resource it is returning

  • Accept

    • Indicates one or more Internet Media Types that the client will accept for the Response Body

Content-type: text/html; charset=ISO-8859-1

REST - Hypermedia

  • Hyperlinks - Not just HTML!

  • HTML

    • Unbiquitous - understood by your Web-browser

<h2>Apples</h2>
<ul>
    <li>
    	<a href="/apple/pink-lady">Pink Lady</a>
    </li>
    <li>
    	<a href="/apple/golden-delicious">Golden Delicious</a>
    </li>
    <li>
    	<a href="/apple/hollowcore">Hollowcore</a>
    </li>
</ul>
  • JSON

[
  {"name": "Pink Lady", "href": "/apple/pink-lady"},
  {"name": "Golden Delicious", "href": "/apple/golden-delicious"},
  {"name": "Hollowcore", "href": "/apple/hollowcore"}
]

REST - Hypermedia

  • XML

    • Syntax for linking is up to you! Various standards exist

<apples xmlns:xlink="http://www.w3.org/1999/xlink">
    <apple xlink:href="/apple/pink-lady">
      <name>Pink Lady</name>
    </apple>
    <apple xlink:href="/apple/golden-delicious">
      <name>Golden Delicious</name>
    </apple>
    <apple xlink:href="/apple/hollowcore">
      <name>Hollowcore</name>
    </apple>
</apples>

eXist-db HTTP APIs

eXist-db REST Server

  • API Base URI: /exist/rest/db

  • REST API for Document Database CRUD Operations

    • Get Collection

    • Create Document

    • Get Document

    • Replace Document

    • Update Document

    • Delete Document

    • Delete Collection

  • Additionally

    • Execute Dynamic Query

    • Execute Stored Query

eXist-db REST Server - Get Collection

  • HTTP GET http://localhost:8080/exist/rest/db/

  • Web Browser Example

❯ curl -v -X GET http://localhost:8080/exist/rest/db/

> GET /exist/rest/db/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 10:50:26 GMT
< Content-Type: application/xml; charset=UTF-8
< Last-Modified: Sun, 21 Mar 2021 10:49:57 GMT
< Created: Sun, 21 Mar 2021 10:49:57 GMT
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist">
    <exist:collection name="/db" created="2021-03-21T11:49:56.377+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x">
        <exist:collection name="system" created="2021-03-21T11:49:56.381+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x"/>
        <exist:collection name="apps" created="2021-03-21T11:49:57.135+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x"/>
    </exist:collection>
</exist:result>
  • cURL Example:

eXist-db REST Server - Get Collection

  • HTTP GET http://localhost:8080/exist/rest/db/

  • Python Example:

#!/usr/bin/python3

import http.client

conn = http.client.HTTPConnection("localhost", 8080)
conn.request("GET", "/exist/rest/db/")
response = conn.getresponse()

print(response.status, response.reason)
responseBody = response.read()
print(responseBody.decode())

conn.close()
200 OK
<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist">
    <exist:collection name="/db" created="2021-03-21T11:49:56.377+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x">
        <exist:collection name="system" created="2021-03-21T11:49:56.381+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x"/>
        <exist:collection name="apps" created="2021-03-21T11:49:57.135+01:00" owner="SYSTEM" group="dba" permissions="rwxr-xr-x"/>
    </exist:collection>
</exist:result>
    • Result:

eXist-db REST Server - Create Document

  • HTTP PUT vs. POST

    • Subtly different

    • Think - Document vs. Collection of Documents

    • Can be confusing at first!

  • HTTP PUT - to the URI of the Document

    • e.g. http://localhost:8080/exist/rest/db/apple/pink-lady.xml

    • NOTE: If document already exists, it will be replaced!

  • HTTP POST - to the URI of the Collection in which to store the Document

    • e.g. http://localhost:8080/exist/rest/db/

    • Response will tell you the new URI of the Document

    • WARNING: eXist-db does NOT support this correctly!

      • eXist-db uses POST only for XQuery and XUpdate.

eXist-db REST Server - Create (PUT) Doc.

  • HTTP PUT http://localhost:8080/exist/rest/db/apple/pink-lady.xml

  • cURL Example

❯ curl -v -X PUT -H "Content-Type: application/xml" -d "<apple><name>Pink Lady</name></apple>" \
    http://admin:@localhost:8080/exist/rest/db/apple/pink-lady.xml

> PUT /exist/rest/db/apple/pink-lady.xml HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/xml
> Content-Length: 23
>
< HTTP/1.1 201 Created
< Date: Sun, 21 Mar 2021 11:48:12 GMT
< Content-Length: 0
< Server: Jetty(9.4.38.v20210224)
<
    • TIP: Instead of specifying body inline with -d, use a file:

❯ curl -v -X PUT -H "Content-Type: application/xml" -d "@pink-lady.xml" \
    http://admin:@localhost:8080/exist/rest/db/apple/pink-lady.xml

eXist-db REST Server - Get Document

  • Similar to Get Collection

  • HTTP GET http://localhost:8080/exist/rest/db/apple/pink-lady.xml

  • cURL Example

❯ curl -v -X GET http://localhost:8080/exist/rest/db/apple/pink-lady.xml

> GET /exist/rest/db/apple/pink-lady.xml HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 12:31:09 GMT
< Last-Modified: Sun, 21 Mar 2021 11:48:13 GMT
< Created: Sun, 21 Mar 2021 11:48:13 GMT
< Content-Type: application/xml; charset=UTF-8
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<apple><name>Pink Lady</name></apple>

eXist-db REST Server - Update Document

  • Uses XUpdate to describe the changes to make

    • NOTE: eXist-db should probably use HTTP PATCH instead of POST!

  • HTTP POST http://localhost:8080/exist/rest/db/apple/pink-lady.xml

  • cURL Example

❯ curl -v -X POST -H "Content-Type: application/xml" \
    -d "<xu:modifications version='1.0' xmlns:xu='http://www.xmldb.org/xupdate'>
            <xu:update select='/apple/text()'>Just updated</xu:update>
        </xu:modifications>" \
    http://admin:@localhost:8080/exist/rest/db/apple/pink-lady.xml

> POST /exist/rest/db/apple/pink-lady.xml HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/xml
> Content-Length: 151
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 12:58:46 GMT
< Content-Type: application/xml; charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<?xml version="1.0" ?><exist:modifications xmlns:exist="http://exist.sourceforge.net/NS/exist" count="1">1 modifications processed.</exist:modifications>

eXist-db REST Server - Update Document

  • Can also be applied to an entire Collection!

    • HTTP POST http://localhost:8080/exist/rest/db/

  • Alternatives:

    • HTTP POST an XQuery containing XQuery Update expression(s)

    • HTTP GET or POST to a Stored XQuery

eXist-db REST Server - Delete Document

  • Similar to Get Document

  • HTTP DELETE http://localhost:8080/exist/rest/db/apple/pink-lady.xml

  • cURL Example

❯ curl -v -X DELETE http://admin:@localhost:8080/exist/rest/db/apple/pink-lady.xml
> DELETE /exist/rest/db/apple/pink-lady.xml HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 13:12:15 GMT
< Content-Length: 0
< Server: Jetty(9.4.38.v20210224)
<

eXist-db REST Server - Delete Collection

  • Similar to Get Collection

  • HTTP DELETE http://localhost:8080/exist/rest/db/apple

  • cURL Example

❯ curl -v -X DELETE http://admin:@localhost:8080/exist/rest/db/apple

> DELETE /exist/rest/db/apple HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 13:15:53 GMT
< Content-Length: 0
< Server: Jetty(9.4.38.v20210224)
<

eXist-db REST Server - Dynamic Query

  • You can either HTTP GET or HTTP POST

  • XQuery Context is the Collection or Document indicated by the URI

  • HTTP GET http://localhost:8080/exist/rest/db/?_query=/apple

    • Useful for simple/short queries

    • Query and Parameters are sent as URL Query String Parameters

    • Parameters must be URL Encoded

    • Variety of parameters available, e.g. _start, _howmany
      See eXist-db docs - Get Requests

  • HTTP POST http://localhost:8080/exist/rest/db/

    • Useful for complex/long queries

    • Query and Parameters are sent within HTTP Request Body

    • Variety of parameters available, See eXist-db docs - Post Requests

eXist-db REST Server - GET Query

  • cURL Example

❯ curl -v http://localhost:8080/exist/rest/db/?_query=/apple&_start=1&_howmany=10

> GET /exist/rest/db/?_query=/example HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 18:08:00 GMT
< Content-Type: application/xml; charset=UTF-8
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist" exist:hits="1" exist:start="1" exist:count="1"
              exist:compilation-time="0" exist:execution-time="0">
    <apple><name>Pink Lady</name></apple>
</exist:result>
    • TIP: specify _wrap=no if you want the raw results

eXist-db REST Server - POST Query

  • cURL Example

❯ curl -v -X POST -H "Content-Type: application/xml" -d "
<query xmlns='http://exist.sourceforge.net/NS/exist' start='1' max='10'>
    <text><![CDATA[
/apple
    ]]></text>
</query>" http://localhost:8080/exist/rest/db/

> POST /exist/rest/db/ HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/xml
> Content-Length: 126
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 18:30:06 GMT
< Content-Type: application/xml; charset=UTF-8
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<exist:result xmlns:exist="http://exist.sourceforge.net/NS/exist" exist:hits="1" exist:start="1" exist:count="1"
              exist:compilation-time="1" exist:execution-time="1">
    <apple><name>Pink Lady</name></apple>
</exist:result>
    • TIP: specify wrap="no" if you want the raw results

eXist-db REST Server - Stored Query

  • You can store an XQuery into the database like any other document!

  • Use the Content-Type: application/xquery

  • HTTP PUT http://localhost:8080/exist/rest/db/time.xq

    • cURL Example:

❯ curl -v -X PUT -H "Content-Type: application/xquery" -d "<now>{current-dateTime()}</now>" \
    http://admin:@localhost:8080/exist/rest/db/time.xq

> PUT /exist/rest/db/time.xq HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/xquery
> Content-Length: 31
>
< HTTP/1.1 201 Created
< Date: Sun, 21 Mar 2021 18:42:54 GMT
< Content-Length: 0
< Server: Jetty(9.4.38.v20210224)
<

eXist-db REST Server - Stored Query

  • Executable via HTTP GET or POST

  • HTTP GET http://localhost:8080/exist/rest/db/time.xq

    • cURL Example:

❯ curl -v -X GET http://admin:@localhost:8080/exist/rest/db/time.xq

> GET /exist/rest/db/time.xq HTTP/1.1
> Host: localhost:8080
> Authorization: Basic YWRtaW46
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sun, 21 Mar 2021 18:46:44 GMT
< X-XQuery-Cached: false
< Content-Type: application/xml; charset=UTF-8
< Vary: Accept-Encoding, User-Agent
< Transfer-Encoding: chunked
< Server: Jetty(9.4.38.v20210224)
<
<now>2021-03-21T19:46:44.393+01:00</now>

Designing your own REST API

  • Consider Resources first (the things you talk about)

    • Design URI that point to Resources and Collections of Resources

  • Consider Representations (of Resources) second

    • One or more representations, e.g. HTML, XML, JSON, etc.

    • Use HTTP Content-Type and Accept Headers to negotiate

  • Consider actions upon those resources

    • Apply HTTP Methods as Verbs to achieve the actions

  • Don't forget Hypermedia (i.e. navigability via Hyperlinks)

  • There are Models for evaluating RESTful'ness

Designing your own REST API with eXist-db

Using REST with eXist-db

By Adam Retter

Using REST with eXist-db

Presentation given for XQuery / eXist-db Bootcamp for Humanists. CIHAM and HiSOMA, University of Lyon - 22 March 2021 - Online

  • 1,496