Exposing
a Flexible, composable & Extensible
REST API

Thierry Delprat

td@nuxeo.com
https://github.com/tiry/

Agenda

Quick introduction
- provide some context
API design constraints & principles
- explain the problem we want to solve
Building Nuxeo API
- REST + Automation + Composition
Design consequences
- price of flexibility

Some Context

What we Do and What Problems We Try to Solve

Nuxeo

we provide a Platform that developers can use to
build highly customized Content Applications

we provide components, and the tools to assemble them

everything we do is open source

various customers - various use cases

me: developer & CTO - joined the Nuxeo project 10+ years ago

Track game builds

Electronic Flight Bags

Central repository for Models

Food industry PLM

https://github.com/nuxeo

Nuxeo Platform

Repository

Services

Workflows, Conversions, Diff, Notifications, Activity ...

Why API is Key for us

Nuxeo Repository is a backend
- Portals, Mobile Apps, ERP, CRM ...
API is UI
- for the developers
- HTML5/JS

Plugability Challenge

In Nuxeo architecture everything is a plugin
- Nuxeo Server can provide a single service or 100's of services
Everything is configurable
- Logic and Data Structrures depends on configuration

API Challenge

"One API"
but

Multiple combinations
of
services, plugins
and Domain Models

Expose a Platform: not an application

developers using the platform
want to expose the API of their Application

Need to find a solution

One Platform "One API"

REST API
Design Principles

what we want to have

Be Efficient

Avoid round trips
- Get all needed data in one call
- Resolve some data on the server side
Avoid fetching too much data

Be Flexible

Adapt to the server side configuration
- Domain model definition
Adapt to client side requirements
- Provide data for the screen mapping
- Application can have different flavors

Keep it clean

Work between transaction boundaries
- do all the work in one call
Ensure isolation
- Other users should not see inconsistent data
Maintain encapsulation
- Client should not make assertion on server implementation

Client consumes a service, it does not build the service.

Be Extensible

Expose any meaningful business API
- Make API clean and application maintenance easy
Adapt API granularity to the target Applications
- one API behind each single button

We can not build the target Business API: users/devs will do it

Server

Manage the meta-model
Choose what API is exposed

Clients

Ask for the data they need
Use custom API

(versioned software artifact)

Client driven

Server controlled

"open bar" seems too messy

"one-size-fits-all" does not work

Balance client/server roles

Blob Friendly

Chunked & Out of band upload
Cachable and Seekable download

Content-Type: multipart/mixed

Be sensible

Do not lose our soul
- fight to keep the dynamicity of the platform!
No REST integrism
- Useful is more important than Beautiful
Dogfooding is key
- if this is not good enough internally, this is not good
Building API is part of the development cycle
- adding http API should never be a task for later

Principles

Build something that works

Disclaimer

Actually, just the chronology has been adjusted !

Building THE REST API

Exposing Resources

ExposING resources

Platform

Target use cases are not defined

Configurable

Target Domain Model is unknown

Expose raw technical resources !

Expose Use Cases !?

Expose the Domain Model !?

Expose simple resources

GET /repo/{repoId}/path/{docPath} HTTP 1.1

GET /repo/{repoId}/id/{docId} HTTP 1.1

GET /user/{userName} HTTP 1.1

GET /group/{groupName} HTTP 1.1

GET /directory/{directoryName}/{entryId} HTTP 1.1

GET /workflowModel/{modelName} HTTP 1.1

GET /workflow/{workflowInstanceId} HTTP 1.1

GET /task/{taskId} HTTP 1.1

Documents

Users &
Groups

Tasks &
Workflows

Expose raw resources as EndPoint with REST Bindings

Expose simple resources

EXPOSE SIMPLE RESOURCES

Get a Document

GET /nuxeo/api/v1/path/movies/star-wars HTTP/1.1

{
  "entity-type": "document",
  "repository": "default",
  "uid": "5b352650-e49e-48cf-a4e3-bf97b518e7bf",
  "path": "/movies/star-wars",
  "type": "MovieCollection",
  "isCheckedOut": true,
  "title": "Star Wars",
  "facets": [
    "Folderish"
  ]
}

Server returns a minimal payload

Adaptative marshaling

Client need to control what data schemas are sent

Adaptative marshaling

Control what data schemas are sent to the client

GET /nuxeo/api/v1/path/movies/star-wars HTTP/1.1
X-NXProperties dublincore, common

{
  "entity-type": "document",
  "repository": "default",
  "uid": "5b352650-e49e-48cf-a4e3-bf97b518e7bf",
  "path": "/movies/star-wars",
  "type": "MovieCollection",
  "isCheckedOut": true,
  "title": "Star Wars",
  "properties": {
    ...
    "common:icon": "/icons/movieCollection.png",
    "dc:description": "Star Wars collection",
    "dc:creator": "tiry",
    "dc:modified": "2015-10-22T02:12:59.07Z",
    "dc:lastContributor": "tiry",
    "dc:created": "2015-10-22T02:12:59.07Z",
    "dc:title": "Star Wars",
    ...
    "dc:contributors": [tiry, "system" ]
  },
  "facets": [
    "Folderish"
  ]
}

Fetching CONTEXTUAL data

Client may require more data
- get Document children at the same time
- get the breadcrumb data
- get thumbnail or preview url
- ...
Client ask for the data
- using Headers
- using Query String parameters

Fetching CONTEXTUAL data

Marshaling registry is pluggable

custom Enrichers can be contributed

"How the data is fetched"
is a server side matter

Fetching CONTEXTUAL data

GET /nuxeo/api/v1/path/movies/star-wars HTTP/1.1
X-NXenrichers.document: thumbnail

{
  "entity-type": "document",
  "repository": "default",
  "uid": "5b352650-e49e-48cf-a4e3-bf97b518e7bf",
  "path": "/movies/star-wars",
  "type": "MovieCollection",
  "isCheckedOut": true,
  "title": "Star Wars",
  "contextParameters":
   {
     "thumbnail":
     {
       "url": "/nuxeo/nxthumb/default/5b352650-e49e-48cf-a4e3-bf97b518e7bf/thumb:thumbnail/Small_photo.jpg"
     }
  },
  "facets": [
    "Folderish"
  ]
}

GET /nuxeo/api/v1/path/movies/star-wars?enrichers.document=thumbnail HTTP/1.1

Retrieve Linked Data

Resolve entity fields
- pointing to a label
- pointing to an other Document
- pointing to a User
- ...

Implicit JOIN

Retrieve Linked Data

Use client side parameter to know what to resolve
- header
- QueryString parameter
Can be recursive
- client need to control that too!

fetch.objectType=fieldToFetch

translate.objectType=fieldToTranslate

depth=children

Retrieve Linked Data

Adapters

Change the return type
- get only ACLs or History info about the Document
- get the tasks associated to document
Use your own business object
- use business Adapters
  - wrap document or documents
  - provide custom marshaling

GET /nuxeo/api/v1/path/movies/star-wars@acl HTTP/1.1

GET /nuxeo/api/v1/path/movies/star-wars@audit HTTP/1.1

GET /nuxeo/api/v1/path/movies/star-wars@bo/MyBusinessObject HTTP/1.1

Adapters

{
    entity-type: "MovieCollection"
    id: "5b352650-e49e-48cf-a4e3-bf97b518e7bf",
    "title": "Star Wars"
    "episodes": 7
}

GET /nuxeo/api/v1/path/movies/star-wars@bo/MovieCollection HTTP/1.1

Blobs

Sent as links
- Digest
- CDN
Uploaded out-of-band
- chunking
- reference in JSON

Blob Upload

Upload EndPoint
Reference Blobs from JSON Payload

{"entity-type": "document",
 "properties": {
  {
   "file:content" : {
     "upload-batch' : "0b0061d48f69b072",
     "upload-fileId" : 0,
     "type" : "blob"
    }
}}

POST /api/v1/upload/{batchId}/{fileIdx} HTTP 1.1
X-Upload-Chunk-Index 0 
X-Upload-Chunk-Count 5

PUT /nuxeo/api/v1/path/movies/star-wars HTTP/1.1

Are We Happy with that ?

Efficiency
- we can get all data in one call
Flexibility
- we can configure the data we want

Extensibility: partial
- enrichers, resolvers & adapters are not always enough
Coverage: poor
- 100+ services and only 5 endpoints
- not everything is CRUD

Need a way to map 100+ Services

Without creating 100 endpoints!

Need an other paradigm !

aUTOMATION api

Exposing service API over HTTP

Principles

Build a coarse gained API on top of service Java API
- select simple
Shell like commands !
- each service can contribute

> commandA(p1,p2) | commandB(p3,p4)

Commands

Command synopsis

Command

INPUT
(Doc, Blob, User ...)

OUTPUT
(Doc, Blob, User ...)

Parameters

Context
(User, Doc ...)

Commands

WebUI.AddErrorMessage   WebUI.AddInfoMessage   WebUI.AddMessage   Document.AddPermission   Document.AddToCollection   DocumentMultivaluedProperty.addItem   Task.ApplyDocumentMapping   Blob.AttachOnDocument   BlobHolder.AttachOnCurrentDocument   AttachFiles   Audit.QueryWithPageProvider   Blob.ImportClipboard   Blob.ImportWorklist   Blob.RunConverter   Document.BlockPermissionInheritance   WorkflowModel.BulkRestartInstances   Business.BusinessCreateOperation   Business.BusinessFetchOperation   Business.BusinessUpdateOperation   Navigation.GoBack   WorkflowInstance.Cancel   Navigation.ChangeCurrentTab   Document.CheckIn   Document.CheckOut   Update.NextStep.ConditionalFolder   WebUI.ClearClipboard   WebUI.ClearSelectedDocuments   WebUI.ClearWorklist   WorkflowTask.Complete   Blob.ConcatenatePDFs   Context.FetchDocument   Context.FetchFile   Blob.ToPDF   Blob.Convert   Document.Copy   Document.Create   FileManager.Import   UserWorkspace.CreateDocumentFromBlob   Seam.CreateDocumentInUI   Picture.Create   Document.CreateLiveProxy   Document.AddRelation   Collection.Create   Workflow.CreateRoutingTask   Task.Create   Directory.CreateEntries   Document.Delete   Document.DeleteRelation   Directory.DeleteEntries   WebUI.DestroySeamContext   Repository.GetDocument   Document.Export   WebUI.DownloadFile   Blob.ExportToFS   Document.FetchByProperty   Blob.CreateFromURL   FileManager.ImportInSeam   FileManager.ImportWithMetaData   FileManager.ImportWithMetaDataInSeam   Document.Filter   Document.FollowLifecycleTransition   Comment.Moderate   Document.GetBlobs   Document.GetChild   Document.GetChildren   Document.GetBlob   Document.GetBlobsByProperty   User.GetUserWorkspace   Document.GetLinkedDocuments   Proxy.GetSourceDocument   User.Get   Document.GetParent   Context.GetEmailsWithPermissionOnDoc   Context.GetTaskNames   Context.GetUsersGroupIdsWithPermissionOnDoc   Document.GetVersions   Directory.Projection   Collection.Suggestion   User.GetCollections   Directory.Entries   Directory.SuggestEntries   Collection.GetDocumentsFromCollection   Favorite.GetDocuments   Document.Routing.GetGraph   Picture.GetView   Workflow.GetOpenTasks   Tag.Suggestion   Task.GetAssigned   UserGroup.Suggestion   Document.GetRendition   Blob.PostToURL   Image.Blob.Resize   WebUI.InitSeamContext   JsonStack.ToggleDisplay   Actions.GET   GetRepositories   Document.Lock   Log   Audit.LogEvent   Auth.LoginAs   Auth.Logout   Document.Move   Document.PublishToSections   NRD-AC-PR-ChooseParticipants-Output   NRD-AC-PR-LockDocument   NRD-AC-PR-UnlockDocument   NRD-AC-PR-ValidateNode-Output   NRD-AC-PR-force-validate   NRD-AC-PR-storeTaskInfo   WebUI.NavigateTo   NuxeoDrive.SetActiveFactories   NuxeoDrive.AddToLocallyEditedCollection   NuxeoDrive.AttachBlob   NuxeoDrive.CanMove   NuxeoDrive.CreateFile   NuxeoDrive.CreateFolder   NuxeoDrive.CreateTestDocuments   NuxeoDrive.Delete   NuxeoDrive.FileSystemItemExists   NuxeoDrive.GenerateConflictedItemName   NuxeoDrive.GetRoots   NuxeoDrive.GetChangeSummary   NuxeoDrive.GetChildren   NuxeoDrive.GetClientUpdateInfo   NuxeoDrive.GetFileSystemItem   NuxeoDrive.GetTopLevelFolder   NuxeoDrive.GetTopLevelChildren   NuxeoDrive.Move   NuxeoDrive.SetSynchronization   NuxeoDrive.Rename   NuxeoDrive.SetVersioningOptions   NuxeoDrive.SetupIntegrationTests   NuxeoDrive.TearDownIntegrationTests   NuxeoDrive.UpdateFile   NuxeoDrive.WaitForElasticsearchCompletion   NuxeoDrive.WaitForAsyncCompletion   Repository.PageProvider   Context.PopDocument   Context.PopDocumentList   Context.PopBlob   Context.PopBlobList   Document.PublishToSection   Context.PullDocument   Context.PullDocumentList   Context.PullBlob   Context.PullBlobList   Context.PushDocument   Context.PushDocumentList   Context.PushBlob   Context.PushBlobList   WebUI.AddToClipboard   WebUI.PushDocumentToSeamContext   WebUI.AddToWorklist   LocalConfiguration.PutSimpleConfigurationParameters   LocalConfiguration.PutSimpleConfigurationParameter   Repository.Query   Audit.Query   Repository.ResultSetPageProvider   WebUI.RaiseSeamEvents   Blob.ReadMetadata   Context.SetMetadataFromBlob   Directory.ReadEntries   WebUI.Refresh   WebUI.Refresh   Document.RemoveACL   Services.RemoveDocumentTags   Document.RemoveEntryOfMultivaluedProperty   Blob.RemoveFromDocument   Document.RemovePermission   Document.RemoveProperty   Collection.RemoveFromCollection   Render.Document   Render.DocumentFeed   TemplateProcessor.Render   Document.ReplacePermission   Document.Reload   Picture.Resize   Context.RestoreDocumentInput   Context.RestoreDocumentsInput   Context.RestoreBlobInput   Context.RestoreBlobsInput   Document.RestoreVersion   Context.RestoreBlobInputFromScript   Context.RestoreBlobsInputFromScript   Context.RestoreDocumentInputFromScript   Context.RestoreDocumentsInputFromScript   Repository.ResultSetQuery   Document.Routing.Resume.Step   Workflow.ResumeNode   Counters.GET   RunOperation   RunDocumentOperation   Context.RunDocumentOperationInNewTx   RunFileOperation   RunOperationOnList   RunOperationOnProvider   RunOperationOnListInNewTx   RunInputScript   RunScript   WebUI.RunOperationInSeam   Document.Save   Seam.SaveDocumentInUI   Repository.SaveSession   SeamActions.GET   Document.Mail   Event.Fire   Document.AddACE   Context.SetVar   Context.SetInputAsVar   LocalConfiguration.SetSimpleConfigurationParameterAsVar   Document.Routing.SetRunningStepFromTask   Document.SetBlob   Document.SetBlobName   WebUI.SetJSFOutcome   Workflow.SetNodeVariable   Document.Routing.Step.Done   Document.Routing.BackToReady   Document.Routing.EvaluateCondition   Context.SetWorkflowVar   WebUI.ShowCreateForm   Document.CreateVersion   Context.StartWorkflow   Search.SuggestersLauncher   Services.TagDocument   Traces.Get   Traces.ToggleRecording   Document.SetMetadataFromBlob   Seam.GetChangeableDocument   Seam.FetchFromClipboard   Seam.GetCurrentDocument   Seam.GetCurrentDomain   Seam.GetCurrentWorkspace   Seam.FetchDocument   Seam.GetSelectedDocuments   Seam.GetDocumentsFromSelectionList   Seam.FetchFromWorklist   Document.Unlock   Document.UnblockPermissionInheritance   Services.UntagDocument   Document.Update   Document.SetProperty   Document.Routing.UpdateCommentsInfoOnDocument   Directory.UpdateEntries   Workflow.UserTaskPageProvider   VersionAndAttachFile   VersionAndAttachFiles   Blob.SetMetadataFromDocument   Blob.SetMetadataFromContext   Blob.CreateZip   acceptComment   addCurrentDocumentToWorklist   blobToPDF   cancelWorkflow   conditionalTask   decideNextStepAndSimpleValidate   downloadFilesZip   evaluateCondition   followLifeCycleTransition   followLifeCycleTransitionTask   initInitiatorComment   logInAudit   nextAssignee   notifyInitiatorEndOfWorkflow   publishDocument   publishTask   reinitAssigneeComment   rejectComment   Workflow.RemoveRoutingTask   sendTaskCreatedNotificationMail   setDone   setNextStep   setTaskDone   simpleChooseNextOption1AndDone   simpleChooseNextOption2AndDone   simpleRefuse   simpleTask   simpleUndo   simpleValidate   terminateWorkflow   undoRunningTask   updateCommentsOnDoc   validateDocument   voidChain   xmlExportRendition   zipTreeExportRendition

Favorite.GetDocuments

Blob.ToPDF

Image.Blob.Resize

Document.AddRelation

Workflow.CreateRoutingTask

lot of contributed operations

Principles

Commands as REST resources

GET to retrieve definition
POST to execute

Get an Operation

GET /nuxeo/api/v1/automation/Document.PageProvider HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
{
    "id":"Document.PageProvider",
    "label":"PageProvider",
    "description":"Perform a query ...",
    "signature":[ "void", "documents" ],
    "params":[
      {  "name":"page",
         "type":"integer",
         "required":false
      },{
         "name":"query",
         "type":"string",
         "required":false, },
      ... ]
}

Get an Operation

Run an Operation

POST /nuxeo/api/v1/automation/Document.PageProvider HTTP/1.1
Content-Type: application/json+nxrequest
{ "params" :
    { "query" : "select * from Note",
      "page" : 0
    }
}

HTTP/1.1 200 OK
Content-Type: application/json
{
  "entity-type": "documents",
  "pageIndex": 0,
  "pageSize": 2,
  "pageCount": 2,
  "entries": [
    {
      "entity-type": "document",
      "repository": "default",
      "uid": "3f76a415-ad73-4522-9450-d12af25b7fb4",
      ...
    }, { ...}, ...
 ]
}

Resources & Automation

Share the marshaling layer and extension
- Enrichers, Resolvers are available too
Compose Resources and Automation API
- Pipe Resources as input for Automation Operation

> cat /doc/path/somedoc | command(p3,p4)

Resources & Automation

RESOURCES > AUTOMATION

POST /nuxeo/api/v1/path/somePath/@op/Blob.ToPDF HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/pdf
...

Are We Happy with that ?

Efficiency
Flexibility
Coverage
- all services and plugins can contribute
Extensibility
- good, but limited to Java Developers
Consistency
- still no way to align Application Transactions

More Composition

assemble API blocks without having to code

build business API

COMPOSABLE API

Expose the API that matches client needs

GOALS

Tailor the API to match application requirements
- one API behind every action / button
Allow business analysts or UI developers to tailor the API
- define what API is exposed
  - UI & Workflow needs

Automation Chain

Assemble operations in a chain
Pipe Output / Input
Give it a name
Call and execute within a
single transaction

Server side assembly

One Context

Assembling Chains

It does work !

Business users & Front end developers leverage this
- to expose custom API for their UI
- to build custom logic inside their Workflows
- to add automatic processing (listeners)

Actually it works almost too well
- users do awfully complicated things
- chains calling chains calling chains ...

Have We Created a Monster ?

Use Nashorn to assemble operations

Operations remain the building blocks
JavaScript is the glue code to assemble them

Go Further than the chain Model

better control of the flow

Are We Happy with that ?

Efficiency
Flexibility
Coverage
Extensibility
Consistency

DYNAMIC AND COMPOSABLE API

What about the trade-off ?

Dynamic API Comes with a price

Documentation is a challenge
- maintain up to date and exhaustive
Clients needs to deal with the dynamic aspect
- data mapping
- discover APIs
Debugging API calls can be challenging

Introspection is a requirement

Introspection

Trace feature
- Allow client to retrieve traces of nested executions (dev mode)
Introspect the Command API
- GET retrieve definition + doc
Add REST API to introspect configuration resources
- Document types, Schemas
- (Widgets & Forms)