APIs on Ruby

(and possibly Rails)

Filippos Vasilakis

Web dev

Topics

1. Modern APIs
2. How to test efficiently
3. Performance!

Modern APIs

• embrace simplicity
• provide J-ORM to client over HTTP
• defaults for the most usual cases
• new features shouldn't affect API compatibility​

​You build the API for the client; not for yourself!

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

Back to basics...

What is a controller in Rails?

Back to basics...

What is a controller in Rails?

Back to basics...

Controllers are just Rack wrappers

1. receive input
2. delegate work to other service objects
3. return (display) as output the processed input

This is the request-response cycle

(and should be really really fast)

Back to basics...

Given that I have authenticated the user in the request, I need to know 3 things:

1. is user authorized for this action on that resource?
2. what params is user allowed to send (update) based on her role?
3. what attributes is user allowed to reveive based on her role ?

Tips:

• Authentication takes place even for unauthenticated requests.
• Authentication != Authorization

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

1. Sparse fields

GET /api/v1/videos?fields[title]=true&fields[description]=true&fields[links]=true
(JSONAPI: GET /api/v1/videos?fields=[title,description,links])

{
  "video": {
    "id": 3031,
    "title": "Leon på havet!",
    "description": " #bohuslän",
    "links":{
      "user": "/api/v1/users/18"
    }
  }
}

• flexibility for the client
• performance improvements in the server and client side
• one serializer/controller per endpoint

2. Granular Permissions

(as a guest): GET /api/v1/user/17
{
  "user": {
    "id": 17,
    "username": "robert",
    "name": "Robert M",
    "links": {
      "videos":"/api/v1/users/17/videos"
    }
  }
}

(as a regular user): GET /api/v1/user/17

{
  "user": {
    "id": 17,
    "username": "robert",
    "name": "Robert M",
    "description": "My videos! Just small stories.",
    "avatar": "4c657cfe-116d-4f3f-694d-9164e5294e72.jpg",
    "background": "a6648b87-c64e-d0b9-a75e-18b635a2a89e",
    "url": "http://www.example.com",
    "followers_count": 10,
    "followings_count": 16,
    "videos_count": 405,
    "links": {
      "videos":"/api/v1/users/17/videos"
    }
  }
}

• granular permissions without security concerns
• one serializer/controller per endpoint

3. Associations on demand

• better flexibility/performance
• no hardcoded associations on specific endpoints
• one serializer/controller per endpoint

Example: GET /api/v1/videos?fields[title]&includes=[user]&fields[user][username]=true
(JSONAPI: GET /api/v1/videos?fields=title,user.username&includes=user)

{
  "video": {
    "id": 3031,
    "title": "Leon på havet!",
    "user": {
      "id": 18,
      "username": "robert",
      "links": {
        "videos":"/api/v1/users/17/videos"
      }
    }
    "links":{
      "user": "/api/v1/users/17"
    }
  }
}

4. Defaults (help the client!)

• go through your application and investigate what an average client would need
• If you know that clients requesting a resource will always need a specific association add it to defaults
• Have different defaults for different levels of permissions
• Admin? Take everything
• User? Take what you need before hand (HTTP is chatty!)
• Unauthenticated user? Take what is barely needed

5. Sorting & pagination

Example: GET /api/v1/videos?sort[username]=desc&per_page=5&page=3&offset=2
(JSONAPI: GET /api/v1/videos?sort=-username&per_page=5&page=3&offset=2)

Pagination should support:

• number of resources per page
• page number
• offset

6. Filtering collections

• Much more flexibility for the client
• Should also support 'OR' query
• one serializer/controller per endpoint

Example: GET /api/v1/videos?draft=true
(JSONAPI: GET /api/v1/videos?field[draft]=true)

Example: GET /api/v1/comments?post[group_id]=1 //ask comments that belong to a post that belongs to group_id=1
(JSONAPI: GET /api/v1/comments?post[group_id]=1)

Example: GET /api/v1/videos?draft=true&published=true
(JSONAPI: GET /api/v1/videos?field[draft,published]=true)

7. Aggregation queries

• more flexibility to the client
• no new endpoints just for some numbers (use of meta instead)

Example: GET /api/v1/videos?state='published'&aggregate[view_count][sum]=true&aggregate[view_count&[avg]=true&per_page=1

this.store.query('video', {
  published: true,
  aggregate: {
    view_count: {
      sum: true,
      avg: true
    }
  },
  per_page:1 //optimization!
});

The URL might seem a bit complicated but params is the easiest thing to create in a URL using any language.

In JS using ember-data:

8. Consistent (flat) URLs

A video:
GET   /api/v1/videos/:video_id
A post:
GET   /api/v1/posts/:post_id
A group:
GET   /api/v1/groups/:group_id
A Comment:
GET   /api/v1/comments/:comment_id

Let's add comments to a video
GET   /api/v1/videos/:video_id/comments

Let's add comments to a post:
GET   /api/v1/post/:post_id/comments

Let's add comment to a.. comment! (of a video!)
GET   /api/v1/videos/:video_id/comments/:comment_id/comments
Doesn't look promising!

Flat design

(unless you follow a spec like JSONAPI, Sirien, Hal which would be great!) 

8. Consistent (flat) URLs

Let's ask for comments of a post that belongs to a group:
GET   /api/v1/videos/:group_id/posts/:post_id/comments

Let's ask for comments of a comment of a post that belongs to a group:
GET   /api/v1/videos/:group_id/posts/:post_id/comments/:comment_id/comments

You see where is going..... and this is only for 1 resource! For 4 resources
it's going to be hell!

Why not just:
GET   /api/v1/comments?video_id=1
GET   /api/v1/comments?post_id=1
GET   /api/v1/comments?video_id=1&comment_id=3
GET   /api/v1/comments?group_id=1&comment_id=3
GET   /api/v1/comments?group_id=1&post_id=3&comment_id=3
POST  /api/v1/comments //all comment association data are in the body of the POST request

Of course some resources can't be flat:​ followers

(followers of what/whom?!)

8. Consistent (flat) URLs

GET   /api/v1/comments?post_id[]=1&video_id[]=3
GET   /api/v1/comments?user_id[]=1&user_id[]=3

Nested urls: You can't request a resource for multiple parent resources.
For instance, let's say that I want all comments of 2 videos/posts etc.

That's impossible with the nesting pattern. Instead:

Remember that constructing params of a url in the client is much much easier
than constructing the actual URL. After all you want comments, that is, the
resource is the same. Why not having a conistent url for that?

Are all these even possible?

Yes

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

1, 2 ,3 and 4

• Pundit anyone?

• Pundit has black white policy
• We want something in between
• Maybe a client is allowed to access a resource but not all parts of it
• It's still 2xx but with chopped data

1, 2 ,3 and 4

module AuthorizeWithReturn
  def authorize_with_permissions(record, query=nil)
    query ||= params[:action].to_s + '?'
    @_pundit_policy_authorized = true

    policy = policy(record)
    policy.public_send(query)
  end

  def included
    super
    hide_action :authorize
  end
end

module Pundit
  prepend AuthorizeWithReturn
end

(with a small monkey patch)

• Pundit anyone?

It doesn't raise an error but returns what policy object returns

1, 2 ,3 and 4

module FlexiblePermissions
  module RoleMethods
    attr_reader :record, :model

    def initialize(record, model = nil)
      @record = record
      @model = model || record.class
    end

    def fields(asked = nil)
      self.class::Fields.new(asked, record, model).resolve
    end

    def includes(asked = nil)
      self.class::Includes.new(asked, record, model).resolve
    end

    def collection
      record
    end
  end
end

module FlexiblePermissions
  module SparsedMethods
    attr_reader :resolve, :model, :record, :asked
    def initialize(asked, record, model)
      @model = model
      @asked = asked
      @record = record
    end

    def resolve
      return defaults if asked.blank?

      union(permitted, asked)
    end

    def permitted
      []
    end

    def defaults
      permitted
    end

    def union(permitted, asked = nil)
      return permitted unless asked.is_a?(Array)

      permitted.map(&:to_sym) & asked.map(&:to_sym)
    end

    def collection?
      record.kind_of? ActiveRecord::Relation
    end
  end
end

module FlexiblePermissions
  module SparsedFieldMethods
    include SparsedMethods

    def permitted
      model.attribute_names.map(&:to_sym)
    end
  end

  module SparsedIncludeMethods
    include SparsedMethods

    def permitted
      model.reflect_on_all_associations.map(&:name).map(&:to_sym)
    end
  end
end

1, 2 ,3 and 4

class UserPolicy < ApplicationPolicy
  class Admin < DefaultPermissions
    class Fields < self::Fields
      def permitted
        super + [
          :links, :following_state, :follower_state, :notification_count
        ]
      end

      def defaults
        super - [:notification_count]
      end
    end

    class Includes < self::Includes
      def permitted
        []
      end
    end
  end
end

class ApplicationPolicy
  class DefaultPermissions
    include FlexiblePermissions::RoleMethods

    class Fields
      include FlexiblePermissions::SparsedFieldMethods
    end

    class Includes
      include FlexiblePermissions::SparsedIncludeMethods
    end
  end
end

Admin is allowed all model fields + extra computed properties

1, 2 ,3 and 4

class UserPolicy < ApplicationPolicy
  class Owner < Admin
    class Fields < self::Fields
      def permitted
        super - [:updated_at, :suspended_at, :reference_id]
      end

      def defaults
        if collection?
          super - [
            :user_id, :background, :date_of_birth, :description,
            :email_confirmed, :followers_count, :followings_count, :gender,
            :mobile, :private, :suspended_at, :terms_accepted, :url,
            :videos_count, :email, :country_code
          ]
        else
          permitted - [:notification_count, :preferences]
        end
      end
    end

    class Includes < self::Includes
      def permitted
        []
      end
    end
  end

  class Regular < Owner
    class Fields < self::Fields
      def permitted
        super - [:notification_count, :preferences, :email, :email_confirmed,
                 :mobile, :suspended_at, :terms_accepted, :role, :created_at]
      end
    end
  end
end

class UserPolicy < ApplicationPolicy
  def create?
    return Admin.new(record) if user && user.admin?
    return Regular.new(record)
  end

  def show?
    raise Pundit::NotAuthorizedError unless user

    return Admin.new(record) if user.admin?
    return Owner.new(record) if user.id == record.id
    return Regular.new(record)
  end

  def update?
    raise Pundit::NotAuthorizedError unless user

    return Admin.new(record) if user.admin?
    return Owner.new(record) if user.id == record.id
    raise Pundit::NotAuthorizedError
  end

  def destroy?
    return Admin.new(record) if user.admin?
    return Owner.new(record)
  end
end

we don't raise an error but return the actual permissions

(unless you don't have the permissions --> 403)

Embrace inheritance to inherit permissions and defaults!

1, 2 ,3 and 4

class Api::V1::UsersController < Api::V1::BaseController
  def show
    auth_user = authorize_with_permissions(@user, :show?)

    render json: auth_user.record, serializer: UserSerializer::Resource,
      fields: auth_user.fields(params[:fields]),
      include: auth_user.includes(params[:include])
  end
end

class UserSerializer < BaseSerializer
  def notification_count 
    ...
  end
.
.
.
  class Collection < self
    attributes(*User.attribute_names.map(&:to_sym))

    attribute :notification_count
    attribute :following_state
    attribute :follower_state
    attribute :links
  end

  class Resource < Collection
  end
end

Embrace inheritance :)

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

class Api::V1::CommentsController < Api::V1::BaseController
  def index
    comments = Comment.all
    comments = comments.scope_by(params[:scope]) if params[:scope]

    [:id, :video_id, :user_id, :post_id, :tags].each do |prm|
      comments = comments.where(prm => params[prm]) unless params[prm].blank?
    end

    unless params[:published_at_start].blank?
      comments = comments.where('PUBLISHED_AT >= ?', params[:published_at_start])
    end

    unless params[:published_at_end].blank?
      comments = comments.where('PUBLISHED_AT <= ?', params[:published_at_end])
    end

    unless params[:text].blank?
      pdsas = pdsas.where('TEXT ILIKE ?', "%#{params[:text]}%")
    end

    comments = policy_scope(comments)

    render json: comments, each_serializer: Api::V1::CommentSerializer
  end
end

5, 6 and 7

A typical controller index....

5, 6 and 7

Don't re-invent the wheel: active_hash_relation

Runs AR queries from defined in a hash

apply_filters(Video.all, {
  created_at: {geq: 2016-01-24}, user_id: 17, views_count: {geq: 1000}, published: true,
  tags: ['foo', 'bar'], description: { like: { starts_with: 'tennis' } }, limit: 30,
  sort: {property: :published_at, order: :desc}
}

SELECT  "videos".* FROM "videos"
  WHERE (videos.title IN ('foo', 'bar'))
    AND (videos.user_id = 17)
    AND (videos.created_at >= 1991)
    AND (videos.views_count >= 1000)
    AND (videos.tags IN ('foo', 'bar'))
    AND (videos.user_id = 17)
    AND (videos.created_at >= 1991)
    AND (videos.views_count >= 1000)
    AND (videos.description LIKE 'tennis%')
  ORDER BY "videos"."published_at" DESC LIMIT 30

this.get('store').find('video', {
  created_at: {
    geq: '2016-01-24'
  },
  user_id: 17,
  views_count: {
    geq: 1000
  },
  published: true,
  tags: ['foo', 'bar'],
  description: {
    like: { starts_with: 'tennis' }
  },
  limit: 30,
  sort: { property: 'published_at', order: 'desc }
});

This is SUPER important for the front-end team. It facilitates them to construct any UI they want without asking you a new API endpoint

5, 6 and 7

class Api::V1::CommentsFilters < Api::V1::BaseFilter
  def collection
    self.context = apply_filters(context, filter_permitted_params)
    paginate(self.context)
  end

private
  def filter_permitted_params
    params[:sort] = {property: :created_at, order: :desc} if params[:sort].blank?
    params.permit(:id, :video_id, :text, published_at: [:le, :ge, :leq, :geq],
      sort: [:property, :order])
  end
end

Let me introduce you to Filter classes

They help controllers to lose weight

There is also mongoid_hash_query for Mongoid

Both support aggregation queries

(min, max, sum, average)

5, 6 and 7

class Api::V1::CommentsController < Api::V1::BaseController
  before_action :load_resource

  def index
    @comments = policy_scope(@comments)

    render json: @comments, each_serializer: CommentSerializer::Collection,
      meta: meta_attributes(@comments)
  end
.
.
.
  def load_resource
    case params[:action].to_sym
    when :index
      @comments = Api::V1::CommentsFilters.new(Comment.all, params).collection
    when :create
      @comment = Comment.new(permitted_params)
      @comment.user_id = current_user.id if @comment.user_id.nil?
    else
      @comment = Comment.find(params[:id])
    end
  end
end

Super clean controller

note the default user_id there

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

8. Consistent URLs

Easy in Rails

• Just don't nest too much
• Prefer flat urls
• You need as less as possible urls
• Embrace url params :)

1. Sparse fields
2. Granular permissions
3. Associations on demand
4. Defaults (help the client!)
5. Sorting & pagination
6. Filtering collections
7. Aggregation queries
8. Consistent (flat) URLs

Modern APIs

Resource

+

Collection

Collection

Design

}

}

-

Topics

1. Modern APIs
2. How to test efficiently
3. Performance!

API tests can be such a waste of time...

New API endpoint process:

1. Create the model
2. Add factories and model tests
3. Add route
4. Add API controller
5. Add serializer
6. Add API (controller) tests
7. Add Pundit policies
8. Add more (authorization) tests

API tests can be such a waste of time...

describe Api::V1::UsersController, type: :api do
  context :index do
    before do
      create_and_sign_in_user
      5.times{ FactoryGirl.create(:user) }

      get api_v1_users_path, format: :json
    end
    it 'returns the correct status' do
      expect(last_response.status).to eql(200)
    end
    it 'returns the correct number of data in the body' do
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:users].length).to eql(5)
    end
  end
end

describe Api::V1::UsersController, type: :api do
  context :create do
    before do
      create_and_sign_in_user

      @user = FactoryGirl.attributes_for(:user)
      post api_v1_users_path, county: @user.as_json, format: :json
    end

    it 'returns the correct status' do
      expect(last_response.status).to eql(201)
    end

    it 'returns the data in the body' do
      user = User.last!
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:user][:name]).to eql(@user.name)
      expect(body[:user][:updated_at]).to eql(user.updated_at.iso8601) #@user var has nil updated_at
    end
  end
end

API tests can be such a waste of time...

describe Api::V1::UsersController, type: :api do
  context :update do
    before do
      create_and_sign_in_user
      @user = FactoryGirl.create(:user)
      @user.name = 'Another name'
      put api_v1_user_path(@user.id), county: @user.as_json, format: :json
    end

    it 'returns the correct status' do
      expect(last_response.status).to eql(200)
    end

    it 'returns the correct location' do
      expect(last_response.headers['Location'])
        .to include(api_v1_county_path(@user.id))
    end

    it 'returns the data in the body' do
      user = User.last!
      body = HashWithIndifferentAccess.new(MultiJson.load(last_response.body))
      expect(body[:user][:name]).to eql(@user.name)
      expect(body[:user][:updated_at]).to eql(user.updated_at.iso8601)
    end
  end
end

API tests can be such a waste of time...

describe Api::V1::UsersController, type: :api do
  context :destroy do
    context 'when the resource does NOT exist' do
      before do
        create_and_sign_in_user
        @user = FactoryGirl.create(:user)
        delete api_v1_user_path(rand(100..1000)), format: :json
      end

      it 'returns the correct status' do
        expect(last_response.status).to eql(404)
      end
    end

    context 'when the resource does exist' do
      before do
        create_and_sign_in_user
        @user = FactoryGirl.create(:user)

        delete api_v1_user_path(@user.id), format: :json
      end

      it 'returns the correct status' do
        expect(last_response.status).to eql(204)
      end

      it 'actually deletes the resource' do
        expect(User.find_by(id: @user.id)).to eql(nil)
      end
    end
  end
end

API tests can be such a waste of time...

We tested:

• the path input -> controller -> model -> controller -> serializer -> output actually works ok
• controller returns the correct error statuses
• controller responds to the API attributes.

Basically we re-implement the RSpecs methods respond_to and rspec-rails' be_valid methods at a higher level.

In 110 lines of code..

What if I change my serializer and use HAL? Oh crap...

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://videofy.me/user#",
  "type": "object",
  "properties": {
    "user": {
      "id": "user",
      "type": "object",
      "properties": {
        "id": {
          "id": "id",
          "type": "integer"
        },
        "username": {
          "id": "username",
          "type": "string"
        },
        "name": {
          "id": "name",
          "type": "string"
        },
        "gender": {
          "id": "gender",
          "type": { "enum": ["male", "female", "null"] }
        }
      },
      "additionalProperties": false,
      "required": [
        "id",
        "username",
        "name",
        "gender"
      ]
    }
  },
  "additionalProperties": false,
  "required": [
    "user"
  ]
}

JSON Scemas!

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://videofy.me/like#",
  "type": "object",
  "properties": {
    "like": {
      "id": "like",
      "type": "object",
      "properties": {
        "id": {
          "id": "id",
          "type": "integer"
        },
        "video_id": {
          "id": "video_id",
          "type": "integer"
        },
        "user": {
          "$ref": "http://videofy.me/user#/properties/user",
          "additionalProperties": false
        }
      },
      "additionalProperties": false,
      "required": [
        "video_id",
        "user",
        "links"
      ]
    }
  },
  "additionalProperties": false,
  "required": [
    "like"
  ]
}

JSON Scemas!

JSON Scemas!

RSpec::Matchers.define :match_response_schema do |expected|
  match do |actual|
    parse_and_validate(expected, actual)[0]
  end

  failure_message do |actual|
    parse_and_validate(expected, actual)[1].map{|e|
      str = "Error in #{e.path.join('/')} #{e.type}: #{e.message}"
      if e.sub_errors
        e.sub_errors[1].each do |sub_e|
          str += "\n"
          str += "Description #{sub_e.schema.try(:data).try(:[],"properties").try(:[],"type").try(:[],"id")}\n"
          str += "\t\t Error in #{sub_e.path.join('/')} #{sub_e.type}: #{sub_e.message}"
        end
      end

      str
    }.join("\n")
  end

  def parse_and_validate(schema, json)
    schema_directory = "#{Rails.root}/spec/schemas"
    schema_path = "#{schema_directory}/#{schema.gsub('.json','')}.json"

    schema = JsonSchema.parse!(JSON.parse(File.read(schema_path)))
    schema.expand_references!(:store => $document_store)
    #binding.pry

    return schema.validate(
      json.is_a?(String)? JSON.parse(json) : json
    )
  end
end

Using json-schema gem and applying some matchers :)

JSON Scemas!

Using json-schema gem and applying some matchers :)

JSON Scemas!

describe 'Likes API: :create', type: :api do
  context 'with guest user permissions' do
    before do
      video = FactoryGirl.create(:video, state: :published)

      post api_v1_video_like_path(video_id: video.id)
    end

    it_returns_status(401)
  end

  context 'with regular user permissions' do
    context 'when user has already liked' do
      before do
        user = create_and_sign_in_user
        video = FactoryGirl.create(:video, state: 'published')
        like = FactoryGirl.create(:like, user: user, video: video)

        post api_v1_video_like_path(video_id: like.video.id)
      end

      it_returns_status(422)
    end

    context 'with correct params' do
      before do
        create_and_sign_in_user
          video = FactoryGirl.create(:video)

        post api_v1_video_like_path(video_id: video.id)
      end

      it_returns_status(201)

      it_follows_json_schema('strict_default/regular/like/show')
    end
  end
end

We also use rspec-api_helpers to test http status and actual values!

JSON Scemas!

require 'rails_helper'

describe 'Videos API:', type: :api do
  context 'with guest user permissions' do
    before do
      get api_v1_videos_path, format: :json
    end

    it_returns_status(401)
  end

  context 'with regular user permissions' do
    before do
      create_and_sign_in_user
      3.times{ FactoryGirl.create(:video, title: nil) } #drafts

      @videos = 5.times{ FactoryGirl.create(:video, state: 'published') }

      get api_v1_videos_path, format: :json
    end

    it_returns_status(200)

    it_returns_collection_size(resource: 'videos', size: 5)

    it_follows_json_schema('strict_default/regular/video/index')
  end
end

We also use rspec-api_helpers to test http status and actual values!

Topics

1. Modern APIs
2. How to test efficiently
3. Performance!

Performance!

Be ready for huge concenssions when it comes to performance

1. Improve Pagination
2. Background jobs and batch updates with caching
3. HTTP Caching
4. (Smart) low level caching
5. JWT tokens
6. Duplicate data in Postgres
7. Screw JSONAPI or any (current) spec and implement hypermedia in a more efficient way

Performance!

• Pagination is a huge bottleneck
• Counting the resulted objects based on the filters takes time
• Remove it by default (who needs number of pages anyway?)
• But have it there just in case someone needs it..

​

We ditched Kaminari for that and used our own postgres pagination

1. Improve pagination

Embrace infinite scrolling :)

Performance!

Use workers whenever possible. For instance:​

2. Background Jobs and batch updates with caching

• Find the most popular/heavy route in your API (/api/v1/videos/:id/like)

• Use JSON Schemas to reflect model validations without touching the model

• Return success as soon as it's valid and add it to the queue.

• Client should update it's own store after a 2xx

• Related: HTTP Prefer header (and rack-prefer)

You can take it one step further and batch like updates in Redis before adding them to the queue. Then every 1 hour do a batch update in db in a worker.

Batch your AR update callbacks too

Performance!

(given that your client implements it correctly)

3. HTTP Caching

class Api::V1::UsersController < Api::V1::BaseController
  before_action :load_resource

  def index
    auth_users = policy_scope(@users)

    with_cache auth_users.collection do
      render json: auth_users.collection,
        each_serializer: UserSerializer::Collection,
        fields: auth_users.fields(params[:fields]),
        include: auth_users.includes(params[:include]),
        meta: meta_attributes(auth_users.collection)
    end
  end

  def show
    auth_user = authorize_with_permissions(@user, :show?)

    with_cache auth_user.record do
      render json: auth_user.record, serializer: UserSerializer::Resource,
        fields: auth_user.fields(params[:fields]),
        include: auth_user.includes(params[:include])
    end
  end
end

Performance!

3. HTTP Caching

module HttpCaching
  extend ActiveSupport::Concern

  included do
    def with_cache(resource, last_modified_at: :updated_at)
      return yield if Rails.application.secrets.disable_http_caching

      case resource
      when ActiveRecord::Relation
        return yield if stale?(
          versioned_cache_headers(resource.maximum(last_modified_at))
        )
      when Mongoid::Criteria
        return yield if stale?(
          versioned_cache_headers(resource.max(last_modified_at))
        )
      else
        return yield if stale?(
          versioned_cache_headers(resource.send(last_modified_at))
        )
      end
    end

    def versioned_cache_headers(value)
      default_caching_options = {public: true, template: false}
      {
        etag: value.to_s + Rails.application.secrets.etag_version.to_s,
        last_modified: value || DateTime.now + Rails.application.secrets.etag_version.to_i
      }.merge(default_caching_options)
    end
  end
end

Remove caching instantly if needed

Invalidate cache instantly if needed

Performance!

4. (Smart) low level caching

class Following < ActiveRecord::Base
  include ActiveCash
  include FollowingScopes

  caches :existence, find_by: [:user_id, :following_id],
    update_on: [], returns: :state, as: :existence_state
.
.
.
end

ActiveCash

class Metafield < ActiveRecord::Base
  include IdentityCache
  belongs_to :owner, :polymorphic => true
  cache_belongs_to :owner
end

class Product < ActiveRecord::Base
  include IdentityCache
  has_many :metafields, :as => 'owner'
  cache_has_many :metafields, :inverse_name => :owner
end

IdentityCache

class Like < ActiveRecord::Base
  include ActiveCash

  caches :existence, find_by: [:user_id, :video_id]
.
.
.
end

Performance!

4. (Smart) low level caching

ActiveCash

Performance!

5. JWT (and oauth2)

• How do you do authentication/authorization in a world of microservices?
• A client might need to send 3 requests in 3 different services
• Authenticating each request takes unnecessary time (especially if authentication is made through HTTP to another service)
• Save in the token all the necessary client information using JWTs

hmac_secret = 'my$ecretK3y'

payload = {user_id: 17, name: 'Filippos Vasilakis', username: 'vasilakisfil'}

token = JWT.encode payload, hmac_secret, 'HS256'

# eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0ZXN0IjoiZGF0YSJ9._sLPAGP-IXgho8BkMGQ86N2mah7vDyn0L5hOR4UkfoI
puts token

decoded_token = JWT.decode token, hmac_secret, true, { :algorithm => 'HS256' }

# Array
# [
#   {"user_id"=>17, "name" => "Filippos Vasilakis", "username" => "vasilakisfil"}, # payload
#   {"typ"=>"JWT", "alg"=>"HS256"} # header
# ]

Performance!

6. Duplicate data in Postgres

• On some API endpoints you know that client always need nested resources

{
  "video": {
    "id": 3031,
    "title": "Leon på havet!",
    "description": " #bohuslän",
    "state": "published",
    "file": "https://d2tyds7k2z0x7k.cloudfront.net/4bc54b39-a869-42ad-9d2e-d7c0ed49f3d3.mp4",
    "thumbnail": "https://d2rgaput4dpbfs.cloudfront.net/f8b7a960-698c-ca05-d913-f77f832bf118_custom.jpg",
    "views_count": 52,
    "likes_count": 1,
    "comments_count": 0,
    "created_at": "2015-08-05T15:56:05.000+02:00",
    "user_id": 18,
    "music_track_id": 13,
    "links": {
      "self": "/api/v1/videos/3031",
      "comments": "/api/v1/videos/3031/comments",
      "user": "/api/v1/users/18"
    },
    "likes": [{
      "user_id": 18,
      "video_id": 3031,
      "user": {
        "id": 18,
        "username": "robert",
        "name": "Robert M",
        "avatar": "https://dxxs0s19ll2ye.cloudfront.net/4c657cfe-116d-4f3f-694d-9164e5294e72.jpg"
      },
      "links":{
        "video": "/api/v1/videos/3031",
        "user": "/api/v1/users/18"
      }
    }],
    "requester_has_liked": false //cached using ActiveCash
  }
}

We know that 99.9% of video requests, some video likes are also needed by the UI

The iOS UI actually needs the avatars of your friends who liked the video

The Web UI needs avatars and usernames/names

This kills the db!

Performance!

6. Duplicate data in Postgres

  create_table "likes", force: :cascade do |t|
    t.integer  "video_id",   null: false
    t.integer  "user_id",    null: false
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
  end

A like in the database holds only references

• What if we add some user data in each like?
• user name, username and avatar
• in a JSONB column

  create_table "likes", force: :cascade do |t|
    t.integer  "video_id",   null: false
    t.integer  "user_id",    null: false
    t.datetime "created_at", null: false
    t.datetime "updated_at", null: false
    t.jsonb    "info",      default: {},    null: false
  end

irb(main):010:0> Like.first.as_json
{
  "id"=>1, "video_id"=>24454, "user_id"=>257902, "created_at"=>Tue, 19 Jan 2016 05:05:58 UTC +00:00,
  "updated_at"=>Tue, 19 Jan 2016 05:05:58 UTC +00:00}, "info" => {
    "user" => {"username" => "vasilakisfil", "name" => "Filippos Vasilakis", "avatar" =>
    "https://dxxs0s19ll2ye.cloudfront.net/4c657cfe-116d-4f3f-694d-9164e5294e72.jpg"
  }
}

Update every 1 hour in batch updates
or update on every change of the association

(user basic info rarely change)

Performance!

6. Duplicate data in Postgres

It takes some time to update all rows of a large table...

Performance!

Be ready for huge concenssions when it comes to performance

1. Improve Pagination
2. Background jobs and batch updates with caching
3. HTTP Caching
4. (Smart) low level caching
5. JWT tokens
6. Duplicate data in Postgres
7. Screw JSONAPI or any (current) spec and implement hypermedia in a more efficient way

Performance!

Have I talked about any API spec?

JSONAPI, HAL, Sirien etc..

(ok I gave some examples on JSONAPI)

{
  "video": {
    "id": 3031,
    "title": "Leon på havet!",
    "description": " #bohuslän",
    "state": "published",
    "file": "https://d2tyds7k2z0x7k.cloudfront.net/4bc54b39-a869-42ad-9d2e-d7c0ed49f3d3.mp4",
    "thumbnail": "https://d2rgaput4dpbfs.cloudfront.net/f8b7a960-698c-ca05-d913-f77f832bf118_custom.jpg",
    "links": {
      "self": "/api/v1/videos/3031",
      "comments": "/api/v1/videos/3031/comments",
      "user": "/api/v1/users/18"
    },
    "likes": [{
      "user_id": 18,
      "video_id": 3031,
      "user": {
        "id": 18,
        "username": "robert",
        "name": "Robert M",
        "avatar": "https://dxxs0s19ll2ye.cloudfront.net/4c657cfe-116d-4f3f-694d-9164e5294e72.jpg"
      },
      "links":{
        "video": "/api/v1/videos/3031",
        "user": "/api/v1/users/18"
      }
    }],
    "requester_has_liked": false //cached using ActiveCash
  }
}

Do we need all those links?

Has anyone

changed a link for a

SINGLE

unique

only

one

resource?!

Performance!

• Ruby is a bit slow when serializing objects even with the best gem (oj)
• Imagine serializing a collection of 50 resources with 10 embedded resources each (likes)
• Depending on the number of attributes API returns and the number of links for each resource, payload could increase even a lot!
• 99.999% links per object are useless!
• Use OPTIONS request to determine the hypermedia of a resource and representation using JSON SCHEMA
• Use base API url to define all possible resources in the API and their hypermedia

Topics

1. Modern APIs
2. How to test efficiently
3. Performance!

Thanks!

Questions ?

Useful links

Blog posts:

https://labs.kollegorna.se/blog/2014/11/rails-api/

https://labs.kollegorna.se/blog/2015/01/ember-overview/

https://labs.kollegorna.se/blog/2015/02/active-hash-relation/

https://labs.kollegorna.se/blog/2015/04/build-an-api-now/

Gems:

https://github.com/elabs/pundit

https://github.com/rails-api/active_model_serializers

https://github.com/kollegorna/active_hash_relation

https://github.com/kollegorna/mongoid_hash_query

https://github.com/kollegorna/rspec-api_helpers

https://github.com/kollegorna/rack-prefer

https://github.com/vasilakisfil/api_bomb

https://github.com/vasilakisfil/active_cash