Building the X Search API

By @richardkotze

Team Oblivion

About

• Team Oblivian approach
• Challenges
• Tools we used 

** Things might have changed

Built with

Elixir 🔗

Phoenix Framework 🔗

Vagrant 🔗

Elixir getting started

• Running Elixir on windows is fine
  • iex --werl
• Experience better on Linux
• Introduction to Vagrant
• Useful links to learn
  • Roman numeral kata 🔗
  • Portal tutorial 🔗
  • Elixir getting started 🔗
  • Elixir docs 🔗

Mix Elixir

• Get basic project file layout
• Unit testing (ExUnit)

mix new awesome_project --module AwesomeProject

Mix is a build tool, run tasks and tests

mix compile
// after compile
iex -S mix

• Create an iex session (Elixir’s interactive shell)

• Learn more about mix 🔗

Elixir

• Head | tail recursion
• Pattern matching
• Everything is immutable

Thinking functionally

• Basics easy to get
• Dynamically typed language
• Modules and named function
• Piping |>
• :Atoms (name is their value)

Language and syntax

Using Vagrant

Main benefits

• Developing on Windows side (editor and browsers)
• Disposable environment
• Fairly easy to configure
• Everyone has the same environment, close to live
• Easy access to a Linux

vagrant up
vagrant ssh

Phoenix 

• Web framework
• MVC
• Core structure for our API
  • Routing
  • Controllers
• Comes with loads of other stuff
  • Channels - send and receive messages
  • Ecto - writing queries and interacting with databases
  • Templates and views

Phoenix docs 🔗

mix archive.install https://github.com/phoenixframework/archives/raw/master/phoenix_new.ez

Phoenix routing

pipeline :search do
  plug :accepts, ["json"]
  plug XSearchApi.CORS # plug: connection lifecycle
end

scope "/", XSearchApi do # group similar routes together
  pipe_through :search

  get "/category/count", SearchController, :category_search_count

end

# eg scope "/admin" ...

• Simple
• Scope for Search API
• Pipeline used to transform to JSON
• Pipeline are simple plugs
• Plug for CORS

Phoenix controller

defmodule XSearchApi.SearchController do
  use XSearchApi.Web, :controller

  def category_search_count(conn, params) do
    valid_api_params = get_valid_api_params(params)
    safe_result_link = get_safe_result_link(conn, params)
    get_response(conn, valid_api_params, safe_result_link)
  end

  defp get_response(conn, {:ok, valid_api_params}, {:ok, safe_result_link}) do
   {_status, response} = XSearchApi.SAPI.get(
                        "records/record", [], [params: valid_api_params])
 
    conn
    |> put_status(response.status_code)
    |> json(handle_response(response, safe_result_link))
  end
...

• Easy to read
• Public action 'def'
• 'defp' private function

HTTPoison

• Http client library
• Use it to call SAPI
• 'use HTTPoison.base'
• use a module in the current context

defmodule XSearchApi.SAPI do
  use HTTPoison.Base

  def get(_url, [], [{:params, params}]) when is_nil(params), do: 
    {:error, %XSearchApi.Error { reason: "Bad request", status_code: 400 }}

  def get(url, [], [{:params, params}]) do
    {status, response} = request(:get, url, "", [], [{:params, params}])
    set_response(status, response)
  end
...

Tough challenges

• Build simple odata parser
• Use odata to build a link to search result page

@year_fields ["YearOfBirth", "YearOfDeath"]
@year_offset "_offset"

def append_year_offset([[year_field, :eq, value]|tail]) 
when year_field in @year_fields do

  exact_year_phrase = [year_field, :eq, value]
  Enum.concat(
    [exact_year_phrase, :and, ["#{year_field}#{@year_offset}", :eq, "0"]], 
    append_year_offset(tail))
end

Cool piece of code :D

Deployment

• Team city - run mix commands
• Flightplan
• Upstart part of Ubuntu
  • Handle tasks and services
  • Stop and start anywhere

more on upstart 🔗

Hex package

• Package manager 🔗
• Build an Elixir lib and publish to hex 🔗
• Published under a user

# Register
mix hex.user register

# Publish package
mix hex.publish

ExUnit & Mocks

• ExUnit package for unit testing
• Mock package

defmodule XSearchApi.SearchControllerTest do
  use ExUnit.Case
  use Phoenix.ConnTest
  import Mock

  test "should pass odata query to sapi ..." 
  do
    with_mock XSearchApi.SAPI, 
      [get: fn() ->  {:ok, %XSearchApi.Response{ ... } end]
    do
      conn_response = SearchController
            .category_search_count(
                conn, %{"$filter" => "LastName eq Kotze"}
            )

      assert called SAPI.get("/record", [params: %{"$filter" => "LastName eq 'Kotze'"}])
      assert conn_response.status == 200
    end
  end
...

White_bread

• BDD tool for running acceptance tests
• Based on cucumber
• Parses Gherkin formatted text 

more on white_bread 🔗

given_ ~r/^I am a third party consumer$/, fn state ->
  {:ok, state}
end

when_ ~r/I make a request to the API .../, fn state, %{first_name: first_name} ->
  encoded_json = get(
   put_req_header(conn(), "partnerid", "0123"), 
   "/category/count?$filter=FirstName eq #{first_name}")

  {:ok, state |> Dict.put(:response, encoded_json)}
end

then_ ~r/I expect the response to be a link .../, fn state, %{link_first_name: first_name} ->
  response = state |> Dict.get(:response)
  decode_body = json_response(response, 200)
  decoded_uri = decode_body["ResultUrl"]
  assert decoded_uri == "#{@search_link_base}FirstName=#{first_name}"
  {:ok, state}
end

eye_drops

• more on eye_drops 🔗

• It's a watch task
• Run commands based on file changes
• e.g. Run unit tests when '_test.exs' files change
• e.g. Run acceptance tests when '.feature' file changes

config :eye_drops, 
  tasks: [
    %{
      id: :unit_tests,
      name: "Unit tests",
      cmd: "ssh -F .ssh-config default mix test",
      paths: ["web/*", "test/*"]
    }
  ]

Vagrant tip

Questions?