Building the X Search API

By @richardkotze

Team Oblivion

About

  • Team Oblivian approach
  • Challenges
  • Tools we used 

** Things might have changed

Built with

Elixir 🔗

Phoenix Framework 🔗

Vagrant 🔗

Elixir getting started

  • Running Elixir on windows is fine
    • iex --werl
  • Experience better on Linux
  • Introduction to Vagrant
  • Useful links to learn
    • Roman numeral kata 🔗
    • Portal tutorial 🔗
    • Elixir getting started 🔗
    • Elixir docs 🔗

Mix Elixir

  • Get basic project file layout
  • Unit testing (ExUnit)
mix new awesome_project --module AwesomeProject

Mix is a build tool, run tasks and tests

mix compile
// after compile
iex -S mix

  • Create an iex session (Elixir’s interactive shell)

  • Learn more about mix 🔗

Elixir

  • Head | tail recursion
  • Pattern matching
  • Everything is immutable

Thinking functionally

  • Basics easy to get
  • Dynamically typed language
  • Modules and named function
  • Piping |>
  • :Atoms (name is their value)

Language and syntax

Using Vagrant

Main benefits

  • Developing on Windows side (editor and browsers)
  • Disposable environment
  • Fairly easy to configure
  • Everyone has the same environment, close to live
  • Easy access to a Linux
vagrant up
vagrant ssh

Phoenix 

  • Web framework
  • MVC
  • Core structure for our API
    • Routing
    • Controllers
  • Comes with loads of other stuff
    • Channels - send and receive messages
    • Ecto - writing queries and interacting with databases
    • Templates and views

Phoenix docs 🔗

mix archive.install https://github.com/phoenixframework/archives/raw/master/phoenix_new.ez

Phoenix routing

pipeline :search do
  plug :accepts, ["json"]
  plug XSearchApi.CORS # plug: connection lifecycle
end

scope "/", XSearchApi do # group similar routes together
  pipe_through :search

  get "/category/count", SearchController, :category_search_count

end

# eg scope "/admin" ...
  • Simple
  • Scope for Search API
  • Pipeline used to transform to JSON
  • Pipeline are simple plugs
  • Plug for CORS

Phoenix controller

defmodule XSearchApi.SearchController do
  use XSearchApi.Web, :controller

  def category_search_count(conn, params) do
    valid_api_params = get_valid_api_params(params)
    safe_result_link = get_safe_result_link(conn, params)
    get_response(conn, valid_api_params, safe_result_link)
  end

  defp get_response(conn, {:ok, valid_api_params}, {:ok, safe_result_link}) do
   {_status, response} = XSearchApi.SAPI.get(
                        "records/record", [], [params: valid_api_params])
 
    conn
    |> put_status(response.status_code)
    |> json(handle_response(response, safe_result_link))
  end
...
  • Easy to read
  • Public action 'def'
  • 'defp' private function

HTTPoison

  • Http client library
  • Use it to call SAPI
  • 'use HTTPoison.base'
  • use a module in the current context
defmodule XSearchApi.SAPI do
  use HTTPoison.Base

  def get(_url, [], [{:params, params}]) when is_nil(params), do: 
    {:error, %XSearchApi.Error { reason: "Bad request", status_code: 400 }}

  def get(url, [], [{:params, params}]) do
    {status, response} = request(:get, url, "", [], [{:params, params}])
    set_response(status, response)
  end
...

Tough challenges

  • Build simple odata parser
  • Use odata to build a link to search result page
@year_fields ["YearOfBirth", "YearOfDeath"]
@year_offset "_offset"

def append_year_offset([[year_field, :eq, value]|tail]) 
when year_field in @year_fields do

  exact_year_phrase = [year_field, :eq, value]
  Enum.concat(
    [exact_year_phrase, :and, ["#{year_field}#{@year_offset}", :eq, "0"]], 
    append_year_offset(tail))
end

Cool piece of code :D

Deployment

  • Team city - run mix commands
  • Flightplan
  • Upstart part of Ubuntu
    • Handle tasks and services
    • Stop and start anywhere

more on upstart 🔗

Hex package

  • Package manager 🔗
  • Build an Elixir lib and publish to hex 🔗
  • Published under a user
# Register
mix hex.user register

# Publish package
mix hex.publish

ExUnit & Mocks

  • ExUnit package for unit testing
  • Mock package
defmodule XSearchApi.SearchControllerTest do
  use ExUnit.Case
  use Phoenix.ConnTest
  import Mock

  test "should pass odata query to sapi ..." 
  do
    with_mock XSearchApi.SAPI, 
      [get: fn() ->  {:ok, %XSearchApi.Response{ ... } end]
    do
      conn_response = SearchController
            .category_search_count(
                conn, %{"$filter" => "LastName eq Kotze"}
            )

      assert called SAPI.get("/record", [params: %{"$filter" => "LastName eq 'Kotze'"}])
      assert conn_response.status == 200
    end
  end
...

White_bread

  • BDD tool for running acceptance tests
  • Based on cucumber
  • Parses Gherkin formatted text 

more on white_bread 🔗

given_ ~r/^I am a third party consumer$/, fn state ->
  {:ok, state}
end

when_ ~r/I make a request to the API .../, fn state, %{first_name: first_name} ->
  encoded_json = get(
   put_req_header(conn(), "partnerid", "0123"), 
   "/category/count?$filter=FirstName eq #{first_name}")

  {:ok, state |> Dict.put(:response, encoded_json)}
end

then_ ~r/I expect the response to be a link .../, fn state, %{link_first_name: first_name} ->
  response = state |> Dict.get(:response)
  decode_body = json_response(response, 200)
  decoded_uri = decode_body["ResultUrl"]
  assert decoded_uri == "#{@search_link_base}FirstName=#{first_name}"
  {:ok, state}
end

eye_drops

  • more on eye_drops 🔗
  • It's a watch task
  • Run commands based on file changes
  • e.g. Run unit tests when '_test.exs' files change
  • e.g. Run acceptance tests when '.feature' file changes
config :eye_drops, 
  tasks: [
    %{
      id: :unit_tests,
      name: "Unit tests",
      cmd: "ssh -F .ssh-config default mix test",
      paths: ["web/*", "test/*"]
    }
  ]

Vagrant tip

Questions?

Made with Slides.com