Going Meta with Elixir

Running at Compile-time

& (Re)Compiling at Runtime

By: Wiebe-Marten Wijnja/Qqwy

Metaprogramming:

Automating creation & changing of a program

To 'Go Meta'?

Talk Structure & Goal

  • What & How to use
  • Why useful?
  • Examples

 

 

Inspiration

Code: Educational "fits on slide" examples

Links to resources with more detail

Running at Compile-time

 

 

 

 

 

(Re)compiling at Runtime

 

Running at Compile-time

Dynamic Compilation

Running at compile-time

Dynamic Compilation

Running at compile-time

defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Dynamic Compilation

Running at compile-time

defmodule Example do
  if function_exported?(:erlang, :is_map_key, 2) do
    def my_fancy_function(param1, param2) do
      # ...
    end
  else
    raise(CompileError, 
      description: "This library requires an Erlang/OTP version that has the `is_map_key` function!"
      )
  end
end

Dynamic Compilation

Running at compile-time

defmodule Example do
  if function_exported?(:erlang, :is_map_key, 2) do
    def my_fancy_function(param1, param2) do
      # ...
    end
  else
    raise(CompileError, 
      description: "This library requires an Erlang/OTP version that has the `is_map_key` function!"
      )
  end
end
$ iex -S mix
Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 1 file (.ex)

== Compilation error in file lib/compile_error_example.ex ==
** (CompileError)  This library requires an Erlang/OTP version that has the `is_map_key` function!
    lib/compile_error_example.ex:8: (module)
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6

Dynamic Compilation

Running at compile-time

defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Dynamic Compilation

Running at compile-time

defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  @known_answers [0, 1, 1, 2, 3, 5, 8, 13, 25]

  for {answer, index} <- Enum.with_index(@known_answers) do
    def fib(unquote(index)) do
      unquote(answer)
    end
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Dynamic Compilation

Running at compile-time

defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do 0 end
  def fib(1) do 1 end
  def fib(2) do 1 end
  def fib(3) do 2 end
  def fib(4) do 5 end
  def fib(5) do 8 end
  def fib(6) do 13 end
  def fib(7) do 25 end
  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

defmodule String.Unicode.Example do
  @moduledoc """
  Paraphrased from Elixir's unicode-handling modules.
  (See https://github.com/elixir-lang/elixir/tree/master/lib/elixir/unicode)
  """

  ranges =
    Path.join(__DIR__, "UnicodeData.txt")
    |> File.read!()
    |> String.split("\n", trim: true)
    |> Enum.map(&parse_unicode_range/1)

  def unicode_upper?(char)
  for {first, last} <- ranges, uppercase_range?(range) do
    def unicode_upper?(char) when char in unquote(range) do
      true
    end
  end
  def unicode_upper?(char) do
    false
  end

  # ...
end

Ex: Unicode Support

Running at compile-time

Macros

Running at compile-time

  • Functions that run at compile-time
  • AST Transformations: Code -> Code
  • Hygienic (by default): No name clashes

Aside: Abstract Syntax Trees

1 - 3 * 5 + 2

Aside: Abstract Syntax Trees

1 - 3 * 5 + 2
iex> quote do 1 - 3 * 5 + 2 end

{:+, [],
 [
   {:-, [], [1, {:*, [], [3, 5]}]},
   2
 ]}

Ex: ExUnit's assert

Running at compile-time

assert(1 + 2 + 3 > 10)
Comparison with > failed
lhs: 6
rhs: 10
code: 1 + 2 + 3 > 10

Ex: ExUnit's assert

Running at compile-time

assert(1 + 2 + 3 > 10)
Comparison with > failed
lhs: 6
rhs: 10
code: 1 + 2 + 3 > 10
assert_gt(left, right)
assert_lt(left, right)
assert_eq(left, right)
# ...

# Horrible!

Running at compile-time

defmodule AssertExample do
  defmacro assert(expr = {:>, _meta, [lhs, rhs]}) do
    quote do
      left = unquote(lhs)
      right = unquote(rhs)
      if left > right do
        true
      else
        IO.puts """
        Comparison with > failed
        left: #{left}
        right: #{right}
        code: #{unquote(Code.to_string(expr))}
        """
        false
      end
    end
  end
  defmacro assert(expr = {:<, _meta, [lhs, rhs]}), do: #...
  defmacro assert(expr = {:>=, _meta, [lhs, rhs]}), do: #...
  defmacro assert(expr = {:<=, _meta, [lhs, rhs]}), do: #...
  defmacro assert(expr = {:!=, _meta, [lhs, rhs]}), do: #...

  # ...
  
  defmacro assert(other) do
    raise(Assert.CompileError, "Unsupported assert expression #{Macro.to_string(expr)}")
  end
end

assert(1 + 2 + 3 > 10)

Ex: ExUnit's assert

Ex: Ecto Queries

Running at compile-time

from p in Post,
  where: p.category == "fresh and new",
  order_by: [desc: p.published_at],
  select: struct(p, [:id, :title, :body])

Ex: Phoenix HTTP Routing

defmodule AppRouter do
  use HelloWeb, :router
  
  get "/", PageController, :index
  get "/users", UsersController, :index
  get "/users/:id", UsersController, :show
  get "/users/:id/edit", UsersController, :edit
  patch "/users/:id", UsersController, :update
end
  • Safe interpolation of values
  • Compile-time errors on malformed inputs
  • Composition
  • Efficient dispatch
  • Autogenerated helper functions

Tools

Running at compile-time

  • Dynamic Compilation
    • Reduce repetitive code
    • Conditional compilation
    • Definitions based on external sources
    • Compile-time Safety
  • Macros
    • Domain Specific Languages
      • Syntactic Sugar
    • Code Transformations
    • Compile-time Safety

(Re)Compiling at runtime

Elixir & Erlang

Hot-Code Reloading

(Re)Compiling at runtime

Hot-Code Reloading

(Re)Compiling at runtime

  • Deploy new version of app
  • while it is running
  • without dropping any data, sockets etc!

Hot-Code Reloading

(Re)Compiling at runtime


iex> user = MyRepo.get(User, 42)
%User{name: "W-M", id: 42}
iex> MyModule.do_something_with(user)
** (ArgumentError) some problem

# change the module

# now, reload the module
iex> r MyModule
iex> MyModule.do_something_with(user)
{:ok, "Something!"}

(Re)Compiling at runtime

  • Improved efficiency for write-seldom, read-often data
  • "Efficiency hack": (Ab)uses module constant pool

Ex: FastGlobal & :persistent_term

Ex: FastGlobal & :persistent_term

(Re)Compiling at runtime

  • Efficiency for write-seldom, read-often data
defmodule ExampleGlobal do
  def get(key, default \\ nil) when is_atom(key) do
    if function_exported?(module_name(key), :__val__, 0) do
      module_name(key).__val__()
    else
      default
    end
  end

  def put(key, val) when is_atom(key) do
    Code.eval_quoted(
      quote bind_quoted: [module_name: module_name(key), val: val] do
        :code.purge(module_name)
        :code.delete(module_name)
        defmodule module_name do
          def __val__, do: unquote(val)
        end
    end)
    :ok
  end

  defp module_name(key), do: :"ExampleGlobal.#{key}"
end

(Re)Compiling at runtime

  • Don't use it unless you need to (try Agents, ETS, etc. first)
    • Benchmark!
    • Use `:persistent_term.get/put`

Ex: FastGlobal & :persistent_term

Ex: TZData

(Re)Compiling at runtime

Ex: TZData

(Re)Compiling at runtime

  • Checks remote 'timezone database' once a day
  • If there are changes, recompile*

*The current version uses ETS. A version based on :persistent_term is in the works:

https://github.com/lau/tzdata/issues/69

Ex: Introspection

(Re)Compiling at runtime

iex>
defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Ex: Introspection

(Re)Compiling at runtime

iex> Rexbug.start("Fibonacci.fib(1)")
:ok
defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    send_trace_information()
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Ex: Introspection

(Re)Compiling at runtime

iex> Rexbug.start("Fibonacci.fib(1)")
:ok
iex> Fibonacci.fib(4)
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)
         
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)
         
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)
 
defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    send_trace_information()
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Ex: Introspection

(Re)Compiling at runtime

iex> Rexbug.start("Fibonacci.fib(1)")
:ok
iex> Fibonacci.fib(4)
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)
         
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)
         
# 23:28:41 #PID<0.204.0> IEx.Evaluator.init/4
# Fibonacci.fib(1)

iex> Rexbug.stop
:stopped
redbug done, local_done - 5
defmodule Fibonacci do
  @doc """
  Returns the n-th fibonacci number.
  Slow implementation for educational purposes.
  """

  def fib(0) do
    0
  end

  def fib(1) do
    1
  end

  def fib(n) when n > 0 do
    fib(n - 1) + fib(n - 2)
  end

end

Ex: Introspection

(Re)Compiling at runtime

  • :erlang.trace, Redbug, Rexbug
  • Runs in your live app
  • Patch introspection calls into functions matching pattern only!
    • No slowdown for rest of app

Tools

(Re)Compiling at Runtime

  • Recompilation at Runtime
    • Hot code deployments
    • React to things that change independently from your app release schedule
    • Persisting terms for fast global read access.
    • Introspection

Running at Compile-time

 

 

 

 

 

(Re)compiling at Runtime

 

With Great Power...

Resources & Further Reading

Book: 'Metaprogramming Elixir' by Chris McCord https://pragprog.com/book/cmelixir/metaprogramming-elixir

Thank you!

 

 

 

 

 

Questions?

 

Going Meta with Elixir's

By qqwy

Made with Slides.com

Going Meta with Elixir's

  • 704

More from qqwy

  • TypeCheck - Effortless Type-Checking in Elixir
    qqwy
    508
  • Parser Combinators in C++
    qqwy
    818
  • Multiplayer Games & Collaborative Editing with Phoenix LiveView
    qqwy
    1215
  • Copy of Web and Cloud Computing: Group 13
    qqwy
    701