ZeroMQ for Clojurists

Dave Yarwood • @dave_yarwood

Clojure Remote 2017

Agenda

• What is ZeroMQ?
• ZeroMQ vs. core.async
• Libraries
• ezzmq
• Building stuff

screenshot of zeromq.org

screenshot of zguide.zeromq.org

ZeroMQ vs. core.async

clojure.core.async

• Channels are local to a Clojure program
• Useful for unidirectional task pipelines
• A message can be any non-nil Clojure value

ZeroMQ vs. core.async

ZeroMQ

• Can send messages between programs
• A socket can exist outside of your program
• Specialized socket types for high-level patterns
• Can send messages bidirectionally
• Messages are byte arrays

ZMTP

ZeroMQ Message Transport Protocol

libzmq

low-level C/C++ API

czmq

high-level C library wrapping libzmq

source: github.com/zeromq/czmq

source: github.com/zeromq/czmq

jzmq

Java language binding for libzmq

JeroMQ

pure Java implementation of libzmq

(+ some nice higher level abstractions ported from czmq)

github.com/zeromq/jeromq

ezzmq

idiomatic Clojure wrapper around JeroMQ

github.com/daveyarwood/ezzmq

import org.zeromq.ZContext;
import org.zeromq.ZMQ;

public static void main(String[] args) {
    ZContext context = new ZContext();

    ZMQ.Socket server = context.createSocket(ZMQ.REP);
    server.bind("tcp://*:12345");

    while (true) {
        byte[] request = server.recv(0);
        System.out.println("Received msg: " + new String(request));

        // simulate doing work
        Thread.sleep(1000);
        String response = "42";

        server.send(response.getBytes(), 0);
    }

    context.close();
}

jzmq / JeroMQ

(require '[ezzmq.core :as zmq])

(zmq/with-new-context
  (let [socket (zmq/socket :rep {:bind "tcp://*:12345"})]
    (while true
      (let [req (zmq/receive-msg socket :stringify true)
            res "42"]
        (println "Received msg:" req)

        ;; simulate doing work
        (Thread/sleep 1000)

        (zmq/send-msg socket res)))))

ezzmq

import org.zeromq.ZContext;
import org.zeromq.ZMQ;

public static void main (String[] args) {
    ZContext context = new ZContext();

    ZMQ.Socket pull = context.createSocket(ZMQ.PULL);
    pull.connect("tcp://*:12345");

    ZMQ.Socket sub = context.createSocket(ZMQ.SUB);
    sub.connect("tcp://*:12346");
    sub.subscribe("abc ".getBytes());

    ZMQ.Poller poller = new ZMQ.Poller(2);
    poller.register(pull, ZMQ.Poller.POLLIN);
    poller.register(sub, ZMQ.Poller.POLLIN);

    while (!Thread.currentThread().isInterrupted()) {
        byte[] msg;
        poller.poll();
        if (poller.pollin(0)) {
            msg = pull.recv(0);
            System.out.println("PULL: " + new String(msg));
        }
        if (poller.pollin(1)) {
            msg = sub.recv(0);
            System.out.println("SUB: " + new String(msg));
        }
    }

    context.close();
}

jzmq / JeroMQ

(require '[ezzmq.core :as zmq])

(zmq/with-new-context
  (let [pull (zmq/socket :pull {:connect "tcp://*:12345"})
        sub  (zmq/socket :sub  {:connect "tcp://*:12346"
                                :subscribe "abc "})]
    (zmq/polling {:stringify true}
      [pull :pollin [msg]
       (println (format "PULL: %s" msg))

       sub :pollin [msg]
       (println (format "SUB: %s" msg))]

      (zmq/while-polling
        (zmq/poll)))))

ezzmq

Socket Types: REQ/REP

source: zguide.zeromq.org

(let [client (zmq/socket :req {:connect "tcp://*:12345"})
      _      (zmq/send-msg client "oh, hello")
      reply  (zmq/receive-msg client :stringify true)]
  (println "the server says:" reply))

Socket Types: PUB/SUB

source: zguide.zeromq.org

Socket Types: PUB/SUB

(defn publisher
  (let [pub (zmq/socket :pub {:bind "tcp://*:12345"})]
    (while true
      (let [msg (format "%s %s"
                        (rand-nth ["abc" "def" "ghi"])
                        "this is a message")]
        (zmq/send-msg pub msg)))))

(defn subscriber-1
  (let [sub (zmq/socket :sub {:connect "tcp://*:12345" :subscribe "abc "})]
    (while true
      (let [msg (zmq/receive-msg sub)]
        (println "got msg:" msg)))))

(defn subscriber-2
  (let [sub (zmq/socket :sub {:connect "tcp://*:12345" :subscribe "def "})]
    (while true
      (let [msg (zmq/receive-msg sub)]
        (println "got msg:" msg)))))

(defn subscriber-3
  (let [sub (zmq/socket :sub {:connect "tcp://*:12345" :subscribe "ghi "})]
    (while true
      (let [msg (zmq/receive-msg sub)]
        (println "got msg:" msg)))))

Socket Types: PUSH/PULL

source: zguide.zeromq.org

Socket Types: PUSH/PULL

(defn ventilator []
  (let [vent (zmq/socket :push {:bind "tcp://*:3333"})
        sink (zmq/socket :push {:connect "tcp://*:4444"})]
    ;; wait for workers to come online
    (Thread/sleep 30000)

    (zmq/send-msg sink "START BATCH")

    (dotimes [_ 100]
      (zmq/send-msg vent (str (rand-int 100))))))

(defn worker []
  (let [vent (zmq/socket :pull {:connect "tcp://*:3333"})
        sink (zmq/socket :push {:connect "tcp://*:4444"})]
    (while true
      (let [task-ms (-> (zmq/receive-msg vent :stringify true)
                        first
                        Integer/parseInt)]
        (Thread/sleep task-ms) ; simulate doing work
        (zmq/send-msg sink "success")))))

(defn sink []
  (let [sink (zmq/socket :pull {:bind "tcp://*:4444"})]
    ;; wait for start of batch
    (zmq/receive-msg sink)

    (dotimes [_ 100]
      (let [result (zmq/receive-msg sink :stringify true)]
        (println "Result:" result)))))

Socket Types: ROUTER/DEALER

source: zguide.zeromq.org

Socket Types: PAIR

source: zguide.zeromq.org

Socket Types: PAIR

(defn step-1 []
  (let [step2 (zmq/socket :pair {:connect "inproc://step2"})]
    (zmq/send-msg step2 "the eagle has landed")))

(defn step-2 []
  (let [step2 (zmq/socket :pair {:bind "inproc://step2"})
        msg   (zmq/receive-msg step2)
        step3 (zmq/socket :pair {:connect "inproc://step3"})]
    (zmq/send-msg step3 msg)))

(defn step-3 []
  (let [step3 (zmq/socket :pair {:bind "inproc://step3"})
        msg   (zmq/receive-msg step3)]
    (println msg)))

Building a Chat Server/Client

Building a Chat Server/Client

Building a Chat Server/Client

{"command": "join",
 "from":    "dave"}

{"command": "say",
 "from":    "dave",
 "body":    "oh, hello"}

{"command": "leave",
 "from":    "dave"}

Building a Chat Server/Client

the server

Building a Chat Server/Client

the client

Building a Chat Server/Client

opportunities for improvement

• client: don't interleave input/output
• client: handle failure response from server
• client: handle lack of response from server
• server: heartbeating
• feature: chat log w/ datetimes
• feature: login / authentication

Questions?