Consul: Service Discovery & Beyond

24 September 2018

What is Consul?

Service Discovery

Distributed

Key-Value Store

Live Templating

Service Mesh

(coming soon)

Distributed, Scalable Architecture

Architecture

Agents

  • Single binary written in Go
  • Run on every node in the network
  • Can operate as servers or clients
  • Run DNS and HTTP interfaces
  • Responsible for
    • running checks
    • keeping services in sync

Clients

  • Forward RPCs to servers
  • Participate in LAN gossip protocol
  • Consume minimal resources and bandwidth

Servers

  • Operate as Clustered, Distributed Architecture
    • PLOS: 1 cluster of 3 servers (SOMA)
  • Maintain cluster state
  • Respond to queries
  • Exchange WAN gossip (not used at PLOS)
  • Forward queries to leaders or remote DCs

Consensus

  • Agreement upon the Elected Leader
  • Agreement on Ordering of Transactions
  • Consistency of Replicated State Machine
  • Implements Raft Protocol
  • Implementation Notes

Gossip

  • Node-to-Node communication
  • Provides
    • Membership
    • Failure Detection
    • Event Broadcast
  • Implements Serf Protocol
  • Gossip Documentation

Service Discovery

the automatic detection of devices and services offered by these devices on a computer network

Advertising a Service

  • Drop a json file in /etc/consul.d
{
    "service": {
        "checks": [
            {
                "http": "http://solr-210.soma.plos.org:8983/solr",
                "interval": "10s",
                "timeout": "5s"
            }
        ],
        "name": "solr",
        "port": 8983,
        "tags": [
            "dev",
            "cluster=solr-mini"
        ]
    }
}

Advertising a Service

  • Tip: use file.serialize salt state
solr-advertise:
  file.serialize:
    - name: /etc/consul.d/solr.json
    - formatter: json
    - dataset:
        service:
          name: solr
          port: {{ solr_port }}
          checks:
            - http: 'http://{{ hostname }}:8983/solr'
              interval: 10s
              timeout: 5s
          tags:
            - {{ environment }}
            - cluster={{ cluster }}

Service Discovery APIs

  • Command Line
    • 'consul node [...]'
    • 'consul catalog [...]'
    • 'consul service [...]'
  • REST
    • listens on localhost:8500 on every node
    • e.g. 'curl -sS localhost:8500/v1/[...]'
  • DNS
    • listens on localhost:8600 on every node
    • e.g. 'dig @localhost -p 8600 [...]'

Discovering Services: CLI

chaumes@ops-admin-201:~$ consul catalog nodes -service=solr
Node      ID        Address    DC
solr-110  b53e5afd  10.5.3.64  soma
solr-111  5a1dc5a2  10.5.3.68  soma
solr-120  e1058eb7  10.5.2.2   soma
solr-121  b17a2d57  10.5.2.3   soma
solr-122  34074129  10.5.2.4   soma
solr-123  6944d721  10.5.2.5   soma
solr-124  88245dec  10.5.2.6   soma
solr-125  4dca2a93  10.5.2.7   soma
solr-126  9584d894  10.5.2.8   soma
solr-127  de640708  10.5.2.9   soma
solr-210  b94b9aa0  10.5.4.88  soma
solr-211  bbb9c4c9  10.5.4.89  soma
solr-220  2818b2af  10.5.4.2   soma
solr-221  42359f86  10.5.4.3   soma
solr-222  51322276  10.5.4.4   soma
solr-223  3bbabeb3  10.5.4.5   soma
solr-224  9351dc84  10.5.4.6   soma
solr-225  028c8e97  10.5.4.7   soma
solr-226  a74090ac  10.5.4.8   soma
solr-227  e91b0761  10.5.4.9   soma

Discovering Services: HTTP

chaumes@ops-admin-201:~$ curl -sS 'http://localhost:8500/v1/catalog/service/solr?tag=prod' | jq .
[
    {
        "Address": "10.5.3.64",
        "CreateIndex": 6348285,
        "Datacenter": "soma",
        "ID": "b53e5afd-1b9d-8a3a-a012-060a2429c08d",
        "ModifyIndex": 6348285,
        "Node": "solr-110",
        "NodeMeta": {
            "consul-network-segment": ""
        },
        "ServiceAddress": "",
        "ServiceEnableTagOverride": false,
        "ServiceID": "solr",
        "ServiceName": "solr",
        "ServicePort": 8983,
        "ServiceTags": [
            "prod",
            "cluster=solr-mini"
        ],
        "TaggedAddresses": {
            "lan": "10.5.3.64",
            "wan": "10.5.3.64"
        }
    },
    { ... }
]

Discovering Services: DNS

chaumes@ops-admin-201:~$ dig @localhost -p 8600 prod.solr.service.consul

; <<>> DiG 9.9.5-3ubuntu0.17-Ubuntu <<>> @localhost -p 8600 prod.solr.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32640
;; flags: qr aa rd; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;prod.solr.service.consul.	IN	A

;; ANSWER SECTION:
prod.solr.service.consul. 0	IN	A	10.5.2.3
prod.solr.service.consul. 0	IN	A	10.5.3.64
prod.solr.service.consul. 0	IN	A	10.5.2.9
prod.solr.service.consul. 0	IN	A	10.5.2.7
prod.solr.service.consul. 0	IN	A	10.5.2.4
prod.solr.service.consul. 0	IN	A	10.5.2.6
prod.solr.service.consul. 0	IN	A	10.5.2.5
prod.solr.service.consul. 0	IN	A	10.5.3.68
prod.solr.service.consul. 0	IN	A	10.5.2.8
prod.solr.service.consul. 0	IN	A	10.5.2.2

;; Query time: 14 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Fri Sep 21 15:47:31 PDT 2018
;; MSG SIZE  rcvd: 213

Prepared Queries

chaumes@ops-admin-201:~$ cat prep.json
{
  "Name": "solr-mega-dev",
  "Service": {
    "Service": "solr",
    "Tags": ["dev", "cluster=solr-mega"],
  }
}

chaumes@ops-admin-201:~$ curl -X POST -d @prep.json localhost:8500/v1/query
{"ID":"3cab98ae-529a-6f2e-04a5-a88c347c6444"}

chaumes@ops-admin-201:~$ curl -sS localhost:8500/v1/query | jq .[].ID
"3cab98ae-529a-6f2e-04a5-a88c347c6444"
  • Create more complex queries
  • Save queries for reuse
  • Managed via REST API

Prepared Queries

chaumes@ops-admin-201:~$ curl -sS \
> 'http://localhost:8500/v1/query/3cab98ae-529a-6f2e-04a5-a88c347c6444/execute' \
> | jq .Nodes[]?.Node.Node
"solr-226"
"solr-222"
"solr-220"
"solr-221"
"solr-223"
"solr-225"
"solr-224"
"solr-227"

chaumes@ops-admin-201:~$ curl -sS \
> 'http://localhost:8500/v1/query/3cab98ae-529a-6f2e-04a5-a88c347c6444/execute' \
> | jq .Nodes[]?.Node.Address
"10.5.4.5"
"10.5.4.6"
"10.5.4.8"
"10.5.4.3"
"10.5.4.4"
"10.5.4.7"
"10.5.4.9"
"10.5.4.2"

Prepared Queries

chaumes@ops-admin-201:~$ dig @localhost -p 8600 solr-mega-dev.query.consul SRV

; <<>> DiG 9.9.5-3ubuntu0.17-Ubuntu <<>> @localhost -p 8600 solr-mega-dev.query.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22472
;; flags: qr aa rd; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 17
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;solr-mega-dev.query.consul.	IN	SRV

;; ANSWER SECTION:
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-220.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-221.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-224.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-226.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-227.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-223.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-222.node.soma.consul.
solr-mega-dev.query.consul. 0	IN	SRV	1 1 8983 solr-225.node.soma.consul.

Distributed

Key-Value Store

Useful For

  • Dynamic Configuration
  • Service Coordination
  • Leader Election
  • Much more!

Key-Value APIs

  • Consul CLI
  • REST/HTTP

Key-Value CLI

chaumes@ops-admin-201:~$ consul kv put foo/bar baz
Success! Data written to: foo/bar
chaumes@ops-admin-201:~$ consul kv put foo/xyz 123
Success! Data written to: foo/xyz
chaumes@ops-admin-201:~$ consul kv put bar/foo bat
Success! Data written to: bar/foo
chaumes@ops-admin-201:~$ consul kv get -recurse
bar/foo:bat
foo/bar:baz
foo/xyz:123
chaumes@ops-admin-201:~$ consul kv get bar/foo
bat
chaumes@ops-admin-201:~$ consul kv get -recurse foo
foo/bar:baz
foo/xyz:123
chaumes@ops-admin-201:~$ consul kv delete bar/foo
Success! Deleted key: bar/foo
chaumes@ops-admin-201:~$ consul kv delete -recurse foo
Success! Deleted keys with prefix: foo
chaumes@ops-admin-201:~$ consul kv get
chaumes@ops-admin-201:~$ consul kv get -recurse
chaumes@ops-admin-201:~$

Key-Value REST

chaumes@ops-admin-201:~$ curl -X PUT localhost:8500/v1/kv/foo/bar -d '{"value": "bat"}'
true

chaumes@ops-admin-201:~$ curl -sS localhost:8500/v1/kv/foo/bar | jq .
[
  {
    "LockIndex": 0,
    "Key": "foo/bar",
    "Flags": 0,
    "Value": "eyJ2YWx1ZSI6ICJiYXQifQ==",
    "CreateIndex": 8434456,
    "ModifyIndex": 8434456
  }
]

chaumes@ops-admin-201:~$ curl -sS localhost:8500/v1/kv/foo/bar | jq -r .[].Value | base64 -d
{"value": "bat"}

chaumes@ops-admin-201:~$ curl -X DELETE localhost:8500/v1/kv/foo/bar
true

chaumes@ops-admin-201:~$ curl -sS localhost:8500/v1/kv/foo/bar | jq .
chaumes@ops-admin-201:~$

Live Templating

  • Populate variables in a file from Consul data
    • Service Discovery Catalog
    • K/V Store
    • Vault
  • Automatically restart service when values change
  • Coming to PLOS in Q4 2018

Service Mesh

  • Secure Service-to-Service Communication
  • Automatic TLS Encryption
    • Automated certificate management and rotation
  • Identity-Based Authorization via Intentions
  • No Need to Change Existing Code
    • Connect via Proxy Sidecars
  • Highly-Available & Distributed Architecture
  • Beta Feature in Consul 1.2
  • Coming to PLOS in Q1/Q2 2019

Access Controls

  • Capability-Based Security
  • Tokens are Tied to Capabilities
  • Every Token Has:
    • Identifier
    • Name
    • Type
    • Ruleset
  • Managed via API
  • Ops is Currently Developing Best Practices & Docs

Implementation Timeline

  • Service Discovery: Implemented
  • K/V Store: Implemented
    • Best Practices Guide and ACL Advice coming Q4 2018
  • Live Templating: Coming Q4 2018
  • Service Mesh: Coming Q1 2019

consul service discovery

By wryfi

Made with Slides.com

consul service discovery

  • 127

More from wryfi

  • Prometheus Instrumentation
    wryfi
    208