(ClickHouse at Segment)
Alan Braithwaite
@caust1c
A Modern Database
with
Modern Orchestration
What is Segment?
Collect
Standardize
Synthesize
Dispatch
Roadmap
Background &
Motivation
Clickhouse &
Kubernetes
Performance &
Observability
Background & Motivation
Once upon a time...
Hack Week 2018
- Legacy Aggregation System
- Redis and Go based
- Complex and Inflexible to change
- High performance
- Easy to operate
- Cost equal or less than Redis solution
Legacy Counters on Redis
MySQL (Aurora)
Google BigQuery
Snowflake
Spark
Flink
Druid
Legacy Counters on Redis
MySQL (Aurora)
Google BigQuery
Snowflake
Spark
Flink
Druid
Clickhouse & Kubernetes
Amazon's Elastic Container Service
If Kubernetes had an annoying younger sibling, ECS would be it
Used i3 nodes with local NVME disks
Started 1 node for POC
Lasted longer than we wanted it to
Struggled to get Distributed Config working
EKS Ready for use internally — April 19
New i3en Instances Announced — May 8
Design inspired by LogHouse
https://github.com/flant/loghouse
github.com/yandex/ClickHouse/issues/5287
Initial Report — May 15
Root Cause Identified — May 20
Fix Implemented — June 5
Shipped in release — June 24
9 * i3en.3xlarge nodes
7.5TB NVME SSDs, 96GB Memory, 12 vCPUs
Clickhouse has full node usage
Kubernetes mounts NVME disk inside container
Cross Replication using default databases trick
Set env vars for macros in configmap script directory
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: counters
spec:
selector:
matchLabels:
app: clickhouse
updateStrategy:
type: OnDelete
podManagementPolicy: Parallel
serviceName: "clickhouse"
replicas: 9
template:
metadata:
labels:
app: clickhouse
spec:
terminationGracePeriodSeconds: 10
nodeSelector:
segment.com/pool: clickhouse
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- clickhouse
topologyKey: "kubernetes.io/hostname"
containers:
- name: clickhouse
image: "<segment-private-docker-repo>/clickhouse-server:eff47da"
volumeMounts:
- name: datadir
mountPath: /var/lib/clickhouse
- name: configdir
mountPath: /etc/clickhouse-server/config.d
- name: clickhouse-mon
image: "<segment-private-docker-repo>/clickhouse-mon:587ea6b"
volumes:
- name: datadir
hostPath:
path: /data
type: Directory
- name: configdir
configMap:
name: clickhouse-config
FROM yandex/clickhouse-server:19.9.2.4
RUN apt-get update && apt-get install -y iptables ca-certificates
COPY start-clickhouse.sh /usr/local/bin/start-clickhouse
ENTRYPOINT /usr/local/bin/start-clickhouse
#!/usr/bin/env bash -xe
HOST=`hostname -s`
if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
NAME=${BASH_REMATCH[1]}
ORD=${BASH_REMATCH[2]}
else
echo "Failed to parse name and ordinal of Pod"
exit 1
fi
# set the environment before we exec
. /etc/clickhouse-server/config.d/$ORD.sh
exec /entrypoint.sh
Dockerfile
start-clickhouse.sh
<yandex>
<macros replace="replace">
<r0shard from_env="CH_SHARD_R0"/>
<r0replica from_env="CH_REPLICA_R0"/>
<r1shard from_env="CH_SHARD_R1"/>
<r1replica from_env="CH_REPLICA_R1"/>
<cluster>counters_cluster</cluster>
</macros>
</yandex>
macros.xml
$ ls config.d
0.sh 1.sh 2.sh
3.sh 4.sh 5.sh
6.sh 7.sh 8.sh
$ cat config.d/1.sh
export CH_SHARD_R0=01
export CH_REPLICA_R0=00
export CH_SHARD_R1=02
export CH_REPLICA_R1=01
Cross Replication
Cross Replication
ENGINE = Distributed(
'counters_cluster',
-- Database name: use the default database of whichever node is connected to.
-- This will either be r0 or r1, depending on the (shard, node) combination.
-- This mapping is configured in `locals.xml`.
'',
-- Table name: the table name within r0/r1 that will be used to perform the query.
global_events_with_properties,
-- Sharding key: randomly choose a shard to write to for every insertion.
rand()
);
<remote_servers>
<counters_cluster>
<shard>
<internal_replication>true</internal_replication>
<replica>
<default_database>r0</default_database>
<host>counters-0.clickhouse.default.svc.cluster.local</host>
<port>9000</port>
</replica>
<replica>
<default_database>r1</default_database>
<host>counters-1.clickhouse.default.svc.cluster.local</host>
<port>9000</port>
</replica>
</shard>
<!-- ... -->
</counters_cluster>
<remote_servers>
Performance & Observability
Insertion Performance
Hitting wall when inserting 100k batches into distributed table (over raw data)
Batches became ~11k each (100k / 9 nodes)
Active Part Count Climbed
Due to async write nature, no backpressure
Solution: insert directly into shards
Drawback: client doesn't try second replica for shard
If there's a failure of a node,
that shard will stop getting data
Merges were now able to keep up
Active Part count went down
Future
Convert more existing systems to Clickhouse
Better Observability features for Segment Customers
Open Source
Kubernetes Chart &
"clickhouse-mon"
Alan Braithwaite
@caust1c
Thanks
Segment Clickhouse
By Alan Braithwaite
Segment Clickhouse
- 4,266