CASSANDRA SUMMIT 2015

Mario Lazaro

September 24th 2015

#CassandraSummit2015

Multi-Region Cassandra in AWS

WHOAMI

• Mario Cerdan Lazaro

• Big Data Engineer

• Born and raised in Spain

• Joined GumGum 18 months ago
• About a year and a half experience with Cassandra

• #5 Ad Platform in the U.S

• 10B impressions / month

• 2,000 brand-safe premium publisher partners

• 1B+ global unique visitors

• Daily inventory Impressions processed - 213M

• Monthly Image Impressions processed - 2.6B

• 123 employees in seven offices

Agenda

• Old cluster
• International Expansion
  • Challenges
  • Testing
  • Modus Operandi
  • Tips
  • ​Questions & Answers

•  25 Classic nodes cluster
• 1 Region / 1 Rack hosted in AWS EC2 US East
• Version 2.0.8
• Datastax CQL driver
• GumGum's metadata including visitors, images, pages, and ad performance
• Usage: realtime data access and analytics (MR jobs)

OLD C* CLUSTER - MARCH 2015

OLD C* CLUSTER - REALTIME USE CASE

• Billions of rows
• Heavy read workload 
  • 60/40
• TTLs everywhere - Tombstones
• Heavy and critical use of counters
• RTB - Read Latency constraints (total execution time ~50 ms) 

OLD C* CLUSTER - ANALYTICS USE CASE

• Daily ETL jobs to extract / join data from C*
  • ​Hadoop MR jobs
  • AdHoc queries with Presto

Challenges

• Datastax Java Driver Load Balancing
  • Multiple choices

CHALLENGES

• Zone Aware Connection:
  • Webapps in 3 different AZs: 1A, 1B, and 1C
  • C* datacenter spanning 3 AZs with 3 replicas

1A

1B

1C

1B

CHALLENGES

• We added it! -  Rack/AZ awareness to TokenAware Policy

CHALLENGES

• Third Datacenter: Analytics
  • Do not impact realtime data access
  • Spark on top of Cassandra
  • Spark-Cassandra Datastax connector
  • Replicate specific keyspaces
  • Less nodes with larger disk space
  • Settings are different
    • Ex: Bloom filter chance

CHALLENGES

• Third Datacenter: Analytics

Cassandra Only DC

Realtime

Cassandra + Spark DC

Analytics

TESTING - Performance

• Performance tests using new Cassandra 2.1 Stress Tool:
  • Recreate GumGum metadata / schemas
  • Recreate workload and make it 3 times bigger
  • Try to find limits / Saturate clients

# Keyspace Name
keyspace: stresscql
 
#keyspace_definition: |
#    CREATE KEYSPACE stresscql WITH replication = {'class': #'NetworkTopologyStrategy', 'us-eastus-sandbox':3,'eu-westeu-sandbox':3 }

### Column Distribution Specifications ###

columnspec:
  - name: visitor_id
    size: gaussian(32..32)       #domain names are relatively short
    population: uniform(1..999M)  #10M possible domains to pick from

  - name: bidder_code
    cluster: fixed(5)
  - name: bluekai_category_id
  - name: bidder_custom
    size: fixed(32)
  - name: bidder_id
    size: fixed(32)
  - name: bluekai_id
    size: fixed(32)
  - name: dt_pd
  - name: rt_exp_dt
  - name: rt_opt_out
   
### Batch Ratio Distribution Specifications ###

insert:
  partitions: fixed(1)            # Our partition key is the visitor_id so only insert one per batch
 
  select:    fixed(1)/5        # We have 5 bidder_code per domain so 1/5 will allow 1 bidder_code per batch
 
  batchtype: UNLOGGED             # Unlogged batches
 
 
#
# A list of queries you wish to run against the schema
#
queries:
  getvisitor:
      cql: SELECT bidder_code, bluekai_category_id, bidder_custom, bidder_id, bluekai_id, dt_pd, rt_exp_dt, rt_opt_out FROM partners WHERE visitor_id = ?
      fields: samerow

TESTING - Performance

• Main worry:
  • Latency and replication overseas
    • Use LOCAL_X consistency levels in your client
    • Only one C* node will contact only one C* node in a different DC for sending replicas/mutations

TESTING - Performance

• Main worries:
  • Latency

MODUS OPERANDI

• Third step:
  • Start EU West and new US East DCs within the same cluster
    • Replication factor in new DCs: 0
    • Use dc_suffix to differentiate new Virginia DC from old one
    • Clients do not talk to new DCs. Only C* knows they exist
    • Replication factor to 3 on all except analytics 1 
      • ​​Start receiving new data
    • Nodetool rebuild <old-datacenter>
      • ​Old data

MODUS OPERANDI

RF 3

Clients

MODUS OPERANDI

From 39d8f76d9cae11b4db405f5a002e2a4f6f764b1d Mon Sep 17 00:00:00 2001
From: mario <mario@gumgum.com>
Date: Wed, 17 Jun 2015 14:21:32 -0700
Subject: [PATCH] AT-3576 Start using new Cassandra realtime cluster

---
 src/main/java/com/gumgum/cassandra/Client.java     | 30 ++++------------------
 .../com/gumgum/cassandra/Ec2ClassicTranslater.java | 30 ++++++++++++++--------
 src/main/java/com/gumgum/cluster/Cluster.java      |  3 ++-
 .../resources/applicationContext-cassandra.xml     | 13 ++++------
 src/main/resources/dev.properties                  |  2 +-
 src/main/resources/eu-west-1.prod.properties       |  3 +++
 src/main/resources/prod.properties                 |  3 +--
 src/main/resources/us-east-1.prod.properties       |  3 +++
 .../CassandraAdPerformanceDaoImplTest.java         |  2 --
 .../asset/cassandra/CassandraImageDaoImplTest.java |  2 --
 .../CassandraExactDuplicatesDaoTest.java           |  2 --
 .../com/gumgum/page/CassandraPageDoaImplTest.java  |  2 --
 .../cassandra/CassandraVisitorDaoImplTest.java     |  2 --
 13 files changed, 39 insertions(+), 58 deletions(-)

Start using new Cassandra DCs

MODUS OPERANDI

Clients

US East Realtime

EU West Realtime

US East Analytics

RF 3:3:3:1

MODUS OPERANDI

RF 0:3:3:1

Clients

US East Realtime

EU West Realtime

US East Analytics

tips - spark

• Number of workers above number of total C* nodes in analytics
• Each worker uses: 
  • 1/4 number of cores of each instance
  • 1/3 total available RAM of each instance
• Cassandra-Spark connector
  • ​SpanBy
  • .joinWithCassandraTable(:x, :y)
  • Spark.cassandra.output.batch.size.bytes
  • Spark.cassandra.output.concurrent.writes

TIPS - spark

val conf = new SparkConf()
  .set("spark.cassandra.connection.host", cassandraNodes)
  .set("spark.cassandra.connection.local_dc", "us-east-va-analytics")
  .set("spark.cassandra.connection.factory", "com.gumgum.spark.bluekai.DirectLinkConnectionFactory")
  .set("spark.driver.memory","4g")
  .setAppName("Cassandra presidential candidates app")

• Create "translator" if using EC2MultiRegionSnitch
  • Spark.cassandra.connection.factory