ABOUT ME

• WORKING IN RST FOR 3 YEARS

• WORKING AS A DEVELOPER FOR 8 YEARS

• LOVES TO SHARE KNOWLEDGE AND DISCUSS

THE PROBLEM

• NEED TO STORE AND SEARCH 12 BLN DOCUMENTS

• 2-3MLN MESSAGES EACH DAY

• 200+ MESSAGES/SEC DURING THE PEAK

• MESSAGES CONTAIN IMPORTANT DATA

• MESSAGES MUST BE SAVED

INITIAL STATE

THE OBSTACLES

• MESSENGER CURRENTLY USES CUSTOM PROTOCOL

• MESSENGER MIGRATES TO XMPP IN NEAR FUTURE

• NEED TO MIGRATE WHOLE DATA

• USERS MUST HAVE CONTINOUS ACCESS TO ARCHIVE

• DURING MIGRATION WE NEED TO SUPPORT BOTH DATABASES

OLD SOLUTION

• MYSQL DATABASE

• SIMPLE JOIN QUERIES FOR SEARCH (NON-FULLTEXT)

• PROCEDURAL SHARDING TO HARDLIMITED INSTANCES

• ALMOST REACHED MAXIMUM STORAGE CAPABILITY (LIMITED BY HARDWARE)

THE NEW APPROACH GOALS

• SUPPORT 10K WRITE REQs per SECOND

• SERVICE INDEPENDEND ARCHITECTURE

• EASILY TO SCALE HORIZONTALLY

• REST API FOR SEARCHING AND BROWSING

• USE OLD CLIENT DURING MIGRATION

THE DATABASE

TECHNOLOGY

• ELASTICSEARCH AS A DATABASE (YES, REALLY)

• NODE.JS FOR PROCESSING SERVICES

• PHP (ZEND's APIGILITY) AS A REST SERVICE

• REDIS FOR CACHE, RABBITMQ FOR QUEUES

ELASTICSEARCH AS A DATABASE

• IS CONSIDERED JUST AS A SEARCH ENGINE

• EASILY SCALED VERTICALLY AND HORIZONTALLY

• ES2.4+ SOLVES MOST BACKUPING PROBLEMS

• DOING WELL IN AUTO-HEALING

• MONITORING TOOLS (MARVEL, KOPF, HQ ETC.)

ELASTICSEARCH AS A DATABASE: CONS

• IT WAS DESIGNED AS A SEARCH ENGINE

• MORE SHARDS/REPLICAS INCREASE WRITE TIME

• MANY KNOBS TO TWEAK

• 2.4+ HAS PROBLEMS WITH INDEX REBUILD

• UPDATE IS ALMOST IMPOSSIBLE

ARCHITECTURE: DATA MIGRATION

ARCHITECTURE: READ THE ARCHIVE

ARCHITECTURE: WRITE TO ARCHIVE

ARCHITECTURE: CHAOS RESISTANT

Message

XMPP Server

Client's APP

Nginx

CONSUMER

DAEMON

Elasticsearch Cluster

Redis 1

Ext Services

Message Queue

Invalid MSG Queue

PROBLEMS & SOLUTIONS

• SERVICE INDEPENdece REQUIRES LOOOT MORE SPACE

  • ​Hard drivers are cheap ;)

• ES WORKS WELL BY DEFAULT ONLY AS A SEARCH ENGINE

  • ​STILL WORKING ON MAKING CLUSTER STABLE

• BACKUPS ARE HARD IF YOU CANNOT CLOSE INDEX

THINGS TO REMEMBER

• ES IS GREAT AS A READ-ONLY DATABASE

• POC IS IMPORTANT PART OF DEVELOPMENT

• ... BUT THE PRODUCTION MAKES THE VERDICT

• + ALWAYS REMEMBER ABOUT CHAOS MONKEY

THINGS TO REMEMBER