IPT TEAM REVIEW
PI 2017.3
NEW ARCHIVE PROGRESS
NEW ARCHIVE PROGRESS
WE SOLVED ELASTICSEARCH PROBLEMS
THE DATABASE
cassandra
MYSQL / MYROCKS + SPHINX
ELASTICSEARCH
TECHNOLOGY
-
ELASTICSEARCH AS A DATABASE (YES, REALLY)
-
NODE.JS FOR PROCESSING SERVICES
-
PHP (ZEND's APIGILITY) AS A REST SERVICE
-
REDIS FOR CACHE, RABBITMQ FOR QUEUES
ELASTICSEARCH AS A DATABASE
-
IS CONSIDERED JUST AS A SEARCH ENGINE
-
EASILY SCALED VERTICALLY AND HORIZONTALLY
-
ES2.4+ SOLVES MOST BACKUPING PROBLEMS
-
DOING WELL IN AUTO-HEALING
-
MONITORING TOOLS (MARVEL, KOPF, HQ ETC.)
ELASTICSEARCH AS A DATABASE: CONS
-
IT WAS DESIGNED AS A SEARCH ENGINE
-
MORE SHARDS/REPLICAS INCREASE WRITE TIME
-
MANY KNOBS TO TWEAK
-
2.4+ HAS PROBLEMS WITH INDEX REBUILD
-
UPDATE IS ALMOST IMPOSSIBLE
ARCHITECTURE
ARCHITECTURE: DATA MIGRATION
Message
Communication Server
Client's APP
Database
...
QUEUE
DAEMON
CONSUMER
DAEMON
Elasticsearch Cluster
Redis 0
Redis 1
Ext Services
ARCHITECTURE: READ THE ARCHIVE
User
Messages from January to Feburary
Client's APP
REST API
archive_2017_01
Mobile APP
archive_2017_02
archive_2016_12
ARCHITECTURE: WRITE TO ARCHIVE
Message
XMPP Server
Client's APP
Nginx
CONSUMER
DAEMON
Elasticsearch Cluster
Redis 1
Ext Services
Message Queue
Invalid MSG Queue
ARCHITECTURE: CHAOS RESISTANT
Message
XMPP Server
Client's APP
Nginx
CONSUMER
DAEMON
Elasticsearch Cluster
Redis 1
Ext Services
Message Queue
Invalid MSG Queue
PROBLEMS & SOLUTIONS
PROBLEMS & SOLUTIONS
-
SERVICE INDEPENdece REQUIRES LOOOT MORE SPACE
-
Hard drivers are cheap ;)
-
-
ES WORKS WELL BY DEFAULT ONLY AS A SEARCH ENGINE
-
STILL WORKING ON MAKING CLUSTER STABLE
-
-
BACKUPS ARE HARD IF YOU CANNOT CLOSE INDEX
PROBLEMS & SOLUTIONS
HEAP PROBLEMS FOR WIDE SCALE BULK OPERATIONS
NODE.JS IS FASTER THAN ES
THINGS TO REMEMBER
-
ES IS GREAT AS A READ-ONLY DATABASE
-
ALWAYS REMEMBER ABOUT CHAOS MONKEY
-
POC IS IMPORTANT PART OF DEVELOPMENT
-
... BUT THE PRODUCTION MAKES THE VERDICT
THINGS TO REMEMBER
EACH FAILURE LEADS TO BETTER EXPERIENCE
QuESTIONS TIME!