NEW ARCHIVE PROGRESS

WE SOLVED ELASTICSEARCH PROBLEMS

THE DATABASE

TECHNOLOGY

• ELASTICSEARCH AS A DATABASE (YES, REALLY)

• NODE.JS FOR PROCESSING SERVICES

• PHP (ZEND's APIGILITY) AS A REST SERVICE

• REDIS FOR CACHE, RABBITMQ FOR QUEUES

ELASTICSEARCH AS A DATABASE

• IS CONSIDERED JUST AS A SEARCH ENGINE

• EASILY SCALED VERTICALLY AND HORIZONTALLY

• ES2.4+ SOLVES MOST BACKUPING PROBLEMS

• DOING WELL IN AUTO-HEALING

• MONITORING TOOLS (MARVEL, KOPF, HQ ETC.)

ELASTICSEARCH AS A DATABASE: CONS

• IT WAS DESIGNED AS A SEARCH ENGINE

• MORE SHARDS/REPLICAS INCREASE WRITE TIME

• MANY KNOBS TO TWEAK

• 2.4+ HAS PROBLEMS WITH INDEX REBUILD

• UPDATE IS ALMOST IMPOSSIBLE

ARCHITECTURE: DATA MIGRATION

ARCHITECTURE: READ THE ARCHIVE

ARCHITECTURE: WRITE TO ARCHIVE

ARCHITECTURE: CHAOS RESISTANT

Message

XMPP Server

Client's APP

Nginx

CONSUMER

DAEMON

Elasticsearch Cluster

Redis 1

Ext Services

Message Queue

Invalid MSG Queue

PROBLEMS & SOLUTIONS

• SERVICE INDEPENdece REQUIRES LOOOT MORE SPACE

  • ​Hard drivers are cheap ;)

• ES WORKS WELL BY DEFAULT ONLY AS A SEARCH ENGINE

  • ​STILL WORKING ON MAKING CLUSTER STABLE

• BACKUPS ARE HARD IF YOU CANNOT CLOSE INDEX

PROBLEMS & SOLUTIONS

HEAP PROBLEMS FOR WIDE SCALE BULK OPERATIONS

NODE.JS IS FASTER THAN ES 

THINGS TO REMEMBER

• ES IS GREAT AS A READ-ONLY DATABASE

• ALWAYS REMEMBER ABOUT CHAOS MONKEY

• POC IS IMPORTANT PART OF DEVELOPMENT

• ... BUT THE PRODUCTION MAKES THE VERDICT

THINGS TO REMEMBER