IPT TEAM REVIEW

PI 2017.3

NEW ARCHIVE PROGRESS

NEW ARCHIVE PROGRESS

WE SOLVED ELASTICSEARCH PROBLEMS

 

THE DATABASE

cassandra

MYSQL / MYROCKS + SPHINX

ELASTICSEARCH

TECHNOLOGY

  • ELASTICSEARCH AS A DATABASE (YES, REALLY)

  • NODE.JS FOR PROCESSING SERVICES

  • PHP (ZEND's APIGILITY) AS A REST SERVICE

  • REDIS FOR CACHE, RABBITMQ FOR QUEUES

ELASTICSEARCH AS A DATABASE

  • IS CONSIDERED JUST AS A SEARCH ENGINE

  • EASILY SCALED VERTICALLY AND HORIZONTALLY

  • ES2.4+ SOLVES MOST BACKUPING PROBLEMS

  • DOING WELL IN AUTO-HEALING

  • MONITORING TOOLS (MARVEL, KOPF, HQ ETC.)

ELASTICSEARCH AS A DATABASE: CONS

  • IT WAS DESIGNED AS A SEARCH ENGINE

  • MORE SHARDS/REPLICAS INCREASE WRITE TIME

  • MANY KNOBS TO TWEAK

  • 2.4+ HAS PROBLEMS WITH INDEX REBUILD

  • UPDATE IS ALMOST IMPOSSIBLE

ARCHITECTURE

ARCHITECTURE: DATA MIGRATION

Message

Communication Server

Client's APP

Database

...

QUEUE

DAEMON

 

CONSUMER

DAEMON

 

Elasticsearch Cluster

Redis 0

Redis 1

Ext Services

ARCHITECTURE: READ THE ARCHIVE

User

Messages from January to Feburary

Client's APP

REST API

archive_2017_01

Mobile APP

archive_2017_02

archive_2016_12

ARCHITECTURE: WRITE TO ARCHIVE

Message

XMPP Server

Client's APP

Nginx

 

CONSUMER

DAEMON

 

Elasticsearch Cluster

Redis 1

Ext Services

Message Queue

 

Invalid MSG Queue

 

ARCHITECTURE: CHAOS RESISTANT

Message

XMPP Server

Client's APP

Nginx

 

CONSUMER

DAEMON

 

Elasticsearch Cluster

Redis 1

Ext Services

Message Queue

 

Invalid MSG Queue

 

PROBLEMS & SOLUTIONS

PROBLEMS & SOLUTIONS

  • SERVICE INDEPENdece REQUIRES LOOOT MORE SPACE

    • ​Hard drivers are cheap ;)

  • ES WORKS WELL BY DEFAULT ONLY AS A SEARCH ENGINE

    • ​STILL WORKING ON MAKING CLUSTER STABLE

  • BACKUPS ARE HARD IF YOU CANNOT CLOSE INDEX

PROBLEMS & SOLUTIONS

HEAP PROBLEMS FOR WIDE SCALE BULK OPERATIONS

NODE.JS IS FASTER THAN ES 

THINGS TO REMEMBER

  • ES IS GREAT AS A READ-ONLY DATABASE

  • ALWAYS REMEMBER ABOUT CHAOS MONKEY

  • POC IS IMPORTANT PART OF DEVELOPMENT

  • ... BUT THE PRODUCTION MAKES THE VERDICT

THINGS TO REMEMBER

EACH FAILURE LEADS TO BETTER EXPERIENCE

QuESTIONS TIME!

THANK YOU!

Tomasz Banasiak

http://banasiak.pro

RST.COM.PL

Icons by: http://www.flaticon.com

Made with Slides.com