Rolling update Elasticsearch

What issues in current ELKT

• Update ES stack will lose data
• ES nodes are not well configed
• Stack is not based on base-AMI

What the new structure will bring

• Support for rolling upgrade of an ES cluster 
• Minimal of data loss
• More secure
• Support snapshot of ES data
• ES nodes will perform more effective

More secure

This is done by using https in load balancer

Snapshot support

• S3 to store snapshots. The S3 repository can be created using the following command:

curl -XPUT 'https://kibana.property-finance-es-cluster.
property-finance.realestate.com.au:9200/_snapshot/property_finance_s3_repository'
 -d '{
    "type": "s3",
    "settings": {
        "bucket": "rea-mad-elasticsearch-snapshots",
        "region": "ap-southeast-2",
        "base_path": "fp-elasticsearch-cluster-01/"
    }
}'

export SNAPSHOT_NAME=$(date +%Y%m%d)
curl -v -XPUT 
"https://kibana.property-finance-es-cluster.propertyfinance.realestate.com.au:9200/_snapshot/
property_finance_s3_repository/snapshot_$SNAPSHOT_NAME?wait_for_completion=true"

• Create snapshot

Snapshot support

• Restore snapshot

curl -XPOST 
"localhost:9200/_snapshot/my_backup/snapshot_1/_restore?wait_for_completion=true"

1.  Indices have to be closed prior to restore 
2. Both backup and restore operations are incremental

ES nodes configuration

• Master nodes for cluster management
• Data nodes do the heavy duty of indexing and searching 
• Master nodes using small instance while data nodes using larger instance to make full use of nodes

Rolling update 

• Ensure previous stack is green
• Launch new stack
• wait until all nodes to register
• wait until nodes rebalanced
• switch DNS pointer
• scale down previous stack
• delete previous stack