Elasticsearch - Index Lifecycle Management policy for a time-series index

Hot tier nodes handle the indexing load for time series data such as logs or metrics and hold your most recent, most-frequently-accessed data.


Warm tier nodes hold time series data that is accessed less-frequently and rarely needs to be updated.

 

Cold tier nodes hold time series data that is accessed infrequently and not normally updated. To save space, you can keep fully mounted indices of searchable snapshots on the cold tier. These fully mounted indices eliminate the need for replicas, reducing required disk space by approximately 50% compared to the regular indices.

 

Frozen tier nodes hold time series data that is accessed rarely and never updated. The frozen tier stores partially mounted indices of searchable snapshots exclusively. This extends the storage capacity even further — by up to 20 times compared to the warm tier.

FROZEN

GET _ilm/policy/ILM-Policy-For-Data-Stream
PUT _ilm/policy/ILM-Policy-For-Data-Stream-2
{
  "policy": {
    "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "set_priority" : {
              "priority" : 100
            },
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
      "warm" : {
          "min_age" : "60d",
          "actions" : {
            "set_priority" : {
              "priority" : 50
            }
          }
        },
        "cold" : {
          "min_age" : "90d",
          "actions" : {
            "set_priority" : {
              "priority" : 0
            }
          }
        },
        "frozen" : {
          "min_age" : "120d",
          "actions" : {
            "searchable_snapshot" : {
              "snapshot_repository" : "mysnapshotrepository",
              "force_merge_index" : true
            }
          }
        }
      }
  }
}
GET _ilm/policy/ILM-Policy-For-Data-Stream-2
PUT _index_template/index-template-for-data-stream
{
  "index_patterns": ["timeseries"],                   
  "data_stream": { },
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "ILM-Policy-For-Data-Stream-2"     
    }
  }
}
GET _index_template/index-template-for-data-stream
POST timeseries/_doc
{
  "message": "logged the request",
  "@timestamp": "1591890611"
}

GET timeseries
GET timeseries/_search

GET .ds-timeseries-*/_ilm/explain

THANKS

FOR

WATCHING

Elasticsearch-Define-an-Index-Lifecycle-Management-policy-for-a-time-series-index

By Deepak Dubey

Elasticsearch-Define-an-Index-Lifecycle-Management-policy-for-a-time-series-index

  • 685

More from Deepak Dubey