Using OpenSearch after Elasticsearch

Introduction

  • Overview
    • OpenSearch: Open-source fork of Elasticsearch 7.10.2, maintained by AWS.
    • Origin: Forked due to licensing changes in Elasticsearch.

 

  • Objective
    • Understand key features of OpenSearch.
    • Learn how to migrate and leverage OpenSearch
    • Explore differences and enhancements over Elasticsearch.

Key Features of OpenSearch

  • Open-Source Licensing
    • Apache 2.0 License ensures continued open-source development.
  • Enhanced Security
    • Built-in security features such as RBAC, encryption, and audit logging.
  • Advanced Analytics
    • Anomaly detection, alerting, and reporting capabilities.
  • Compatibility
    • Fully compatible with Elasticsearch 7.10.2 indices and APIs.

Migration from Elasticsearch to OpenSearch

  • Preparation
    • Evaluate current Elasticsearch setup.
    • Backup all indices and configurations.
  • Migration Steps
    • Snapshot and Restore: Create a snapshot in Elasticsearch and restore in OpenSearch.
    • Reindexing: Use the Reindex API to transfer data.
POST /_reindex
{
  "source": {
    "remote": {
      "host": "http://source_elasticsearch:9200"
    },
    "index": "source_index"
  },
  "dest": {
    "index": "destination_index"
  }
}
  • Post-Migration Validation
    • Verify data integrity and cluster health.
    • Ensure all applications are pointing to the new OpenSearch cluster.

Enhanced Security Features

  • Role-Based Access Control (RBAC)
    • Fine-grained permissions for indices, documents, and fields.
  • Encryption
    • TLS encryption for data in transit and at rest.
  • Audit Logging
    • Detailed logs of user actions and system changes for compliance and security auditing.

Advanced Analytics

  • Anomaly Detection
    • Detects unusual patterns in data using machine learning.
  • Alerting
    • Set up alerts based on custom rules and thresholds.
  • Reporting
    • Generate and schedule detailed reports from OpenSearch Dashboards.

API Enhancements in OpenSearch

  • Anomaly Detection API
    • Provides machine learning-based anomaly detection.
POST _plugins/_anomaly_detection/detectors
{
  "name": "example-detector",
  "description": "Anomaly detector for example",
  "time_field": "timestamp",
  "indices": ["example-index"],
  "feature_attributes": [
    {
      "feature_name": "mean_response_time",
      "feature_enabled": true,
      "aggregation_query": {
        "response_time_avg": {
          "avg": {
            "field": "response_time"
          }
        }
      }
    }
  ]
}
  • Alerting API
    • Create, manage, and monitor alerts based on conditions.
POST _plugins/_alerting/monitors
{
  "name": "example-monitor",
  "type": "monitor",
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 1,
      "unit": "MINUTES"
    }
  },
  "inputs": [
    {
      "search": {
        "indices": ["example-index"],
        "query": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "response_time": {
                      "gte": 1000
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  ],
  "triggers": [
    {
      "name": "example-trigger",
      "severity": "1",
      "condition": {
        "script": {
          "source": "ctx.results[0].hits.total.value > 0",
          "lang": "painless"
        }
      },
      "actions": []
    }
  ]
}

Query DSL Enhancements

  • Elasticsearch Query DSL
    • Comprehensive query capabilities including full-text search, structured search, and analytics.
    • API Endpoints: _search, _count.
  • OpenSearch Query DSL
    • Extends Elasticsearch Query DSL with additional functions and plugins.
    • Example: Enhanced support for nested fields and custom scoring.

Migration Considerations

  • Compatibility
    • Many APIs are directly compatible due to the shared heritage.
    • Differences in security and plugin APIs may require adjustments.
  • Testing
    • Thorough testing in a staging environment before production migration.
    • Use tools like elasticsearch-dump for data migration and validation.

Q&A

  • Questions from the audience

Migration from elasticsearch to opensearch

By TenantCloud

Migration from elasticsearch to opensearch

Migration from elasticsearch to opensearch

  • 101

More from TenantCloud