Amazon OpenSearch

Hands-On

Demo

In this demo, we will:

  1. Create an Amazon OpenSearch Service domain
  2. Configure access to OpenSearch Dashboards
  3. Ingest sample log data
  4. Create index patterns and visualizations
  5. Perform basic searches
  6. Test the setup
  7. Clean up resources

Agenda

log-analysis-demo

Create OpenSearc h domain

Engine options

master
Master@123

Create

Wait 20 minutes

Launch CloudShell

cat << EOF > sample_logs.json
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:00:00Z", "ip": "192.168.1.1", "method": "GET", "url": "/index.html", "status": 200, "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:01:00Z", "ip": "192.168.1.2", "method": "POST", "url": "/api/data", "status": 201, "user_agent": "Apache-HttpClient/4.5.5" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:02:00Z", "ip": "192.168.1.3", "method": "GET", "url": "/about.html", "status": 404, "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Mobile/15E148 Safari/604.1" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:03:00Z", "ip": "192.168.1.4", "method": "GET", "url": "/products", "status": 200, "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:04:00Z", "ip": "192.168.1.5", "method": "POST", "url": "/api/user", "status": 400, "user_agent": "PostmanRuntime/7.28.4" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:05:00Z", "ip": "192.168.1.1", "method": "GET", "url": "/contact", "status": 200, "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:06:00Z", "ip": "192.168.1.6", "method": "GET", "url": "/nonexistent", "status": 404, "user_agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:07:00Z", "ip": "192.168.1.7", "method": "PUT", "url": "/api/update", "status": 204, "user_agent": "curl/7.64.1" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:08:00Z", "ip": "192.168.1.8", "method": "GET", "url": "/search?q=opensearch", "status": 200, "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" }
{ "index" : { "_index": "apache-logs" } }
{ "timestamp": "2023-09-26T10:09:00Z", "ip": "192.168.1.9", "method": "DELETE", "url": "/api/resource/123", "status": 403, "user_agent": "PostmanRuntime/7.28.4" }

EOF

Create Sample Data File 

Populate Sample Data to OpenSearch

USERNAME=master
PASSWORD=Master@123
DOMAIN_ENDPOINT=
curl -XPOST \
  -u "${USERNAME}:${PASSWORD}" \
  "${DOMAIN_ENDPOINT}/_bulk" \
  --data-binary @sample_logs.json \
  -H 'Content-Type: application/json'

Login to OpenSearch

DevTools

// Basic search query to retrieve all documents
GET apache-logs/_search
{
  "query": {
    "match_all": {}
  }
}

Query 1

// Search for specific HTTP method (GET)
GET apache-logs/_search
{
  "query": {
    "match": {
      "method": "GET"
    }
  }
}

Query 2

// Search for 404 status codes
GET apache-logs/_search
{
  "query": {
    "term": {
      "status": 404
    }
  }
}

Query 3

// Range query to find logs within a specific time range
GET apache-logs/_search
{
  "query": {
    "range": {
      "timestamp": {
        "gte": "2023-09-26T10:00:00Z",
        "lte": "2023-09-26T10:01:30Z"
      }
    }
  }
}

Query 4

Clean Up

Delete the OpenSearch Domain 

🙏

Thanks

for

Watching

Amazon OpenSearch - Hands-On Demo

By Deepak Dubey

Amazon OpenSearch - Hands-On Demo

Amazon OpenSearch - Hands-On Demo

  • 129

More from Deepak Dubey