Future Architecture

Technical Principles

Architecture

Product Impacts

Log

Event

10 21 2016 10:16:50 10.3.10.52 <LOC0:INFO> Oct 21 10:16:50 192.168.1.9 
EVID:0022 Denial of service detected 181.222.137.14:40089 -> FTPServer.acme.com:1723 UDP
{
  "keyField": "messageId",
  "count": 1,
  "classificationId": 2500,
  "classificationName": "Denial Of Service",
  "classificationTypeName": "Security",
  "commonEventName": "Host Denial Of Service",
  "commonEventId": 1034537,
  "direction": 3,
  "directionName": "External",
  "entityId": 7,
  "entityName": "Child2B",
  "impactedEntityName": "Global Entity",
  "impactedHost": "161.200.1.6",
  "impactedIp": "161.200.1.6",
  "impactedLocation": "Thailand, Krung Thep Maha Nakhon, Bangkok",
  "impactedCountry": "Thailand",
  "impactedRegion": "Krung Thep Maha Nakhon",
  "impactedCity": "Bangkok",
  "impactedPort": 401,
  "impactedZoneName": "External",
  "logDate": 1477045118000,
  "logMessage": "10 21 2016 10:18:38 10.3.10.52 <LOC0:INFO> Oct 21 10:18:38 192.168.1.6 EVID:0022 Denial of service detected 238.231.189.41:40037 -> 161.200.1.6:401 UDP",
  "logSourceHost": "192.168.1.6",
  "logSourceHostId": 20,
  "logSourceHostName": "192.168.1.6",
  "logSourceId": 39,
  "logSourceName": "192.168.1.6 LR SyslogGen",
  "logSourceType": 105,
  "logSourceTypeName": "Syslog - LogRhythm Syslog Generator",
  "messageId": "807988387",
  "messageTypeEnum": 2,
  "mpeRuleId": 30272,
  "mpeRuleName": "Denial of Service",
  "normalDate": 1477066718007,
  "normalDateMin": 1477066718007,
  "normalMsgDateMax": 1477066718007,
  "originEntityId": -100,
  "originEntityName": "Global Entity",
  "originHostId": -1,
  "originHost": "238.231.189.41",
  "originIp": "238.231.189.41",
  "originPort": 40037,
  "originZone": 3,
  "originZoneName": "External",
  "priority": 47,
  "protocolId": 17,
  "protocolName": "UDP",
  "serviceId": 3580,
  "serviceName": "UPS - Uninterruptible Power Supply",
  "portProtocol": "UPS - Uninterruptible Power Supply",
  "vendorMessageId": "0022"
}

vs

Composition

Themes

  • Consolidate, isolate, & reuse
  • "Pinning" => discovery & load balancing
  • Denormalize data at processing time

Netmon

  • Consolidate storage
  • Share infrastructure
  • Merge UIs

Event Schema

  • Support extensible schema
  • Support array-type values
  • Data-driven consumers
{
    logDate : "2016-10-18T20:18:17Z",
    priority : 47,
    logSourceName : "Syslog - LogRhythm Syslog Generator",
    ...
    listMatches : [14, 48, 103],    
    extensions : [ { 
        key : "emailAttachments",
        value : "babyPicture.jpg",
        type : "String",
    } ]
}

Repositories

  • Move away from "logs" / "events" tiers
  • One DX cluster per data center
  • Allow configurable repositories
  • Promote % of storage over TTL
Repositories

Name: "High Risk"            Capacity: 50%    Filter: priority >= 70
Name: "Financial Servers"    Capacity: 30%    Filter: originIp in list:financialServers
Name: "SSH Traffic"          Capacity: 20%    Filter: application == "SSH - Secure Shell"

Filtering

  • Filters are first class objects
  • Universal query DSL
  • Simple vs Complex filter features

Archiving

  • Downstream of processing
  • Store all metadata
  • Support basic searching?

Denormalize!

  • Log metadata
  • List lookups
  • Alarm attribution
  • Case evidence

New Feature Areas

  • LUA Rules
  • Enrichment Jobs
  • Powerful Log Distribution

Future Architecture #2

By autoric

Future Architecture #2

  • 536