Future Architecture
Technical Principles
Architecture
Product Impacts
Log
Event
10 21 2016 10:16:50 10.3.10.52 <LOC0:INFO> Oct 21 10:16:50 192.168.1.9
EVID:0022 Denial of service detected 181.222.137.14:40089 -> FTPServer.acme.com:1723 UDP
{
"keyField": "messageId",
"count": 1,
"classificationId": 2500,
"classificationName": "Denial Of Service",
"classificationTypeName": "Security",
"commonEventName": "Host Denial Of Service",
"commonEventId": 1034537,
"direction": 3,
"directionName": "External",
"entityId": 7,
"entityName": "Child2B",
"impactedEntityName": "Global Entity",
"impactedHost": "161.200.1.6",
"impactedIp": "161.200.1.6",
"impactedLocation": "Thailand, Krung Thep Maha Nakhon, Bangkok",
"impactedCountry": "Thailand",
"impactedRegion": "Krung Thep Maha Nakhon",
"impactedCity": "Bangkok",
"impactedPort": 401,
"impactedZoneName": "External",
"logDate": 1477045118000,
"logMessage": "10 21 2016 10:18:38 10.3.10.52 <LOC0:INFO> Oct 21 10:18:38 192.168.1.6 EVID:0022 Denial of service detected 238.231.189.41:40037 -> 161.200.1.6:401 UDP",
"logSourceHost": "192.168.1.6",
"logSourceHostId": 20,
"logSourceHostName": "192.168.1.6",
"logSourceId": 39,
"logSourceName": "192.168.1.6 LR SyslogGen",
"logSourceType": 105,
"logSourceTypeName": "Syslog - LogRhythm Syslog Generator",
"messageId": "807988387",
"messageTypeEnum": 2,
"mpeRuleId": 30272,
"mpeRuleName": "Denial of Service",
"normalDate": 1477066718007,
"normalDateMin": 1477066718007,
"normalMsgDateMax": 1477066718007,
"originEntityId": -100,
"originEntityName": "Global Entity",
"originHostId": -1,
"originHost": "238.231.189.41",
"originIp": "238.231.189.41",
"originPort": 40037,
"originZone": 3,
"originZoneName": "External",
"priority": 47,
"protocolId": 17,
"protocolName": "UDP",
"serviceId": 3580,
"serviceName": "UPS - Uninterruptible Power Supply",
"portProtocol": "UPS - Uninterruptible Power Supply",
"vendorMessageId": "0022"
}
vs
Composition
Themes
- Consolidate, isolate, & reuse
- "Pinning" => discovery & load balancing
- Denormalize data at processing time
Netmon
- Consolidate storage
- Share infrastructure
- Merge UIs
Event Schema
- Support extensible schema
- Support array-type values
- Data-driven consumers
{
logDate : "2016-10-18T20:18:17Z",
priority : 47,
logSourceName : "Syslog - LogRhythm Syslog Generator",
...
listMatches : [14, 48, 103],
extensions : [ {
key : "emailAttachments",
value : "babyPicture.jpg",
type : "String",
} ]
}
Repositories
- Move away from "logs" / "events" tiers
- One DX cluster per data center
- Allow configurable repositories
- Promote % of storage over TTL
Repositories
Name: "High Risk" Capacity: 50% Filter: priority >= 70
Name: "Financial Servers" Capacity: 30% Filter: originIp in list:financialServers
Name: "SSH Traffic" Capacity: 20% Filter: application == "SSH - Secure Shell"
Filtering
- Filters are first class objects
- Universal query DSL
- Simple vs Complex filter features
Archiving
- Downstream of processing
- Store all metadata
- Support basic searching?
Denormalize!
- Log metadata
- List lookups
- Alarm attribution
- Case evidence
New Feature Areas
- LUA Rules
- Enrichment Jobs
- Powerful Log Distribution
Future Architecture #2
By autoric
Future Architecture #2
- 536