How Grapl Uses Rust

Github:

https://github.com/insanitybit/grapl

Twitter:

@InsanityBit

Grapl

Graph Analytics Platform

for Detection, Forensics, and Incident Response

 

  • Consumes logs
  • Turn logs into subgraphs
  • Merge subgraphs into master graph
  • Executes code against master graph to find scary patterns
{
  "host_id": "cobrien-mac",
  "parent_pid": 3,
  "pid": 4,
  "image_name": "word.exe",
  "create_time": 600,
}
{
  "host_id": "cobrien-mac",
  "parent_pid": 4,
  "pid": 5,
  "image_name": "payload.exe",
  "create_time": 650,
}

word.exe

payload.exe

ssh.exe

/secret/file

evil.com

explorer.exe

payload.exe

word.exe

GrapL DESIGN GOALS

  • Handle thousands of logs per second

 

  • Assume some of these logs contain attacker controlled information (after all, the whole point is to catch logs that attackers are triggering)

 

  • Be as low maintenance as possible - automatic scaling, deployment, safe by default, etc

generic-subgraph-generator

generated-subgraphs

identified-subgraphs

generated-subgraphs-events

node-identifier-queue

node-identifier

  • S3, SQS, SNS
  • Futures-compatible
  • Active development
  • >100 AWS Services supported

 

    https://github.com/rusoto/rusoto

#[async]
fn read_message<M, S>(s3_client: Rc<S>, bucket: String, path: String)
    -> Result<M, Error>
    where M: Message + Default,
          S: S3 + 'static
{
    info!("Fetching data from {} {}", bucket, path);
    let object = await!(
            s3_client.get_object(&GetObjectRequest {
                bucket: bucket.to_owned(),
                key: path,
                ..GetObjectRequest::default()
           })
        )?;

    let mut body = vec![];

    #[async]
    for chunk in object.body.unwrap() {
        body.extend_from_slice(&chunk);
    }

    Ok(M::decode(body)?)
}

COMMUNICATION

Prost:

 

 

 https://github.com/danburkert/prost

message NodeDescription {
    oneof which_node {
        ProcessDescription process_node = 1;
        FileDescription file_node = 2;
        IpAddressDescription ip_address_node = 3;
        OutboundConnection outbound_connection_node = 4;
        InboundConnection inbound_connection_node = 5;
    }
}

Communication

Prost:


pub mod graph_description {
    include!(concat!(env!("OUT_DIR"), "/graph_description.rs"));
}

// Implement helper functions directly on proto message struct
impl OutboundConnection {
    pub fn clone_key(&self) -> String {
        self.node_key.clone()
    }

    pub fn set_key(&mut self, key: impl Into<String>) {
        self.node_key = key.into();
    }

    pub fn set_asset_id(&mut self, asset_id: impl Into<String>) {
        self.asset_id = Some(asset_id.into())
    }
}

Communication

Prost

extern crate prost_build;

fn main() {
    let mut config = prost_build::Config::new();
    config.type_attribute(".", "#[derive(Eq)]");

    config.type_attribute(".graph_description.FileDescription", 
                          "#[derive(Builder)]");
    config.type_attribute(".graph_description.FileDescription",
                          "#[builder(setter(into))]");

    config.field_attribute(".graph_description.FileDescription.node_key",
                           "#[builder(field(private))]");
    config.field_attribute(".graph_description.FileDescription.node_key",
                           "#[builder(default = \"::uuid::Uuid::new_v4().to_string()\")]");

    config
        .compile_protos(&[
            "proto/graph_description.proto",
        ], &["proto/"])
        .unwrap_or_else(|e| panic!("protobuf compilation failed: {}", e));
}

Communication

Prost

fn handle_outbound_traffic(conn_log: OutboundConnectionLog) -> GraphDescription {
    let mut graph = GraphDescription::new(
        conn_log.timestamp
    );

    // High level builder, keeps 'node_key' field private
    // and auto-generated
    let process = ProcessDescriptionBuilder::default()
        .host_ip(conn_log.src_addr.clone().into_bytes())
        // ProcessState is automatically converted to u64
        .state(ProcessState::Existing)
        .pid(conn_log.pid)
        .timestamp(conn_log.timestamp)
        .build()
        .unwrap();

    // etc etc
}

Communication

zstd

https://docs.rs/zstd/0.4.21+zstd.1.3.7/zstd/

extern crate zstd;

use std::io;

fn main() {
    // Uncompress input and print the result.
    zstd::stream::copy_decode(io::stdin(), io::stdout()).unwrap();
}

Rust on AWS Lambda

Crowbar

Run rust on lambda by leveraging CFFI from Python

lambda!(|event, context| {
    println!("hi cloudwatch logs, this is {}", context.function_name());
    // return the event without doing anything with it
    Ok(event)
});

Rust on AWS Lambda

Rust AWS Lambda

 

 

https://github.com/srijs/rust-aws-lambda

extern crate aws_lambda as lambda;

fn main() {
    // start the runtime, and return 
    // a greeting every time we are invoked
    lambda::start(|()| Ok("Hello ƛ!"))
}
extern crate sqs_microservice;

use sqs_microservice::handle_s3_sns_sqs_proto;

pub fn main() {
    // We can cache across 'hot' lambda reloads
    let lru_cache = IdentityCache::new(1234, 1234, b"cache_secret");

    handle_s3_sns_sqs_proto(
        move |subgraph: GraphDescription| {
            let mut dgraph_client =
                dgraph_client::new_client("db.mastergraph:9080");
            merge_subgraph(&dgraph_client, &subgraph.into(), lru_cache.clone())
        },
        // Upon success
        move |(earliest, latest)| {
            let event = SubgraphMerged {
                earliest,
                latest,
            };

            publish_graph_merge_event(&event)?;

            Ok(())
        }
    )
}

re:Invent 2018

https://github.com/awslabs/aws-lambda-rust-runtime

re:Invent 2018

#[derive(Serialize, Deserialize)]
struct GreetingEvent {
    greeting: String,
    name: String,
}

#[derive(Serialize, Deserialize)]
struct GreetingResponse {
    message: String,
}

fn main() -> Result<(), Box<dyn Error>> {
    simple_logger::init_with_level(log::Level::Debug).unwrap();
    lambda!(my_handler);

    Ok(())
}

fn my_handler(event: GreetingEvent, ctx: Context) -> Result<GreetingResponse, HandlerError> {
    if event.name == "" {
        error!("Empty name in request {}", ctx.aws_request_id);
        return Err(ctx.new_error("Empty name"));
    }

    Ok(GreetingResponse {
        message: format!("{}, {}!", event.greeting, event.name),
    })
}

Building Grapl

rust-musl-builder

 

 

https://github.com/emk/rust-musl-builder/

(Requires custom fork with extra deps for linking grpcio, zstd)

alias rust-musl-builder='docker run --rm -it -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder'
rust-musl-builder cargo build --release

Deploying Grapl

aws-cdk

https://github.com/awslabs/aws-cdk/

 let environment = {
    "HISTORY_DB_USERNAME": process.env.HISTORY_DB_USERNAME,
    "HISTORY_DB_PASSWORD": process.env.HISTORY_DB_PASSWORD,
    "BUCKET_PREFIX": process.env.BUCKET_PREFIX
};

let service = new Service(this, 'node-identity-mapper', environment, vpc);
    let s3_client = S3Client::simple(Region::UsEast1);
    let bucket_prefix = std::env::var("BUCKET_PREFIX").unwrap();
    s3_client.put_object(&PutObjectRequest {
        bucket: bucket_prefix + "-grapl-raw-log-bucket",
        key,
        body: Some(logs.into()),
        ..Default::default()
    }).wait()?;

RUST:

Why Rust?

?

Questions?

?

@InsanityBit

Grapl - A

By Colin

Grapl - A

  • 1,262