"Hear me  my flows"

(.. and see my music)

Thanks to:

• http://www.georgeandjonathan.com/

• http://www.fourmilab.ch/webtools/midicsv/

• https://github.com/markc/midicomp

• https://github.com/nsfmc/atoll.js

• https://github.com/aol/moloch

Andrew Leahy

Mark Constable

John Walker

Marcos Ojeda

Andy Wick

bits, packets & flows

Flows

Packets

bits

try to memorize two

and compare to 

one what you see

minor thing like different scales

sessions

time buckets

var names = ['bits','packets'];
var buckets = Array(numberofbuckets);
while ( sessions.length > 1 ) {
    var {start, duration, data} = sessions.shift();
    var firstbucket = ~~((start - begin) / step);
    var lastbucket = ~~((start - begin + duration) / step) 
    for (var n = firstbucket; (n <= lastbucket ); n++){
        for (var name in names){
	    buckets[n][names[name]].push(data[name]);
	}
    }
}

sum, yeah, but what about min ?

max ?

mean ?

central moment ?

variance ?

skewness ?

...

        /*
	helper to get function names what return numeric results
	*/
	function _atollNumericFunctions(){
		var ret = {};
		var obj = atoll([1,2,3,4,5,6]);
		for (var prop in obj) {
			if (typeof(obj[prop]) === 'function') {
				try {
					var tmp = obj[prop]();
				} catch(err) {
					//noop
				} finally {
					if ( typeof(tmp) === 'number') {
						ret[prop] = NaN;	
					}			
				}
			}
		}
		return ret;
	}

    while ( buckets.length > 1 ) {
	var bucket = buckets.shift();
	for (var column in bucket){		
	    var stat = atoll(bucket[column],true);
	    for (var f in functions) {
		var trackname = column +'_'+ f;
		if (!stats[trackname]){
		    stats[trackname] = [];
		}
		if (bucket[column].length > 0) 	{
		    stats[trackname].push(stat[f]());
		} else {
		    stats[trackname].push(NaN);
		}
            }
	}
    }

how do i debug !?

https://gist.github.com/hillar/54a7dc0b8abbdf03cc07#file-molo2midi-js

The human eye can differentiate about 10 million colors... 

please count colors for me

• green
• yellow
• red
• blue
• gray
• bläh...

The human ear can generally hear sounds with frequencies between 20 Hz and 20 kHz

count some musical instruments to me, please ;)

• acoustic guitar
• eletric guitar
• bass guitar
• harp
• violin
• piano
• organ
• big bass drum
• trombone
• oboe
• please stopppp

tons of
moving 3D pictures

$ man csvmidi

NAME

csvmidi - encode CSV file as MIDI
SYNOPSIS

csvmidi [ -u -v -x -z ] [ infile [ outfile ] ]
DESCRIPTION

csvmidi reads a CSV (Comma-Separated Value) file in the format written by midicsv 
and creates the equivalent standard MIDI file.

OPTIONS

-u
Print how-to-call information.
-v
Print verbose debugging information on standard error. The MIDI file header is dumped, along with the length of each track in the file.
-x
MIDI streams support a rudimentary form of compression in which successive events with the same “status” (event type and channel) may omit the status byte. By default csvmidi avails itself of this compression. If the -x option is specified, the status byte is emitted for all events—it is never compressed even when the MIDI standard permits it to be.
-z
Most errors detected in CSV records cause a warning message to be displayed on standard error and the record ignored. The -z option causes csvmidi to immediately terminate processing when the first error is detected.
EXIT STATUS

If no errors or warnings are detected csvmidi exits with status 0. A status of of 1 is returned if one or more errors were detected in the CSV input file, while a status of 2 indicates a syntax error on the command line or inability to open the input or output file.

FILES

If no infile is specified or infile is “-”, csvmidi reads its input from standard input; if no outfile is given or outfile is “-”, MIDI output is written to standard output. The input and output are processed in a strictly serial manner; consequently csvmidi may be used in pipelines without restrictions.

BUGS

csvmidi assumes its input is in the format written by midicsv. If supplied a CSV file with well-formed records which nonetheless makes no semantic sense as a MIDI file, the results will, in all likelihood, simply perplex any program or instrument to which it's sent. csvmidi checks for missing fields and range checks all numeric values, but does not perform higher-level consistency checking (for example, making sure that every note on event is paired with a subsequent note off). That level of verification, if required, should be done on the CSV file before it is processed by csvmidi.

Exporting a file to CSV with midicsv and then importing it with csvmidi is not guaranteed to create an identical MIDI file. MIDI files support compression modes which are not obligatory. A MIDI file exported to CSV and then re-imported should, however, be equivalent to the original file and should, if exported to CSV, be identical to the CSV exported from the original file.

Please report problems to bugs at fourmilab.ch.

SEE ALSO

midicsv(1), midicsv(5)

A picture is worth a thousand words 

A man page is worth a thousand pictures 

1. curl -ks "https://moloch:moloch@molo.ch/sessions.csv/sessions.csv?&date=-1&expression=protocols%3D%3Ddns" > dns-sessions.csv
    

2. nodejs molo2midi.js   dns-sessions.csv > dns-sessions.txt
    

3. midicomp -c dns-sesstions.txt > dns.mid

or just pipe it together ;)

dns & ssh sessions data from

Locked Shields 2015 testrun

https://ccdcoe.org/locked-shields-2015.html

Locked Shields is a real-time network defence exercise, organised annually  by the NATO Cooperative Cyber Defence Centre of Excellence. 

16 nations and NATO Computer Incident Response Capability participated as the Blue Teams of Locked Shields 2015.

to be continued .. (?)

http://research.gold.ac.uk/9795/