Anylog: Anomaly detection of heterogeneous logs using deep transformer models

Table of Contents

Introduction

Introduction

• Logs importance
• Daily life use
• Critical systems logs
• Security and reliability
• Difficulty in anomaly detection

Background

Background

• Recent work
• Logmin, Deeplog, Zibrium....
• Data format
• Techniques
• Sequential and Quantitative

Problems

Problems

• Productive Systems
• False alarms
• Require templates/labels
• Log Parsing
• Heterogeneous logs
• Previous methods are costly

Methodology

Methodology

• Unsupervised techniques
• Transformer based models
• Data Streaming
• Retrain model process

Transformer Model

Transformers

Continued...

• Parallel Computation
• Non Sequential
• Self Attention
• Positional Embeddings

Scope

Scope

• Target production level systems
• Automate anomaly detection process
• Real-time data incoming and feeding
• Available publicaly

Tools

Tools

Work Breakdown

FYP-1

1

5

4

3

2

6

10

9

8

7

14

13

12

11

16

15

Project Puposal

Project Defense

Literature Review

Dataset gathering

& Prepration

Data Streaming

Model Selection

Traing/Testing

Logs  Clustering

User Interface

Initial Model(v0.1)

Documentation

Weeks:

Literature Review

Tasks:

FYP-2

1

5

4

3

2

6

10

9

8

7

14

13

12

11

16

15

Inititial Results

Models Variations

Improve Results

Finalize User Interface

Validating Benchmarks

Fine Tuning

Research Paper

Testing

Weeks:

Documentation

Tasks:

Final Results

Conclusion

Conclusion

• At the end of the project, we will have a working framework
• Reveal further uses of transformers on unsupervised problems
• Automate Anomaly detection process

Thank You

&

Any Question?