Anylog: Anomaly detection of heterogeneous logs using deep transformer models

Table of Contents

Introduction

Introduction

• Importance of logs

• Critical system logs

• Security and reliability

• Difficulty in anomaly detection

Background

Background

• Recent work
• Logmin[1], Deeplog[2], LogAnomaly[4]....
• Data format
• Techniques
• Sequential and Quantitative
• Motivation

Problems

Problems

• False alarms
• Require templates/labels
• Log parsing
• Heterogeneous logs

Methodology

Methodology

• Unsupervised techniques
• Transformer based models
• Novelty
• Data Streaming

Scope

Scope

• Target productive systems
• Any critical system can use
• Automate anomaly detection process
• Real-time data streaming and feeding

Tools

Tools

Work Breakdown

FYP-1

1

5

4

3

2

6

10

9

8

7

14

13

12

11

16

15

Project Purposal

Project Defense

Literature Review

Dataset gathering

& Prepration

Data Streaming

Model Selection

Traing/Testing

Logs  Clustering

User Interface

Initial Model(v0.1)

Documentation

Weeks:

Literature Review

Tasks:

FYP-2

1

5

4

3

2

6

10

9

8

7

14

13

12

11

16

15

Inititial Results

Models Variations

Improve Results

Finalize User Interface

Validating Benchmarks

Fine Tuning

Research Paper

Testing

Weeks:

Documentation

Tasks:

Final Results

Conclusion

Conclusion

• Full working framework
• Reveal further uses of transformers on unsupervised problems
• Reveal further uses of logs with transformer models
• Automate Anomaly detection process

References

Refrences

[1] Hamooni, Hossein, et al. "Logmine: Fast pattern recognition for log analytics." Proceedings of the 25th ACM International onConference on Information and Knowledge Management. 2016.

[2] Du, Min, et al. "Deeplog: Anomaly detection and diagnosis from system logs through deep learning." Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017.

[3] Landauer, Max, et al. "Dynamic log file analysis: An unsupervised cluster evolution approach for anomaly detection." computers & security 79 (2018): 94-116.

[4] Meng, Weibin, et al. "LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs." IJCAI. 2019.

[5] Farzad, Amir, and T. Aaron Gulliver. "Unsupervised log message anomaly detection." ICT Express 6.3 (2020): 229-237.

[6] Wang, Jin, et al. "LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things." Sensors 20.9 (2020): 2451.

[7] Nedelkoski, Sasho, et al. "Self-Attentive Classification-Based Anomaly Detection in Unstructured Logs." arXiv preprint arXiv:2008.09340 (2020).

[8] Zhang, Xu, et al. "Robust log-based anomaly detection on unstable log data." Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2019.

[9] Bertero, Christophe, et al. "Experience report: Log mining using natural language processing and application to anomaly detection." 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2017.

Thank You

&

Any Question?