Memory leak debugging

Vít Jouda

Agenda

  • a little bit of theory
     
  • heap analyzers
     
  • common problems
     
  • K8s
     
  • practical examples

Sources

https://www.digitalocean.com/community/tutorials/java-jvm-memory-model-memory-management-in-java
https://www.freecodecamp.org/news/garbage-collection-in-java-what-is-gc-and-how-it-works-in-the-jvm/
https://www.baeldung.com/java-gc-cyclic-references#tracing-gcs

JVM memory model

GC

Tracing GC

GC Roots

Thread

 

System Class

 

Local variable

 

JNI

 

 Others

Glossary

  • shallow size
    • memory occupied by the object itself
       
  • retained size
    • shallow size of all objects reachable from selected object
       
  • dominator
    • object A dominates B if every path from root to B goes thru A
       
  • merged path
    • paths from multiple objects grouped by class

Heap analyzers

IntelliJ Idea Profiler

Visual VM


Memory Analyzer Tool

 

...

Common problems

  • caches
    • keys, improper sizing
  • metrics
    • tag cardinality
      • URL params
      • device SN
  • improper resource handling
    • I/O
    • iterators
  • native memory
    • RocksDB
    • webClient

Native memory

  • -XX:NativeMemoryTracking

    • ​summary

    • detail

       

  • malloc arena

    • glibc

    • performance optimization

      • ​may not return to OS

    • defaults to 8x CPU core #

    • jemalloc

K8s

How to get a heap dump?

  • K8s aware memory analyzers (JProfiler, licensed)
     
  • On Java OOM 
    • -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/dump"

    • does not work for system level OOMKilled
       

  • PreStop hook

    • terminationGracePeriodSeconds
      ​

  • manually

Where to store a heap dump?

  • /dump
    • emptyDir volume

    • created when pod is first assigned to a node

    • removed once the pod is removed
       

  • OOM Kill

    • pod is not evicted

    • RestartPolicy
    • kubectl describe - Last State

Manual heap dump creation

  • jmap

    • needs JDK container image

    • jmap -dump:live,format=b,file=/dump/dump.hprof 1

  • jattach

    • JVM Swiss knife
    • https://github.com/jattach/jattach
    • available in Temurin JRE APT repositories
    • jattach 1 dumpheap /dump/dump.hprof

Download heapdump

  • Compress
    • gzip -k dump.hprof
  • Download
    • kubectl cp --retries=10 -c {container} {namespace}/{pod name}:/dump/dump.hprof.gz ./dump.hprof.gz

​

Heapdump is a "stop the world" event!



Practical examples

Made with Slides.com