Java Thread dump, Memory dump and Garbage Collector dump Analysis

In a situation like web server or app server or application runs much slower, then we have to investigate thread dump and memory dumps as well as Java Garbage collector dumps. Threads are running in synchronous manner in order to share computer resources without providing any notice on system delays to system users. 

Java Thread life-cycle and Thread Status

Steps to analyze Java thread dumps

1. Find the JVM that used in your app server / web server or application

2. Find out java processor ids by bellow command
     ps -ef | grep java

    Ex : dev64     8393  1641  6 14:04 pts/0    00:00:42 /common/Software/jdk-11

3. Get the thread dump using bellow command. It will dump to file called "threadump"
    /bin/jstack 8393 > threaddump

4. Upload dump file to available online analyzers. 
    Ex : https://fastthread.io/   ,    https://jstack.review/
   

5. It will display the content of thread dump. Similar to bellow samples

"http-bio-8180-exec-11425" daemon prio=10 tid=0x00007f89e00a7800 nid=0x3536 runnable [0x00007f89b5294000]
   java.lang.Thread.State: RUNNABLE

"http-bio-8443-exec-39" daemon prio=10 tid=0x00007fd680035000 nid=0x78b6 waiting for monitor entry [0x00007fd6c2994000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:678)

at com.testapp.getUserFromTbl(UserDao.java:90)

"http-bio-8443-exec-10" daemon prio=10 tid=0x00007fd6a0021800 nid=0x3702 waiting on condition [0x00007fd6c2a99000]
   java.lang.Thread.State: TIMED_WAITING 


 Check bellow items from thread dump Analysis
 
 5.1. Check Blocked and Deadlock threads. You can find actual method that involved in actual slowness of the application similar to bellow

"http-bio-8443-exec-39" daemon prio=10 tid=0x00007fd680035000 nid=0x78b6 waiting for monitor entry [0x00007fd6c2994000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:678)

at com.testapp.getUserFromTbl(UserDao.java:90)

In this thread dump, there is an issue related to line 90 in UserDao.java . Check java class for possible improvements.


5.2. Find Blocked and Deadloack thread names and search it in your application log. Here it is http-bio-8443-exec-39

5.3. Add additional start and end logs to the method getUserFromTbl. Print the total time taken for given method to get more insight of the issue.

5.4. Get another thread dump and check the Waiting and Timed Waiting thread count. If these counts getting increased time to time, then it makes system slowness.

5.5. Now you able to find out  laces that made system slowness. 

6. Think about possible improvements similar to bellow approaches.

      6.1. Check CPU and Memory utilization for each thread. If required increase memory and cpu.

               ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

      6.2 Integrate your source code with code quality measurement tools and do the recommended improvements. Ex : sonar .

      6.3 Upgrade System libraries, JDK version and optimize your application/web server thread pools

      6.4 Introduce timeouts for integrations ( Ex: DB, APIs )

      6.5. Move from monolithic model to microservice models.



Steps to analyze Java memory dumps


1. Find process id of your application
    ps -ef | grep java

    Ex : dev64     8393  1641  6 14:04 pts/0    00:00:42 /common/Software/jdk-11

2. Get the memory dump from bellow command. It will dump to file called dump.hprof in current directory

     jmap -dump:live,format=b,file=dmp.hprof 8393

3. Install currently available java memory analyzer or Use online memory dump analyzers
 
 Ex : Eclipse memory analyzer , https://heaphero.io/

4. Upload and check the analysis of bellow reports

    1. Object count per each class ( Histogram )
        Ex: If String has many objects then you can fix the issue by introducing StringBuilder

    2. Memory utilization per each class, You can sort it by memory utilization

    3. Duplicated classes loaded by multiple class loaders.

    4. Memory leaks. They occurred whenever garbage collectors unable to collect memory objects which are no longer in use. You can avoid memory leaks by bellow good practices

        4.1. Reduce static variables including static collections which are resides full lifetime of the application
        4.2.  Close resources one use. You can use try-with-resources block
        4.3. Properly override equals() and hashCode() method to avoid duplicate objects
        4.4. Reduce non static inner classes
        4.5. Implement AutoClosealble and avoid finalizes for resource usage codes
     


Steps to analyze Java GC ( Garbage Collector ) dumps

   
In Java, there are two memory areas called stack and heap. Bellow table compares difference between stack and heap. Both are resides in RAM.

Stack	Heap
Used for static memory allocations	Used for dynamic memory allocations
Used for temporary variables created by  functions	Used for entire application
Thread level variables will be maintained	Elements can be access global level
Maintained by CPU FIFO mechanism	Maintain by JVM GC
Allocation, De-allocation and access is very fast	Comparatively slow

GC analysis is basically related to Heap memories. Heap area has three basic components. They are younger generation, old generation and permanent generation. New elements will be created in younger generation. Whenever younger generation getting filled, then JVM runs first level garbage cleaning.  It will remove all non referenced and non used objects. On specific time threshold, remaining younger generation will be moved to old generation. JVM again runs second level garbage cleaning on old generation. Permanent generation contains master data which are required by JVM for its operations. 

 

Steps to obtain Java GC dump

1. Add bellow JVM parameters to dump heap memory in to a dump file located at /home/gc.log

     -XX:+PrintGCDetails -verbose:gc -Xloggc:/home/gc.log
   
     Note : parameters may be changed based on JVM verison

2. Upload it to freely available tools or view the gc.log using text viewer
      Ex : https://gceasy.io/

3. You can find bellow parameters

    3.1. Allocated memory for younger generation, old generation and JVM master data
    3.2.  Max usage of each memory types
    3.3. GC pause time ( All user threads will be user stopped when GC runs )
    3.4. Any memory leaks ( Comes when GC unable to clean memory object which are no longer used )
 
4. GC health can be improved based on bellow configuration and coding best practices 

        4.1. Reduce static variables including static collections which are resides full lifetime of the application
        4.2. Close resources one use. You can use try-with-resources block
        4.3. Properly override equals() and hashCode() method to avoid duplicate objects
        4.4. Reduce non static inner classes
        4.5. Implement AutoClosealble and avoid finalizes for resource usage codes
        4.6. After above improvements,  add enough stack and heap spaces in JVM options
                 -Xss is initial size of stack space
                 -Xms and -Xmx for initial and max size of heap spaces

                  Note : JVM paramter names may be changed based on your JVM version

        4.7. StackOverFlowError and OutOfMemoryError comes whenever JVM has insufficient memory configurations in stack and heap respectively. Also you may required more improvements in your codes.