1. Symptoms:
 
Memory analysis of Java applications is necessary to troubleshoot the following symptoms:
 

1. When a Java application is exhibiting abnormal memory growth.
2. When a Java application is performing very slowly.
3. When a Java application is showing high CPU usage due to frequent Major Garbage Collection.

 
2. Basic Information About Memory Spaces In Java:
 
Memory Management in Java Application is a very complex phenomenon. The entire memory usage by a Java application is divided into following distinct spaces.
 
A. Heap: Heap memory is the runtime data area from which the Java VM allocates memory for all class instances and arrays from the Java Application. Any object created in the heap space has global access and can be referenced from anywhere of the application. The heap may be of a fixed or variable size.
 
The Initial and Maximum Heap Size for a Java application is set by –Xms and –Xmx JVM flags. When total Heap memory usage of the application hits the Maximum Heap Size set via the JVM flag –Xmx ( default value=256M), the application throws the following error: java.lang.OutOfMemoryError: Java heap space
 
Heap space is sub-divided into two sub-spaces, YoungGen and OldGen.
 
a. YoungGen/Nursery: YoungGen is the amount of heap reserved for allocation of new objects. When the YoungGen becomes full, Minor Garbage Collection is triggered and all objects that have lived long enough in the YoungGen are promoted (moved) to the OldGen, thus freeing up the YoungGen for more new object allocation.
 
The reasoning behind YoungGen is that most objects are temporary and short lived. A minor GC is designed to be swift at finding newly allocated objects that are still alive and moving them away from the YoungGen. Typically, a minor GC frees a given amount of memory much faster.
 
Just like Initial and Maximum Heap size, Initial and Maximum Size of the YoungGen can be specified by the JVM flags: –XX:NewSize (same as –Xmn) and –XX:MaxNewSize.
 
There is a JVM option –XX:NewRatio to specifiy the YoungGen size as a ratio of OldGen size.
 
i.e. setting –XX:NewRatio=3 means that OldGen will be 3 times of the YoungGen.
 
When combining –XX:NewSize, –XX:MaxNewSize and –XX:NewRatio, the absolute values specified for the YoungGen using –XX:NewSize and –XX:MaxNewSize flags will take precedence over the ratio.
 
YoungGen space is again sub-divided into three parts, EdenSpace, SurvivorSpace1 and SurvivorSpace2.
 
1. EdenSpace: Most of the newly created objects are initially located in the Eden memory space. When Eden space is filled with objects, Minor GC is performed and moved to one of the Survivor spaces.
 
2. SurvivorSpace: Minor GC also checks the survivor objects and moves them to the other survivor space. So at any point of time, one of the SurvivorSpace is always empty. Objects that have survived after many cycles of Minor GC are moved to the OldGen memory space.
 
The JVM flag –XX:SurvivorRatio is used to set the value of the EdenSpace as a ratio of SurvivorSpaces.
 
i.e. setting –XX:ServivorRatio=3 means that EdenSpace will be 3 times of each of the two SurvivorSpaces, i.e. EdenSpace will take 3/5 of the entire YoungGen, where as SurvivorSpace1 and SurvivorSpace2 will take 1/5 of the entire YoungGen.
 
b. OldGen/Tenured: OldGen contains the objects that are long lived and survived after many rounds of Minor GC. Usually a Full/Major Garbage Collection is performed in OldGen memory when it’s full.
 
Major GC takes longer as compared to Minor GC, as it checks for all live objects and all application threads are stopped until the operation completes, thus impacting the CPU usage and responsiveness of the application.
 
B. PermGen (Prior to Java 1.8): PermGen space is a contiguous extension of Heap space that was used to store class definitions and their metadata prior to Java 8. Starting from Java 8, this has been replaced with MetaSpace, which is a part of Non-Heap (Native) memory.
 
As we know, PermGen has been removed as of Java 8, so if you are running on Java 8 or beyond, feel free to skip this section.
 
The Initial and Maximum PermGen Size for a Java application is set by –XX:PermSize and –XX:MaxPermSize JVM flags. When total PermGen memory usage of the application hits the Maximum PermGen size set via the JVM flag –XX:MaxPermSize, the application throws the following error: java.lang.OutOfMemoryError: PermGen space
 
Unexpected growth of the PermGen or an OutOfMemoryError in this memory space meant that either the classes are not getting unloaded as expected, or the specified PermGen size is too small to fit all the loaded classes and their metadata.
 
PermGen was inefficient due to following reasons:
 
a. PermGen always has a fixed maximum size.
b. Comparatively inefficient Garbage collection. Frequent GC pauses and no concurrent de-allocation.
 
C. Stack: Stack memory is used for execution of a thread. They contain method specific values that are short-lived (i.e. local primitive variables) and references to other objects in the heap that are getting referred from the method. Stack memory is always referenced in LIFO (Last-In-First-Out) order, which makes it very fast due to simplicity of memory allocation. Whenever a method is invoked, a new block is created in the stack memory for the method to hold local primitive values and reference to other objects in the method. As soon as method ends, the block becomes unused and become available for next method, hence making it short-lived.
 
The Maximum Stack Size for a Java application is set by –Xss and JVM flag. When total Stack memory usage of the application hits the Maximum Stack size set via the JVM flag –Xss, the application throws the following error: java.lang.StackOverFlowError
 
D. Native Memory: Native Memory concept is not very important for Java 1.7 applications (as they rarely use native memory), but this part is very crucial for Java applications running with Java 1.8 or higher.
 
MetaSpace: As already discussed, there is no concept of PermGen space starting from Java 1.8, rather the applications running with Java 1.8 and higher versions use MetaSpace to store class definitions and their metadata. MetaSpace is a part of Native Memory.
 
As native memory is limited only by OS (unlike the maximum value for heap memory that can be specified as a JVM flag, there is no option to specify the maximum limit of native memory usage by the Java application), a leak in the classloader/classes has potential to bring down the entire server (not just the JVM).
 
As discussed above, the most important part of the Native Memory Usage of a Java application is the size of the MetaSpace (which is unlimited/limited by the host OS by default). In order to minimize the impact of a ClassLoader leak that might bring down the entire server, it is always recommended to set a Initial/Maximum Size for the MetaSpace by using the following JVM flags: –XX:MetaSpaceSize and –XX:MaxMetaSpaceSize.
 
MetaSpace can be divided into of distinct spaces in the Native memory, one space to store class definitions and another space to store their metadata. This can be achieved by enabling –XX:+UseCompressedOops and –XX:+UseCompressedClassesPointers (enabled by default if UseCompressedOops is turned on) JVM options.
 
When the OS cannot allocate further native memory to the Java application, it may report the following OutOfMemory Exceptions:
 
java.lang.OutOfMemoryError: Metaspace : This means that the OS cannot allocate further native memory to the Java application's MetaSpace, which indicates that either there is an acute shgortage of memory in the server or the Java Application has already hit the MaxMetaspaceSize limit.
 
java.lang.OutOfMemoryError: Compressed class space : This means that UseCompressedClassesPointers JVM option is enabled for the Java Application and it is hitting the default compressed class space size (1 GB). This can be tuned using the JVM flag: -XX:CompressedClassSpaceSize
 
java.lang.OutOfMemoryError: Out of swap space : This means that Swap Space is insufficient for the Java application
 
java.lang.OutOfMemoryError: unable to create new native Thread : This means that Process Memory is insufficient
 
Any of the above OutOfMemory exceptions signify an issue with Native/Non-Heap memory and you do not need to analyze JVM Heap/HeapDumps.
 
3. Garbage Collection Basics In Java:
 
As mentioned earlier, there are two different techniques for GC in Java. MinorGC, which happens to clean up YoungGen of the Heap, is very simple and straightforward. MajorGC, which happens to clean up OldGen of the Heap is very complex, time-consuming and affects the performance of the Java application.
 
Major/Full GC implements one of the following five mechanisms.
 
a. Serial GC (-XX:+UseSerialGC): Serial GC must not be used on an operating server. This GC type was created when there was only one CPU core on desktop computers. Using this serial GC will drop the application performance significantly.
 
This uses a mark-sweep-compaction algorithm. The first step of this algorithm is to mark the surviving objects in the OldGen. Then, it checks the heap from the front and leaves only the surviving ones behind (sweep). In the last step, it fills up the heap from the front with the objects so that the objects are piled up consecutively, and divides the heap into two parts: one with objects and one without objects (compact).
 
The serial GC is suitable for a small memory and a small number of CPU cores.
 
b. Parallel GC (-XX:+UseParallelGC): While the serial GC uses only one thread to process a GC, the parallel GC uses several threads to process a GC, and therefore, faster. This GC is useful when there is enough memory and a large number of cores. It is also called the "throughput GC."
 

SerialGC vs ParallelGC
 
c. Parallel Old GC (-XX:+UseParallelOldGC): This algorithm goes through three steps: mark-summary-compaction. The mark step just behaves exactly like serial/parallel GC. The summary step identifies the surviving objects separately for the areas that the GC have previously performed, and thus different from the sweep step of the mark-sweep-compact algorithm. It goes through a little more complicated steps.
 
d. CMS GC (-XX:+UseConcMarkSweepGC): Concurrent Mark-Sweep GC is much more complicated than any other GC types described above. The early initial mark step is simple. The surviving objects among the objects the closest to the classloader are searched. So, the pausing time is very short. In the concurrent mark step, the objects referenced by the surviving objects that have just been confirmed are tracked and checked. The difference of this step is that it proceeds while other threads are processed at the same time. In the remark step, the objects that were newly added or stopped being referenced in the concurrent mark step are checked. Lastly, in the concurrent sweep step, the garbage collection procedure takes place. The garbage collection is carried out while other threads are still being processed. Since this GC type is performed in this manner, the pausing time for GC is very short. The CMS GC is also called the low latency GC, and is used when the response time from all applications is crucial.
 

Serial GC vs CMS GC
 
While this GC type has the advantage of short stop-the-world time, it also has the following disadvantages.
 
1. It uses more memory and CPU than other GC types.
2. The compaction step is not provided by default.
 
You need to carefully review before using this type. Also, if the compaction task needs to be carried out because of the many memory fragments, the stop-the-world time can be longer than any other GC types. You need to check how often and how long the compaction task is carried out.
 
e. G1 GC (–XX:+UseG1GC): If you want to understand G1 GC, forget everything you know about the young generation and the old generation. As you can see in the picture, one object is allocated to each grid, and then a GC is executed. Then, once one area is full, the objects are allocated to another area, and then a GC is executed. The steps where the data moves from the three spaces of the young generation to the old generation cannot be found in this GC type. This type was created to replace the CMS GC, which has causes a lot of issues and complaints in the long term.
 
The biggest advantage of the G1 GC is its performance. It is faster than any other GC types that we have discussed so far.
 
GC Tuning: Refer to following articles on how to tune GC parameters for the JVM:
 
  https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/
http://www.cakesolutions.net/teamblogs/low-pause-gc-on-the-jvm
http://blog.takipi.com/java-performance-tuning-how-to-get-the-most-out-of-your-garbage-collector/
 
GC Logging: To log GC logs, start the application with the following JVM parameters along with the JVM option needed to enable the specific GC and specific parameters needed for the specific GC algorithm:
 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<logfile_name>

Debugging/Troubleshooting memory issues for Java applications by reviewing heapdumps, GC logs, jvmtop, jmap, jhat and jcmd tools.

All