Garbage Collection: G1GC, ZGC, ShenandoahGC - Tuning & Choice

mhoffmann

I am currently working on a well utilised engine which currently has ~ 49 GB of heap space at maximum, we're planning to increase this to 110 GB since we have the resources available.

Sadly, the engine guide (10 LTS) does not state anything about the GC used nor does it recommend to set the heap to a constant size which is recommended for enterprise applications (IIRC) to avoid additional overhead from shrinking / expanding the committed heap memory.

So I took a look at the 12 LTS doc to find nothing here as well.

The main reason why I even bother with the GC is that I suspect it to be partly responsible for bad performance on a 500+ CCU engine.

We currently have lines like:

[2025-09-13T06:25:01.903+0200][15590.520s][info][gc] GC(597) Pause Young (Concurrent Start) (G1 Evacuation Pause) 7207M->4582M(25280M) 5867.931ms

[2025-09-13T06:27:12.366+0200][15720.982s][info][gc] GC(628) Pause Young (Normal) (G1 Evacuation Pause) 6280M->5072M(24576M) 6274.184ms

[2025-09-16T15:30:20.618+0200][307509.512s][info][gc] GC(25650) Pause Young (Normal) (G1 Evacuation Pause) 3602M->3265M(25952M) 5690.326ms

which indicate issues with the GC's pause length, the default for MaxGCPauseMillis is 200 indicating that the GC has trouble meeting this soft goal.

Are there any recommendations for the GC tuning in Axon Ivy which you could share with us, especially for medium - big installations?

I'd currently go with the following settings for G1GC (160 GB RAM, 16 Cores):

-Xms110g
-Xmx110g
 
XX:G1HeapRegionSize=16m
XX:G1ReservePercent=15
XX:InitiatingHeapOccupancyPercent=30
XX:ParallelGCThreads=16
XX:ConcGCThreads=4

Reto Weiss

Hi mhoffmann,

The recommended GC settings can be found in the configuration/jvm.options file. We use the G1 garbage collector, and the most important settings are -Xmx and -Xms.

-Xmxspecifies the maximum memory the JVM can use, while -Xms defines the initial memory allocated by the JVM.
The G1 GC includes an auto-tuning feature, so it is generally recommended to avoid adjusting other settings unless your application has specific requirements.
Increasing -Xmx or -Xms does not necessarily reduce pause times. In fact, it often has the opposite effect because a larger heap means the GC has more memory to clean. If your application runs smoothly with 49 GB, I would advise against increasing -Xms and -Xmx.

For very large heaps (greater than 100 GB), you might consider using ZGC (-XX:+UseZGC instead of -XX:+UseG1GC). ZGC is known for its extremely low pause times but requires more CPU resources to run alongside your application.

To debug GC activity, you can use the JVM flag -verbose:gc -Xlog:gc*:file=logs/gc.log. Several tools are available to parse the generated log files and visualize GC activity. For more information on GC performance tuning, refer to Oracle's GC Tuning Guide.

Before tuning your GC, I recommend analyzing why the GC is doing so much work. Investigate the types of objects your application is allocating that require frequent cleanup. Axon Ivy provides several tools to assist with this analysis:

Axon Ivy Engine Cockpit: Check GC times in the Monitor > Java > Memory view. This is a good starting point to identify long pause times. If issues are detected, enable GC logging.
Java Flight Recorder: Use the Monitor > Java > Flight Recorder in the Engine Cockpit alongside Java Mission Control to analyze GC behavior and object allocations.
Class Histogram: Access the Monitor > Java > Class Histogram in the Engine Cockpit to identify classes with the most objects on the heap. Take a heap dump and use tools like Eclipse Memory Analyzer to inspect all objects in the heap.

For more information about trouble shooting memory problems, refer to Troubleshoot Memory Problems

mhoffmann

Hi Reto Weiss,

I fully agree with you that just increasing the heap size will most likely make the issue worse, that's why I also took a look at the additional tuning parameters and the lecture in university that if you have to touch the GC you're doing something wrong still sticks to my head.

However, I already did some research regarding the memory usage, and I guess it might be related to a dozen byte[] instances. So it's really just the load the engine has to perform because with Axon Ivy, we and other companies are currently using some monolithic approach to use the engine for every part of the application.

This is something which is on my list for quite some time, because I think especially projects which use the DocFactory (namely Aspose) a lot, shouldn't bear this load on the Engine itself and I personally would prefer a microservice approach here which would be better scalable. This would probably remove a lot of work from the engine itself and I fear that in the future we'll have to go down this path regardless and move the rendering of these files to a dedicated microservice just to keep the response times and load on the engine managable.

But that's the future, for the present I'l try if I can convince the engine maintainer to move to ZGC to give it a try. I did some performance testing yesterday and on our internal engine, it felt a bit worse but the VM is much weaker than the system we're actually talking about.

I'll report back as soon as we've evaluated this, maybe the findings will help others.

mhoffmann

mhoffmann
After doing some more research about the GC options available, I think that ZGC might be the best choice even for smaller heaps for us.

G1GC works a lot with predictability - so if the load is steady on the engine, it works like intended and pretty well and I wouldn't advice anyone to change, unless your heap is bigger or your load is pretty unstable. However, if the pause times are not an issue and in range (shouldn't exceed 200 ms or 500 ms if generous), most likely your GC or its tuning is not contributing (much) to performance issues.

ZGC with its concurrent approach seems to work way better for load peaks and generally a highly fluctuating load curve on the engine itself - I however strongly advice to whitelist Axon Ivy's service and JVM in your virus scan solution before you use it because the magic pointers can be detected as false positive due to their unusual but intended behaviour - I'd everyone recommend to read the official deep dive.

Oracle also states the following for ZGC:

ZGC's low latency and ability to scale to handle large live sets would make it a good candidate for web applications. Web applications frequently service many requests simultaneously, often resulting in maintaining larger live sets. Consistent responsiveness also helps with system stability. This pairs well with ZGC's strengths.

Source: https://dev.java/learn/jvm/tool/garbage-collection/zgc-overview/#when-to-use-zgc-and-when-to-avoid-zgc

Since it's somewhat hard to mimic proper user behaviour (different workflows are more/less utilised at certain times) for load testing, I can currently hardly say anything about the CPU impact. Since we have to get rid of the long pause times however, it's worth a shot for me. I think that improving our own process architecture and a more tailored GC for "web workloads" will help the engine rather than hurt it. I'll report back with further findings after we switched the GC on the prod engine.

But given that we are not in a resource restricted environment, we could also adjust the core count if needed. So if you ran into long pause times with your GC and you have multiple cores available which are mostly idle, ZGC might fix these times and improve the performance of your process engine.