ZGC Unable to reclaim memory for long time

[Sundara Mohan M]

Hi,
   I ran into this issue where ZGC is unable to reclaim memory for few
hours/days. It just keep printing "Exception in thread "RMI TCP
Connection(idle)" java.lang.OutOfMemoryError: Java heap space"  and
Allocation Stall happening on that thread.


Here is the metrics which shows for some reason even though there is
Garbage but it is unable to Reclaim

....
[2019-11-04T*08:39:53.986+0000*][1765465.981s][info][gc,heap     ]
GC(112126)      Live:         -              6366M (78%)        6366M (78%)
       6366M (78%)
    -                  -
*[2019-11-04T08:39:53.986+0000][1765465.981s][info][gc,heap     ]
GC(112126)   Garbage:         -              1735M (21%)        1735M (21%)
       1731M (21%)*
    -                  -
[2019-11-04T08:39:53.986+0000][1765465.981s][info][gc,heap     ] GC(112126)
Reclaimed:         -                  -                 0M (0%)
 4M (0%)
...

[2019-11-04T16:48:53.742+0000][1794805.738s][info][gc,heap     ] GC(135520)
     Live:         -              6367M (78%)        6367M (78%)
 6367M (78%)
    -                  -
*[2019-11-04T16:48:53.742+0000][1794805.738s][info][gc,heap     ]
GC(135520)   Garbage:         -              1730M (21%)        1730M (21%)
       1724M (21%)*
    -                  -
[2019-11-04T16:48:53.742+0000][1794805.738s][info][gc,heap     ] GC(135520)
Reclaimed:         -                  -                 0M (0%)
 6M (0%)

Here it was in this state for ~8hours and it is still happening. It says
has a Garbage of 21G but it is not able to Reclaim it everytime it reclaims
only 4-6M.

Any idea what might be the issue here.


TIA
Sundar