SoftReference incorrect javadoc?

[Michael Pollmeier]

Thanks Per, your detailed insights make sense.

On 17/04/19 10:22 pm, Per Liden wrote:
> Hi Michael,
> 
> On 04/17/2019 03:59 AM, Michael Pollmeier wrote:
>> Hi Per,
>>
>> While testing different JVMs I realized that it's fixed in openjdk 11,
>> e.g. openjdk version "11.0.1" 2018-10-16 LTS (zulu build), maybe by this
>> commit: https://hg.openjdk.java.net/jdk/jdk/rev/6464882498b5
> 
> That bug only affected ZGC (which was introduced in 11), so it didn't
> change the behavior of any other GC.
> 
>> That's great to know, but is it worth updating the javadocs of older
>> versions?
>>
>> To reproduce it I attached a simple SoftRefTest.java to easily reproduce
>> it. It allocates (only) softly referenced objects that occupy some heap,
>> occasionally printing counts (instantiated, finalized, free heap).
> 
> Thanks for the test. The problem with the test is that you have a
> finalize() function on Instance. This means the object will be kept
> around until it has been finalized. So even if the SoftReference
> referring to it is cleared, the memory will not be freed until the
> object also has been finalized. In your test, you're simply creating too
> many objects, and the Finalizer thread is simply unable to keep up
> finalizing them at the same pace as they are created, so the heap fills up.
> 
> I adjusted your test a bit, and removed the use of finalize(). You can
> run this test forever without running into OOMEs. By having the same
> thread that creates the objects also poll the reference queue, the test
> will automatically be throttled. If SoftReferences weren't cleared as
> they should, this test would OOME pretty fast.
> 
> import java.lang.ref.Reference;
> import java.lang.ref.ReferenceQueue;
> import java.lang.ref.SoftReference;
> import java.util.HashMap;
> 
> public class SoftRefTest2 {
>   public static void main(String[] args) throws Exception {
>     final ReferenceQueue<Instance> queue = new ReferenceQueue<>();
>     final HashMap<Object, SoftReference<Instance>> instances = new
> HashMap<>();
>     long createdCount = 0;
>     long clearedCount = 0;
> 
>     for (;;) {
>       SoftReference<Instance> softref = new SoftReference<Instance>(new
> Instance(), queue);
>       instances.put(softref, softref);
>       if (++createdCount % 100000 == 0) {
>         System.out.println(createdCount + " instances created; free=" +
> Runtime.getRuntime().freeMemory() / 1024 / 1024 + "M");
>       }
> 
>       // Drain queue
>       Reference<?> ref;
>       while ((ref = queue.poll()) != null) {
>           instances.remove(ref);
>           if (++clearedCount % 100000 == 0) {
>               System.out.println(clearedCount + " instances cleared");
>           }
>       }
>     }
>   }
> 
>   static class Instance {
>     static int finalizedCount = 0;
>     String[] occupySomeHeap = new String[50];
>   }
> }
> 
> 
> cheers,
> Per
> 
>>
>> Usage:
>> javac SoftRefTest.java
>> java -Xms256m -Xmx256m SoftRefTest
>>
>> Output:
>> 100000 instances created; free=212M
>> 200000 instances created; free=181M
>> 300000 instances created; free=152M
>> 400000 instances created; free=121M
>> 500000 instances created; free=93M
>> 600000 instances created; free=61M
>> 700000 instances created; free=33M
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>          at Instance.<init>(SoftRefTest.java:27)
>>          at SoftRefTest.main(SoftRefTest.java:12)
>>
>> Interpretation:
>> It doesn't free any of the only softly referenced objects, resulting in
>> an OutOfMemoryError, contradicting the 'guarantee' in the javadoc.
>>
>> Workaround: if you additionally configure
>> `-XX:SoftRefLRUPolicyMSPerMB=0`, it finalizes them and doesn't run out
>> of memory.
>>
>> Tested with:
>> openjdk version "1.8.0_212"
>> java version "1.8.0_144" (oracle)
>> openjdk version "9.0.4.1" (zulu build)
>> openjdk version "10.0.2" 2018-07-17  (zulu build)
>>
>> Cheers
>> Michael
>>
>>
>> On 16/04/19 6:27 pm, Per Liden wrote:
>>> Hi Michael,
>>>
>>> On 4/16/19 4:19 AM, David Holmes wrote:
>>>> Hi Michael,
>>>>
>>>> Re-directing to core-libs-dev and hotspot-gc-dev.
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>> On 16/04/2019 12:14 pm, Michael Pollmeier wrote:
>>>>> Quoting
>>>>> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ref/SoftReference.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> All soft references to softly-reachable objects are guaranteed to
>>>>>> have
>>>>> been cleared before the virtual machine throws an OutOfMemoryError
>>>>>
>>>>> That statement was true when soft references were first introduced in
>>>>> java 1.2, but from java 1.3.1 the jvm property
>>>>> `-XX:SoftRefLRUPolicyMSPerMB` was introduced.
>>>
>>> That statement is still true. When the GC gets into a situation where it
>>> is having trouble satisfying an allocation, then SoftRefLRUPolicyMSPerMB
>>> will be ignored and all soft references will be cleared, before the GC
>>> gives up and throws an OOME.
>>>
>>>>> It defaults to 1000 (milliseconds), meaning that if there’s only 10MB
>>>>> available heap, the garbage collector will free references that have
>>>>> been used more than 10s ago. I.e. everything else (including young
>>>>> softly reachable objects) will *not* be freed, leading to an
>>>>> OutOfMemoryError, contradicting the above quoted 'guarantee'.
>>>>>
>>>>> That's also the behaviour I observed on various JREs. Would you agree,
>>>>> i.e. should I propose an updated doc?
>>>
>>> Could you elaborate on what kind of test you did to come to this
>>> conclusion? Preferably provide a re-producer. In OOME situations, if a
>>> SoftReference isn't cleared, it is typically because you have
>>> unknowingly made it strongly reachable.
>>>
>>> cheers,
>>> Per
>>>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://mail.openjdk.org/pipermail/hotspot-gc-dev/attachments/20190418/9b06fef2/signature.asc>