LSan Instrumentation

[Justin King]

Hi,

Internally we use sanitizers to great effect. A month or two ago I was
curious if properly instrumenting Hotspot to play nice with sanitizers
would be useful, so I started with LSan as it is the least invasive,
https://github.com/jcking/jdk/tree/lsan. Currently when Hotspot is
configured with --enable-asan it needs to be run with detect_leaks=0 to
disable LSan as well, or it will complain about many false positive leaks.
This is due to LSan only looking for pointers in the malloc heap by
default, while Hotspot/Java frequently stores pointers to malloc heap
memory in the collected heap, metaspace, and sometimes the code cache as
immediate values in generated machine code. Using the linked approach I
have so far identified https://bugs.openjdk.org/browse/JDK-8297309 and
https://bugs.openjdk.org/browse/JDK-8297911, which indicates that it is
indeed useful as a defense-in-depth mechanism.

I am writing to see if anybody has any
reservations/questions/comments/concerns about upstreaming some variation
of https://github.com/jcking/jdk/tree/lsan, so that it can be used by the
wider community either as part of --enable-asan or standalone as
--enable-lsan. My suggestion would be to keep LSan and ASan separate and
always disable leak checking in ASan. That way ASan can be used on paths
that require UseCompressedOops, UseCompressedClassPointers,
UseSharedSpaces, or RequireSharedSpaces (see below).

*Implementation Details*

   - In order for LSan to work, UseCompressedOops
   and UseCompressedClassPointers must be set to false. LSan needs pointers to
   be property aligned and sized. The former two options break this
   expectation. Additionally we were unable to get LSan to play nicely with
   CDS, so we also disabled CDS by setting UseSharedSpaces and
   RequireSharedSpaces to false. It might be possible to get it to play nice
   with CDS in the future though, it needs more investigation.
   - Leak checking is performed early during normal JVM shutdown within
   before_exit in src/hotspot/share/runtime/java.cpp, as we are not super
   interested in leaks caused as a result of JVM termination. Leak checking is
   skipped if the JVM is being halted or there is an active JVM error.
   - Leak checking is never performed on program exit, this option is set
   by default in the launcher by exporting __lsan_default_options.
   - There are some probably benign leaks that are explicitly ignored by
   using __lsan_ignore_object.
   - We have to ignore any pointers to malloc heap memory that are embedded
   as immediates in generated machine code using __lsan_ignore_object. LSan is
   not able to find these as they are either encoded or not property aligned.
   - We inform LSan to look for pointers to malloc heap memory by
   registering memory regions that make up the metaspace, collected heap, and
   the code cache using __lsan_register_root_region and
   __lsan_unregister_root_region.
   - Some tests exercise UseCompressedOops, UseCompressedClassPointers,
   UseSharedSpaces, or RequireSharedSpaces. When LSan is enabled these tests
   will be skipped.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.openjdk.org/pipermail/hotspot-runtime-dev/attachments/20221202/942ef86d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3999 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://mail.openjdk.org/pipermail/hotspot-runtime-dev/attachments/20221202/942ef86d/smime.p7s>