RFR: 8376855: ASAN reports out-of-range read in strncmp in MethodHandles::is_basic_type_signature

[Afshin Zafari]

On Mon, 2 Feb 2026 01:13:35 GMT, David Holmes <dholmes at openjdk.org> wrote:

> An ASAN enabled build reported heap-buffer-overflow in `MethodHandles::is_basic_type_signature` with `ASAN_OPTIONS=strict_string_checks=true` when running test `jdk/jdk/jfr/api/metadata/annotations/TestThrottle.java`
> 
> The code is here:
> 
> bool MethodHandles::is_basic_type_signature(Symbol* sig) {
>   assert(vmSymbols::object_signature()->utf8_length() == (int)OBJ_SIG_LEN, "");
>   assert(vmSymbols::object_signature()->equals(OBJ_SIG), "");
>   for (SignatureStream ss(sig, sig->starts_with(JVM_SIGNATURE_FUNC)); !ss.is_done(); ss.next()) {
>     switch (ss.type()) {
>     case T_OBJECT:
>       // only java/lang/Object is valid here
>       if (strncmp((char*) ss.raw_bytes(), OBJ_SIG, OBJ_SIG_LEN) != 0)
> 
> The ASAN `strncmp` interceptor acts as follows:
> 
> INTERCEPTOR(int, strncmp, const char *s1, const char *s2, size_t n) {
>   void *ctx;
>   ASAN_INTERCEPTOR_ENTER(linker, strncmp); // Sets up context
>   ASAN_READ_RANGE(s1, n); // Validates s1
>   ASAN_READ_RANGE(s2, n); // Validates s2
>   return REAL(strncmp)(s1, s2, n); // Calls original function
> }
> 
> With the test given `s1` is a buffer of size 15, containing a non-nul-terminated string, and `n` is 18, so `ASAN_READ_RANGE` fails for `s1` as we could potentially read beyond the end of the buffer. In practice however, given `s1` is guaranteed to be a valid type-string from a signature symbol of type `T_OBJECT`, its final character is `;` and the final character of `s2` is also `;` (it is the string constant `Ljava/lang/Object;`). Hence the comparison must terminate before we can run off the end of `s1`.
> 
> To appease ASAN we can make a simple change to the `strncmp` call to compare at most `ss.raw_length()` bytes. 
> 
> Testing 
>  - ASAN no longer reports an error
>  - tiers 1-3 sanity
> 
> Thanks

Thank you David for taking and fixing this.

-------------

Marked as reviewed by azafari (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/29516#pullrequestreview-3738832412