RFR: 8376855: ASAN reports out-of-range read in strncmp in MethodHandles::is_basic_type_signature

[David Holmes]

On Mon, 2 Feb 2026 11:26:01 GMT, Afshin Zafari <azafari at openjdk.org> wrote:

>> An ASAN enabled build reported heap-buffer-overflow in `MethodHandles::is_basic_type_signature` with `ASAN_OPTIONS=strict_string_checks=true` when running test `jdk/jdk/jfr/api/metadata/annotations/TestThrottle.java`
>> 
>> The code is here:
>> 
>> bool MethodHandles::is_basic_type_signature(Symbol* sig) {
>>   assert(vmSymbols::object_signature()->utf8_length() == (int)OBJ_SIG_LEN, "");
>>   assert(vmSymbols::object_signature()->equals(OBJ_SIG), "");
>>   for (SignatureStream ss(sig, sig->starts_with(JVM_SIGNATURE_FUNC)); !ss.is_done(); ss.next()) {
>>     switch (ss.type()) {
>>     case T_OBJECT:
>>       // only java/lang/Object is valid here
>>       if (strncmp((char*) ss.raw_bytes(), OBJ_SIG, OBJ_SIG_LEN) != 0)
>> 
>> The ASAN `strncmp` interceptor acts as follows:
>> 
>> INTERCEPTOR(int, strncmp, const char *s1, const char *s2, size_t n) {
>>   void *ctx;
>>   ASAN_INTERCEPTOR_ENTER(linker, strncmp); // Sets up context
>>   ASAN_READ_RANGE(s1, n); // Validates s1
>>   ASAN_READ_RANGE(s2, n); // Validates s2
>>   return REAL(strncmp)(s1, s2, n); // Calls original function
>> }
>> 
>> With the test given `s1` is a buffer of size 15, containing a non-nul-terminated string, and `n` is 18, so `ASAN_READ_RANGE` fails for `s1` as we could potentially read beyond the end of the buffer. In practice however, given `s1` is guaranteed to be a valid type-string from a signature symbol of type `T_OBJECT`, its final character is `;` and the final character of `s2` is also `;` (it is the string constant `Ljava/lang/Object;`). Hence the comparison must terminate before we can run off the end of `s1`.
>> 
>> To appease ASAN we can make a simple change to the `strncmp` call to compare at most `ss.raw_length()` bytes. 
>> 
>> Testing 
>>  - ASAN no longer reports an error
>>  - tiers 1-3 sanity
>> 
>> Thanks
>
> Thank you David for taking and fixing this.

Thanks for the reviews @afshin-zafari  and @jdksjolen !

-------------

PR Comment: https://git.openjdk.org/jdk/pull/29516#issuecomment-3837656354