RFR: 8335237: ubsan: vtableStubs.hpp is_vtable_stub exclude from ubsan checks

[Axel Boldt-Christmas]

On Thu, 27 Jun 2024 09:29:46 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> [JDK-8331725](https://bugs.openjdk.org/browse/JDK-8331725) shows an ubsan (undefined behavior) issue in vtableStubs.hpp is_vtable_stub() .
> 
> /open_jdk/jdk_test/jdk/src/hotspot/share/code/vtableStubs.hpp:176:60: runtime error: load of value 204, which is not a valid value for type 'bool'
>     #0 0x110a6ad7e in VtableStubs::entry_point(unsigned char*) vtableStubs.cpp:280
>     #1 0x10f4cc8e6 in CompiledIC::is_megamorphic() const compiledIC.cpp:293
>     #2 0x10f4cc95d in CompiledIC::update(CallInfo*, Klass*) compiledIC.cpp:268
>     #3 0x110592eed in SharedRuntime::resolve_helper(bool, bool, JavaThread*) sharedRuntime.cpp:1366
>     #4 0x11058c0b3 in SharedRuntime::resolve_virtual_call_C(JavaThread*) sharedRuntime.cpp:1514
>     #5 0x12cd2e55a (<unknown module>)
>     #6 0x12580e03b (<unknown module>)
>     #7 0x12cc1f321 (<unknown module>)
>     #8 0x12cc1f321 (<unknown module>)
> 
> From the comments of the issue it seems while the coding could be improved, the problem is not very severe ('The reason bad bool values are seen is because there is no VtableStub object at that location. The reason this works is that we use the data at this location to generate a hash code, do a hash table lookup and then check the actual address for equality. Generating a bogus hash is harmless, ... ')
> so as long as nothing better was found, we can exclude the method from ubsan checking.

It is only harmless from the point of view that we cannot regardless of hash generated find an entry which have its entrypoint at the location we are checking if it is not a vtable_stub. 

However we have no guarantees what the compiler might do in the presence of UB. So far it has not been a problem.

[JDK-8331725](https://bugs.openjdk.org/browse/JDK-8331725) has not been a priority for me. But I can get my fix out in a couple of days. The issue with fixing this is that C++ does not define an object representation for `false`. So either it has to rely on implementation specific behaviour (that false is all 0 bits in memory) or we have to create a boolean like enum class which is used and specifies the actual values for `true` and `false`.

I suspect we have assumptions elsewhere in the code base that `false` is represented by 0x0. It is a lot easier to work with Java's jni boolean in this regard.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/19925#issuecomment-2194422094