RFR: 8356865: C2: Unreasonable values for debug flag FastAllocateSizeLimit can lead to left-shift-overflow, which is UB

[Evgeny Astigeevich]

On Mon, 16 Jun 2025 14:50:46 GMT, Benoît Maillard <bmaillard at openjdk.org> wrote:

> This PR adds a range constraint for the `-XX:FastAllocateSizeLimit` debug flag. This prevents undefined behavior caused by left-shift overflow of the flag value in `GraphKit::new_array`.
> 
> ### Testing
> - [x] [GitHub Actions](https://github.com/benoitmaillard/jdk/actions?query=branch%3AJDK-8356865)
> - [x] tier1-3, plus some internal testing
> - [x] Manual testing with values known to previously cause undefined behavior
> 
> Thanks!

src/hotspot/share/opto/graphKit.cpp line 3807:

> 3805:     int log2_esize = Klass::layout_helper_log2_element_size(layout_con);
> 3806:     fast_size_limit <<= (LogBytesPerLong - log2_esize);
> 3807:     assert (fast_size_limit > 0, "increasing the size limit should not produce negative values");

Prior C++14 left shit producing a negative value is undefined behavior: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2161.pdf

Do we compile c++ source specifying the C++ standard?

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/25834#discussion_r2152715500