RFR: 8338197: ubsan: ad_x86.hpp:6417:11: runtime error: shift exponent 100 is too large for 32-bit type 'unsigned int'

[Vladimir Kozlov]

On Fri, 22 Aug 2025 00:47:48 GMT, Boris Ulasevich <bulasevich at openjdk.org> wrote:

> This reworks the recent update https://github.com/openjdk/jdk/pull/24696 to fix a UBSan issue on aarch64. The problem now reproduces on x86_64 as well, which suggests the previous update was not optimal.
> 
> The issue reproduces with a HeapByteBufferTest jtreg test on a UBSan-enabled build. Actually the trigger is `XX:+OptoScheduling` option used by test (by default OptoScheduling is disabled on most x86 CPUs). With the option enabled, the failure can be reproduced with a simple `java -version` run.
> 
> This fix is in ADLC-generated code. For simplicity, the examples below show the generated fragments.
> 
> The problems is that shift count `n` may be too large here:
> 
> class Pipeline_Use_Cycle_Mask {
> protected:
>   uint _mask;
>   ..
>   Pipeline_Use_Cycle_Mask& operator<<=(int n) {
>     _mask <<= n;
>     return *this;
>   }
> };
> 
> The recent change attempted to cap the shift amount at one call site:
> 
> class Pipeline_Use_Element {
> protected:
>   ..
>   // Mask of specific used cycles
>   Pipeline_Use_Cycle_Mask _mask;
>   ..
>   void step(uint cycles) {
>     _used = 0;
>     uint max_shift = 8 * sizeof(_mask) - 1;
>     _mask <<= (cycles < max_shift) ? cycles : max_shift;
>   }
> }
> 
> However, there is another site where `Pipeline_Use_Cycle_Mask::operator<<=` can be called with a too-large shift count:
> 
> // The following two routines assume that the root Pipeline_Use entity
> // consists of exactly 1 element for each functional unit
> // start is relative to the current cycle; used for latency-based info
> uint Pipeline_Use::full_latency(uint delay, const Pipeline_Use &pred) const {
>   for (uint i = 0; i < pred._count; i++) {
>     const Pipeline_Use_Element *predUse = pred.element(i);
>     if (predUse->_multiple) {
>       uint min_delay = 7;
>       // Multiple possible functional units, choose first unused one
>       for (uint j = predUse->_lb; j <= predUse->_ub; j++) {
>         const Pipeline_Use_Element *currUse = element(j);
>         uint curr_delay = delay;
>         if (predUse->_used & currUse->_used) {
>           Pipeline_Use_Cycle_Mask x = predUse->_mask;
>           Pipeline_Use_Cycle_Mask y = currUse->_mask;
> 
>           for ( y <<= curr_delay; x.overlaps(y); curr_delay++ )
>             y <<= 1;
>         }
>         if (min_delay > curr_delay)
>           min_delay = curr_delay;
>       }
>       if (delay < min_delay)
>       delay = min_delay;
>     }
>     else {
>       for (uint j = predUse->_lb; j <= predUse->_ub; j++) {
>         const Pipeline_Use_Element *currUse = element(j);
>         if (predUse->_used & currUse->_used) {
>  ...

src/hotspot/share/adlc/output_h.cpp line 774:

> 772:     fprintf(fp_hpp, "  Pipeline_Use_Cycle_Mask& operator<<=(int n) {\n");
> 773:     fprintf(fp_hpp, "    int max_shift = 8 * sizeof(_mask) - 1;\n");
> 774:     fprintf(fp_hpp, "    _mask <<= (n < max_shift) ? n : max_shift;\n");

sizeof(_mask) is know - it is sizeof(uint).
Lines 760-768 should be cleaned: ` <= 32` checks are redundant because of check at line 758. This is leftover from SPARC code (not clean) removal.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/26890#discussion_r2294201108