RFR: 8338197: ubsan: ad_x86.hpp:6417:11: runtime error: shift exponent 100 is too large for 32-bit type 'unsigned int'

[Boris Ulasevich]

This reworks the recent update https://github.com/openjdk/jdk/pull/24696 to fix a UBSan issue on aarch64. The problem now reproduces on x86_64 as well, which suggests the previous update was not optimal.

The issue reproduces with a HeapByteBufferTest jtreg test on a UBSan-enabled build. Actually the trigger is `XX:+OptoScheduling` option used by test (by default OptoScheduling is disabled on most x86 CPUs). With the option enabled, the failure can be reproduced with a simple `java -version` run.

This fix is in ADLC-generated code. For simplicity, the examples below show the generated fragments.

The problems is that shift count `n` may be too large here:

class Pipeline_Use_Cycle_Mask {
protected:
  uint _mask;
  ..
  Pipeline_Use_Cycle_Mask& operator<<=(int n) {
    _mask <<= n;
    return *this;
  }
};

The recent change attempted to cap the shift amount at one call site:

class Pipeline_Use_Element {
protected:
  ..
  // Mask of specific used cycles
  Pipeline_Use_Cycle_Mask _mask;
  ..
  void step(uint cycles) {
    _used = 0;
    uint max_shift = 8 * sizeof(_mask) - 1;
    _mask <<= (cycles < max_shift) ? cycles : max_shift;
  }
}

However, there is another site where `Pipeline_Use_Cycle_Mask::operator<<=` can be called with a too-large shift count:

// The following two routines assume that the root Pipeline_Use entity
// consists of exactly 1 element for each functional unit
// start is relative to the current cycle; used for latency-based info
uint Pipeline_Use::full_latency(uint delay, const Pipeline_Use &pred) const {
  for (uint i = 0; i < pred._count; i++) {
    const Pipeline_Use_Element *predUse = pred.element(i);
    if (predUse->_multiple) {
      uint min_delay = 7;
      // Multiple possible functional units, choose first unused one
      for (uint j = predUse->_lb; j <= predUse->_ub; j++) {
        const Pipeline_Use_Element *currUse = element(j);
        uint curr_delay = delay;
        if (predUse->_used & currUse->_used) {
          Pipeline_Use_Cycle_Mask x = predUse->_mask;
          Pipeline_Use_Cycle_Mask y = currUse->_mask;

          for ( y <<= curr_delay; x.overlaps(y); curr_delay++ )
            y <<= 1;
        }
        if (min_delay > curr_delay)
          min_delay = curr_delay;
      }
      if (delay < min_delay)
      delay = min_delay;
    }
    else {
      for (uint j = predUse->_lb; j <= predUse->_ub; j++) {
        const Pipeline_Use_Element *currUse = element(j);
        if (predUse->_used & currUse->_used) {
          Pipeline_Use_Cycle_Mask x = predUse->_mask;
          Pipeline_Use_Cycle_Mask y = currUse->_mask;

>        for ( y <<= delay; x.overlaps(y); delay++ )
            y <<= 1;
        }
      }
    }
  }

  return (delay);
}

**Fix:** cap the shift **inside** `Pipeline_Use_Cycle_Mask::operator<<=` so all call sites are safe:

class Pipeline_Use_Cycle_Mask {
protected:
  uint _mask;
  ..
  Pipeline_Use_Cycle_Mask& operator<<=(int n) {
    int max_shift = 8 * sizeof(_mask) - 1;
    _mask <<= (n < max_shift) ? n : max_shift;
    return *this;
  }
};

class Pipeline_Use_Element {
protected:
  ..
  // Mask of specific used cycles
  Pipeline_Use_Cycle_Mask _mask;
  ..
  void step(uint cycles) {
    _used = 0;
    _mask <<= cycles;
  }
}


Note: on platforms where PipelineForm::_maxcycleused > 32 (e.g., ARM32), the Pipeline_Use_Cycle_Mask implementation already handles large shifts, so no additional check is needed:

class Pipeline_Use_Cycle_Mask {
protected:
  uint _mask1, _mask2, _mask3;

  Pipeline_Use_Cycle_Mask& operator<<=(int n) {
    if (n >= 32)
      do {
        _mask3 = _mask2; _mask2 = _mask1; _mask1 = 0;
      } while ((n -= 32) >= 32);

    if (n > 0) {
      uint m = 32 - n;
      uint mask = (1 << n) - 1;
      uint temp2 = mask & (_mask1 >> m); _mask1 <<= n;
      uint temp3 = mask & (_mask2 >> m); _mask2 <<= n; _mask2 |= temp2;
      _mask3 <<= n; _mask3 |= temp3;
    }
    return *this;
  }
}

-------------

Commit messages:
 - 8338197: ubsan: ad_x86.hpp:6417:11: runtime error: shift exponent 100 is too large for 32-bit type 'unsigned int'

Changes: https://git.openjdk.org/jdk/pull/26890/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=26890&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8338197
  Stats: 4 lines in 1 file changed: 1 ins; 1 del; 2 mod
  Patch: https://git.openjdk.org/jdk/pull/26890.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/26890/head:pull/26890

PR: https://git.openjdk.org/jdk/pull/26890