RFR: 8253191: C2: Masked byte comparisons with large masks produce wrong result on x86

[Vladimir Ivanov]

`testUB_mem_imm` generates erroneous code when `mask` constant is larger than `Byte.MAX_VALUE`.

AD instruction in question:
instruct testUB_mem_imm(rFlagsReg cr, memory mem, immU8 imm, immI0 zero) %{
  match(Set cr (CmpI (AndI (LoadUB mem) imm) zero));

  ins_encode %{ __ testb($mem$$Address, $imm$$constant); %}

The following instruction sequence is problematic:
testb  $0x80,0x10(%rdi,%r9,1)
jle    0x00000001168789a0

It performs *signed* byte comparison and the immediate is interpreted as a negative value.

The original code shape was as follows:
movzbl 0x10(%rcx,%r9,1),%r9d
test   $0x80,%r9d
jle    0x000000010a9b6a00

The fix is to narrow the range of accepted mask constants and set the upper limit to `Byte.MAX_VALUE`.

Testing: hs-precheckin-comp, hs-tier1, hs-tier2.

-------------

Commit messages:
 - 8253191: C2: Masked byte comparisons with large masks produce wrong result

Changes: https://git.openjdk.java.net/jdk/pull/538/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=538&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8253191
  Stats: 78 lines in 2 files changed: 28 ins; 23 del; 27 mod
  Patch: https://git.openjdk.java.net/jdk/pull/538.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/538/head:pull/538

PR: https://git.openjdk.java.net/jdk/pull/538