RFR: 8287052: comparing double to max_intx gives unexpected results

[Dean Long]

It isn't safe to compare a double to 2^63-1 because the latter will be changed to 2^63 when converting to a double.
This fixes a compiler warning with clang-12 and prevents the code from returning a negative value in some cases (observed on x64 with g++).

-------------

Commit messages:
 - better overflow check

Changes: https://git.openjdk.java.net/jdk/pull/8798/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=8798&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8287052
  Stats: 9 lines in 1 file changed: 6 ins; 2 del; 1 mod
  Patch: https://git.openjdk.java.net/jdk/pull/8798.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/8798/head:pull/8798

PR: https://git.openjdk.java.net/jdk/pull/8798