RFR: 8300207: Add a pre-check for the number of canonical equivalent permutations in j.u.r.Pattern

[Raffaello Giulietti]

- Strengthen a computation that could overflow.
- Specify that use of CANON_EQ could exhaust memory in the compilation phase.

-------------

Commit messages:
 - 8300207: Add a pre-check for the number of canonical equivalent permutations in j.u.r.Pattern

Changes: https://git.openjdk.org/jdk/pull/12027/files
 Webrev: https://webrevs.openjdk.org/?repo=jdk&pr=12027&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8300207
  Stats: 30 lines in 1 file changed: 24 ins; 0 del; 6 mod
  Patch: https://git.openjdk.org/jdk/pull/12027.diff
  Fetch: git fetch https://git.openjdk.org/jdk pull/12027/head:pull/12027

PR: https://git.openjdk.org/jdk/pull/12027