[jdk21u-dev] RFR: 8325083: jdk/incubator/vector/Double512VectorTests.java crashes in Assembler::vex_prefix_and_encode

[Martin Doerr]

Clean backport of [JDK-8325083](https://bugs.openjdk.org/browse/JDK-8325083).

-------------

Commit messages:
 - Backport 01060ad4ab18581aa46bc16e64c7f12a591a682b

Changes: https://git.openjdk.org/jdk21u-dev/pull/763/files
  Webrev: https://webrevs.openjdk.org/?repo=jdk21u-dev&pr=763&range=00
  Issue: https://bugs.openjdk.org/browse/JDK-8325083
  Stats: 14 lines in 1 file changed: 12 ins; 2 del; 0 mod
  Patch: https://git.openjdk.org/jdk21u-dev/pull/763.diff
  Fetch: git fetch https://git.openjdk.org/jdk21u-dev.git pull/763/head:pull/763

PR: https://git.openjdk.org/jdk21u-dev/pull/763