Konstantin,
I'd like to get an answer to my previous question: "How reliable the test is if it ignores NoSuchMethodException & VirtualMachineError? Are there other manifestations of the problem?"
Most notably, have you seen VM crashes w/ -Xcomp running that test? I have seen no crashes with -Xcomp in our tests results base, only NoSuchMethodException and VirtualMachineError. Good. Thanks for the info.
This failure does not occur with JDK 9 on -Xcomp, I used the same random seeds (-Dseed), and the problem exists only with JDK 8u, not JDK 9.
Because there is no failure with JDK 9 I can suppose there could be a product issue in JDK 8u. That's strange. There should be no difference in MethodHandle stub management between 8u & 9 now (once allocated they stay forever). 9 has segmented code cache feature and it can lead to observable differences in behavior.
How many iterations does the test perform on 8u & 9? What if you increase the number? Also, I'd suggest you to monitor code cache usage during the run and when the test finishes. Best regards, Vladimir Ivanov