Request for reviews (S): 6973308: Missing zero length check before repne scas in check_klass_subtype_slow_path()

[Vladimir Kozlov]

I updated webrev with additional changes in assembler_x86.cpp
to verify r12_heapbase even in zero based case.
These changes exposed this bug.

Thanks,
Vladimir

Vladimir Kozlov wrote:
> http://cr.openjdk.java.net/~kvn/6973308/webrev
> 
> Fixed 6973308: Missing zero length check before repne scas in 
> check_klass_subtype_slow_path()
> 
> Z flag value will not be set by repne_scan(); if RCX == 0 since
> 'repne' instruction does not change flags (only scas instruction
> which is repeated sets flags).
> 
> Setting Z = 1 before repne_scan() in check_klass_subtype_slow_path()
> will cause exception java.lang.IncompatibleClassChangeError:
> 
>      // pointer were to sneak in here.  Note that we have already loaded 
> the
>      // Klass::super_check_offset from the super_klass in the fast path,
>      // so if there is a null in that register, we are already in the 
> afterlife.
> +    push(rax);
> +    xorq(rax,rax); // Set Z = 1
> +    pop(rax);
>      repne_scanl();
> 
> % java -Xmx128M -jar GCBasher.jar -time:300
> Exception in thread "main" java.lang.IncompatibleClassChangeError
>         at 
> java.lang.StringCoding$StringDecoder.decode(StringCoding.java:153)
> 
> Solution:
> Instead of checking RCX for zero set Z = 0 (not equal) before 'repne'
> to indicate that class was not found when RCX == 0.
> 
> Also fixed printed free stack size in hs_err files.
> Added ability to abort VM only on the exception with specific message.
> 
> Tested with failed case.