jdk8-aarch32: Crashing when running test case ParallelProbes.java

Anton Kozlov akozlov at azul.com
Thu Dec 1 09:40:35 UTC 2016


Hi,

Chi Vo, have you tried it the patch?

I checked once again, I'm still sure the problem is in native wrapper
syncronization. 

I'm using fastdebug build of aarch32 port and starting java like this to force only
one native wrapper compiled

java -Xcomp -Xbatch -XX:+PrintCompilation -XX:CompileCommand=compileonly,sun.nio.fs.GnomeFileTypeDetector::probeUsingGio ParallelProbes 10

Then I see next output 

CompilerOracle: compileonly sun/nio/fs/GnomeFileTypeDetector.probeUsingGio
Using <10> threads.
Write test file </tmp/prefix4916256039969939274.json>
Write test file </tmp/prefix6953504207421553297.json>
Write test file </tmp/prefix6728671690016035262.json>
Write test file </tmp/prefix8628750911095033714.json>
Write test file </tmp/prefix6869821973335621830.json>
Write test file </tmp/prefix9169485634055433217.json>
Write test file </tmp/prefix1354533543898818920.json>
Write test file </tmp/prefix6230527425500921461.json>
Write test file </tmp/prefix3800141328870272882.json>
Write test file </tmp/prefix6995679810723084303.json>
   2405    1  s  n       sun.nio.fs.GnomeFileTypeDetector::probeUsingGio (native)   (static)
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
...

thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
thread-2 -> application/json
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/synchronizer.cpp:1428
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/home/akozlov/src/aarch32-port-jdk8u/hotspot/src/share/vm/runtime/synchronizer.cpp:1428), pid=27680, tid=0x630de460
#  guarantee(obj->mark() == markOopDesc::encode(mid)) failed: invariant
#
# JRE version: OpenJDK Runtime Environment (8.0_102-b15) (build 1.8.0_102-internal-fastdebug-b15)
# Java VM: OpenJDK Embedded Client VM (25.102-b15-fastdebug compiled mode linux-aarch32 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/akozlov/src/aarch32-port-jdk8u/test-jtreg/hs_err_pid27680.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Current thread is 1661854816
Dumping core ...

But if I add -XX:+PreferInterpreterNativeStubs to command line 
(or use just `java -Xint`), I see pretty healty output and 0 exit code

Using <10> threads.
Write test file </tmp/prefix743927173084110988.json>
Write test file </tmp/prefix5843362962060469176.json>
Write test file </tmp/prefix5510457022909048313.json>
Write test file </tmp/prefix1390896761998247453.json>
Write test file </tmp/prefix8083543013963436664.json>
Write test file </tmp/prefix6791616484519529051.json>
Write test file </tmp/prefix1967926681770404025.json>
Write test file </tmp/prefix3738635124010065307.json>
Write test file </tmp/prefix7907003975940423096.json>
Write test file </tmp/prefix7591914614380640575.json>
thread-8 -> application/json
thread-4 -> application/json
thread-0 -> application/json
thread-2 -> application/json
thread-4 -> application/json
thread-1 -> application/json
thread-3 -> application/json
thread-1 -> application/json
thread-3 -> application/json
thread-8 -> application/json
thread-6 -> application/json
thread-3 -> application/json
thread-6 -> application/json
thread-8 -> application/json
thread-3 -> application/json

Note, that first run have only one thread id printed, in contrast with second 
run.

You could use -XX:+PrintNativeNMethods for second run to see generated code 
(hsdis.so required). In the output there is unlock fast path

   0x72e943b4:  add     r9, sp, #20
   0x72e943b8:  ldr     r3, [r9]
   0x72e943bc:  cmp     r3, #0
   0x72e943c0:  beq     0x72e943ec
   0x72e943c4:  ldrex   r9, [r2] <= r9 trashed 
   0x72e943c8:  cmp     r9, r9 <= broken, always 
   0x72e943cc:  bne     0x72e943e0 <= always not taken 
   0x72e943d0:  strex   r9, r3, [r2] <= always written
   0x72e943d4:  cmp     r9, #0
   0x72e943d8:  beq     0x72e943ec
   0x72e943dc:  b       0x72e943c4
   0x72e943e0:  dmb     ish
   0x72e943e4:  mov     r9, r9
   0x72e943e8:  b       0x72e944ac

As you can see, it's incorrect because of simultaneous use of r9 (rscratch1) 
for different purposes. Using rscratch2 as scratch to cmpxchgptr fixes the 
problem as well as run with first argument set.

Proposed change to jdk java code adds `syncronized` to next function

public static String probeContentType(Path path)
    throws IOException
{
    // try installed file type detectors
    for (FileTypeDetector detector: FileTypeDetectors.installeDetectors) {
	String result = detector.probeContentType(path);
	if (result != null)
	    return result;
    }

    // fallback to default
    return FileTypeDetectors.defaultFileTypeDetector.probeContentType(path);
}

What kind of accesses it's synchronizes? Could you provide a race example? 
>From function definition, I assume that it will eventually call `probeUsingGio`
with faulty unlock. Providing extra lock above could help to avoid crash 
in this particular jtreg test, but not in general.

Could you check that patch for native_wrapper works for you too?

diff --git a/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp b/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp
--- a/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp
+++ b/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp
@@ -2053,7 +2053,7 @@
 
     // Atomic swap old header if oop still contains the stack lock
     Label succeed;
-    __ cmpxchgptr(lock_reg, old_hdr, obj_reg, rscratch1, succeed, &slow_path_unlock);
+    __ cmpxchgptr(lock_reg, old_hdr, obj_reg, rscratch2, succeed, &slow_path_unlock);
     __ bind(succeed);
 
     // slow path re-enters here

Thanks,
Anton


On 11/10/2016 02:49 PM, Anton Kozlov wrote:
> Stack trace shows that ObjectMonitor::enter performs some illegal acesss, 
> triggering segfault.
> 
> As a second chance, are various combinations of UseFastLocking and 
> UseBiasedLocking changes the behavior?
> 
> I'm asking because I recall some bugfix that I had no time to properly
> prepare. That bug have exactly same symptoms, crash in vm runtime locking 
> code after platform "fast" locks failed to do something.
> 
> And as a last chance, is something differ from applying next patch? It fixes
> fast path locking code by using rscratch2 instead rscratch1 (which is 
> lock_reg also below)
> 
>> diff -r 1de174b764f9 -r 3d0ebcac084a src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp
>> --- a/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp      Thu Nov 03 13:12:50 2016 +0300
>> +++ b/src/cpu/aarch32/vm/sharedRuntime_aarch32.cpp      Thu Nov 10 14:41:09 2016 +0300
>> @@ -2053,7 +2053,7 @@
>>
>>      // Atomic swap old header if oop still contains the stack lock
>>      Label succeed;
>> -    __ cmpxchgptr(lock_reg, old_hdr, obj_reg, rscratch1, succeed, &slow_path_unlock);
>> +    __ cmpxchgptr(lock_reg, old_hdr, obj_reg, rscratch2, succeed, &slow_path_unlock);
>>      __ bind(succeed);
>>
>>      // slow path re-enters here
>>
> 
> 
> Thanks,
> Anton
> 
> 
> On 11/10/2016 02:20 PM, Chi Vo wrote:
>> Hi all,
>>
>> This issue still occurred with ‎-XX:-UseBiasedLocking option.
>>
>> Thanks & Best Regards,
>> Chi Vo
>>
>> Sent from my BlackBerry 10 smartphone.
>>   Original Message
>> From: Anton Kozlov
>> Sent: Thursday, November 10, 2016 18:04
>> To: Chi Vo; Sergey Nazarkin
>> Cc: aarch32-port-dev
>> Subject: Re: jdk8-aarch32: Crashing when running test case ParallelProbes.java
>>
>> Hi, all,
>>
>> looks familar.
>> Chi, could you add -XX:-UseBiasedLocking option?
>>
>> Thanks,
>> Anton
>>
>> On 11/10/2016 02:00 PM, Chi Vo wrote:
>>> Hi Sergey Nazarkin,
>>>
>>> Stack trace of the crash:
>>>
>>> [New LWP 26548]
>>> [New LWP 26530]
>>> [New LWP 26542]
>>> [New LWP 26531]
>>> [New LWP 26551]
>>> [New LWP 26535]
>>> [New LWP 26545]
>>> [New LWP 26544]
>>> [New LWP 26536]
>>> [New LWP 26546]
>>> [New LWP 26538]
>>> [New LWP 26543]
>>> [New LWP 26550]
>>> [New LWP 26539]
>>> [New LWP 26537]
>>> [New LWP 26552]
>>> [New LWP 26541]
>>> [New LWP 26540]
>>> [New LWP 26549]
>>> [New LWP 26547]
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library
>>> "/lib/arm-linux-gnueabihf/libthread_db.so.1".
>>> Core was generated by `/home/pi/workspace/openjdk/./j2re-image/bin/java
>>> -Dtest.src=/home/pi/workspace/'.
>>> Program terminated with signal SIGABRT, Aborted.
>>> #0 0x76df4f70 in __GI_raise (sig=sig at entry=6) at
>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>> 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>>> (gdb) bt
>>> #0 0x76df4f70 in __GI_raise (sig=sig at entry=6) at
>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>> #1 0x76df6324 in __GI_abort () at abort.c:89
>>> #2 0x758acee8 in os::abort(bool) () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #3 0x75917b64 in VMError::report_and_die() () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #4 0x758b1952 in JVM_handle_linux_signal () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #5 0x758ac140 in signalHandler(int, siginfo_t*, void*) () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #6 <signal handler called>
>>> #7 0x758a7424 in ObjectMonitor::enter(Thread*) () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #8 0x758e865a in ObjectSynchronizer::slow_enter(Handle, BasicLock*,
>>> Thread*) () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #9 0x758d33fe in SharedRuntime::complete_monitor_locking_C(oopDesc*,
>>> BasicLock*, JavaThread*) () from
>>> /home/pi/workspace/openjdk/j2re-image/lib/aarch32/client/libjvm.so
>>> #10 0x734b5500 in ?? ()
>>> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>>> (gdb)
>>>
>>> If you want to get core file. Please contact me.
>>>
>>> Thanks & Best Regards,
>>> Chi Vo
>>>
>>> On Thu, Nov 10, 2016 at 5:10 PM, Sergey Nazarkin <snazarkin at azul.com> wrote:
>>>
>>>> Hi Chi Vo,
>>>>
>>>> Could you provide stack trace of the crash? I see no reason for race
>>>> condition here since the method doesn’t update any global variable.
>>>>
>>>>
>>>> Sergey Nazarkin
>>>>
>>>>
>>>>
>>>>
>>>>> On 10 Nov 2016, at 12:49, Chi Vo <thegeek108 at gmail.com> wrote:
>>>>>
>>>>> Hi All,
>>>>>
>>>>> Hotspot crashing when I running test case
>>>>> */jdk/test/java/nio/file/Files/probeContentType/ParallelProbes.java*
>>>>>
>>>>> As my analysis, the probeContentType is static method so It is class
>>>> level.
>>>>> In this case, we allow many threads access it without synchronize. so It
>>>>> should be protected by synchronizing to avoid any inconsistent data or
>>>> also
>>>>> corrupt data. I test this patch and it works fine.
>>>>>
>>>>> diff -r dbe7e2abe8b2 -r c9e2721b4711
>>>>> 02.src/01.aarch32_jdk8u/jdk/src/share/classes/java/nio/file/Files.java
>>>>> --- a/02.src/01.aarch32_jdk8u/jdk/src/share/classes/java/nio/
>>>> file/Files.java
>>>>> Fri
>>>>> Oct 07 15:23:45 2016 +0700
>>>>> +++ b/02.src/01.aarch32_jdk8u/jdk/src/share/classes/java/nio/
>>>> file/Files.java
>>>>> Tue
>>>>> Oct 25 10:56:47 2016 +0700
>>>>> @@ -1610,7 +1610,7 @@
>>>>> * If a security manager is installed and it denies an
>>>>> unspecified
>>>>> * permission required by a file type detector
>>>> implementation.
>>>>> */
>>>>> - public static String probeContentType(Path path)
>>>>> + public synchronized static String probeContentType(Path path)
>>>>> throws IOException
>>>>> {
>>>>> // try installed file type detectors
>>>>>
>>>>> Please help take a look my patch.
>>>>>
>>>>> Thanks you with best regards,
>>>>> Chi Vo
>>>>
>>>>
>>>
>>>


More information about the aarch32-port-dev mailing list