hsperfdata causing long GC/safepoint times: Don't use mmap?

[Mikael Gerdin]

On 2015-03-27 13:21, Staffan Larsen wrote:
>
>> On 26 mar 2015, at 16:33, Mikael Gerdin <mikael.gerdin at oracle.com> wrote:
>>
>> Hi Evan,
>>
>> On 2015-03-26 16:14, Evan Jones wrote:
>>> At Twitter, I recently discovered that the hsperfdata file that is created
>>> by default by the JVM causes long safepoint and GC pauses. It turns out
>>> that writes to mmap-ed files can block until disk I/O completes, even if
>>> the I/O is to another disk. For details see:
>>> http://www.evanjones.ca/jvm-mmap-pause.html
>>
>> Truly interesting stuff!
>>
>> I know we've seen similar discrepancies in user/sys/real time before, this could indeed be a cause for at least some of them.
>>
>>>
>>> We have been experimenting with adding the -XX:+PerfDisableSharedMem JVM
>>> flag on a number of our latency sensitive services, and have seen a
>>> significant improvement. Our JVM team (which I am *not* part of), is
>>> investigating potential changes to the JVM to prevent this.
>>>
>>> Any suggestions for an approach to solving this problem that could be
>>> accepted into Hotspot itself? Some options:
>>>
>>> * Make the location of this file configurable (this was set with
>>> java.io.tmpdir for a time, but then was reverted; see
>>> http://bugs.java.com/view_bug.do?bug_id=6447182)
>>>
>>> * Use shared memory that is not backed by a file?
>>
>> Yeah, something like shm_open("/hsperfdata_<user>_<pid>", O_RDWR|O_CREAT|O_EXCL, 0600)
>
> The problem with that is that tools such as jcmd and jps uses these files to find currently running JVMs on the system. We would need a separate method to do that discovery in that case.

I see. I didn't realize that the attach mechanism relied on the 
hsperfdata file as well, I thought it used its own temp files.

I guess the hsperfdata could be separated into a mostly-static version 
which lives in /tmp and a continuously updated one in shared memory, the 
location of the shared memory one could be encoded in the mostly-static 
file to avoid exposing yet another fixed file name API.

/Mikael

>
>>
>>>
>>> * Something else I'm not considering?
>>
>> Have the VM asynchronously update the values in the hsperfdata file by caching the new values somewhere in memory and using the ServiceThread to commit the updated values to the mmaped perf data?
>
> That will add a latency to when the values are available for watchers. Maybe not a problem since the data can’t be read synchronously anyway.
>
> /Staffan
>
>>
>> I'll file and issue for this and copy the relevant information into it.
>>
>> /Mikael
>>
>>>
>>>
>>> Thanks!
>>>
>>> Evan Jones
>>>
>>
>