SIGBUS on linux in perfMemory_init (containers)

[Severin Gehwolf]

Hi,

On Fri, 2022-04-29 at 14:19 -0700, Ioi Lam wrote:
> 
> 
> On 4/29/2022 1:55 PM, Ioi Lam wrote:
> > 
> > 
> > On 4/29/2022 6:44 AM, Vitaly Davidovich wrote:
> > > Hi all,
> > > 
> > > We've been seeing intermittent SIGBUS failures on linux with jdk11.  
> > > They
> > > all have this distinctive backtrace:
> > > 
> > > C  [libc.so.6+0x12944d]
> > > 
> > > V  [libjvm.so+0xcca542]  perfMemory_init()+0x72
> > > 
> > > V  [libjvm.so+0x8a3242]  vm_init_globals()+0x22
> > > 
> > > V  [libjvm.so+0xedc31d]  Threads::create_vm(JavaVMInitArgs*, 
> > > bool*)+0x1ed
> > > 
> > > V  [libjvm.so+0x9615b2]  JNI_CreateJavaVM+0x52
> > > 
> > > C  [libjli.so+0x49af]  JavaMain+0x8f
> > > 
> > > C  [libjli.so+0x9149]  ThreadJavaMain+0x9
> > > 
> > > 
> > > Initially, we suspected that /tmp was full but that turned out to not be
> > > the case.  After a few more instances of the crash and investigation, we
> > > believe we know the root cause.
> > > 
> > > 
> > > The crashing applications are all running in a K8 pod, with each JVM 
> > > in a
> > > separate container:
> > > 
> > > 
> > > container_type: cgroupv1 (from the hs_err file)
> > > 
> > > 
> > > /tmp is mounted such that it's shared by multiple containers. Since 
> > > these
> > > JVMs are running in containers, we believe what happens is the 
> > > namespaced
> > > (i.e. per container) PIDs overlap between different containers - 2 
> > > JVMs, in
> > > separate containers, can end up with the same namespaced PID. Since /tmp
> > > is shared, they can now "contend" on the same perfMemory file since 
> > > those
> > > file names are PID based.
> > 
> > Hi Vitaly,
> > 
> > Is there any reason for sharing the same /tmp directory across 
> > different containers?
> > 
> > Are you using the /tmp/hsperfdata_$USER/<pid> files at all. If not,
> > for the time being, you can disable them with the -XX:-UsePerfData flag,
> > 
> > https://bugs.openjdk.java.net/browse/JDK-8255008 has a related proposal:
> > 

This bug is private. Could this one be made accessible somehow?

Another related bug seems to be, though not quite the same:
https://bugs.openjdk.java.net/browse/JDK-8284330

> > Java: -Djdk.attach.tmpdir=/container-attachdir 
> > -XX:+UnlockCommercialFeature -XX:+FlightRecorder -XX:+StartAttachListener
> > Docker: --volume /tmp/container-attachdir:/container-attachdir
> > 
> > In this case, we probably will run into the same UID clash problem as 
> > well.
> > 
> > Maybe we should have an additional property like 
> > -Djdk.attach.use.global.pid=true
> > 
> 
> I read the proposal in JDK-8255008 again and realized that the JVM 
> inside the container doesn't know what it's host PID is. The proposal is 
> to create these files:
> 
> $jdk_attach_dir/hsperfdata_{user}/e4f3e2e4fd97:10
> $jdk_attach_dir/.java_pid:e4f3e2e4fd97:10
> 
> where the e4f3e2e4fd97 is the container ID which is visible as 
> /tmp/hostname from inside the container.
> 
> I'll try to implement a prototype for the proposal.

Please be aware that the container's hostname is also user-settable.
E.g.

$ docker run --hostname foo ...

Would set the hostname to 'foo'.

Ioi, did you end up creating a bug for this?

Thanks,
Severin

> Thanks
> - Ioi
> 
> > 
> > > 
> > > Once multiple JVMs can contend on the same file, a SIGBUS can
> > > arise 
> > > if one
> > > JVM has mmap'd the file and another ftruncate()'s it from under
> > > it (e.g.
> > > https://github.com/openjdk/jdk11/blob/37115c8ea4aff13a8148ee2b8832b20888a5d880/src/hotspot/os/linux/perfMemory_linux.cpp#L909
> > >  
> > > 
> > > ).
> > > 
> > > 
> > > Is this a known issue? I couldn't find any existing JBS entries
> > > or 
> > > mailing
> > > list discussions around this specific circumstance.
> > > 
> > > 
> > > As for possible solutions, would it be possible to use the global
> > > PID
> > > instead of the namespaced PID to "regain" the uniqueness
> > > invariant of 
> > > the
> > > PID?
> > 
> > I think this needs to be optional. E.g., if you run a tool such as 
> > "jcmd" inside the same container as the jvm process, the jcmd tool 
> > would expect the PID to be the local version specific to this
> > container.
> > 
> > >   Also, might it make sense to flock() the file to prevent
> > > another
> > > process from mucking with it?
> > > 
> > That will not solve the fundamental problem where two processes are
> > using the same hsperfdata file. One of them would fail to write to
> > it, 
> > and tools won't be able to monitor both processes.
> > 
> > Thanks
> > - Ioi
> > 
> > > Happy to provide more info if needed.
> > > 
> > > 
> > > Thanks
> > 
>