<br><font size=2 face="sans-serif">Thank you for your reply. The
speed and depth of your response is encouraging.</font>
<br>
<br><font size=2 face="sans-serif">Let me confess something I should have
done up-front. The behavior we're seeing is using JDK 5 via JRockit
R27.6. We're in the process of reproducing these problems under HotSpot
JDK 6 Update 12, though it'll be a few days before we can do so. The
reason I'm pinging this forum is to research in advance what differences
we might expect between the two JVMs.</font>
<br>
<br><font size=2 face="sans-serif">Let me describe exactly what we're seeing
as provided by doing an strace on the process:</font>
<ol>
<li value=1><font size=2 face="sans-serif">A Java thread calls a native
C code that ultimately calls a pwrite(). We suspect that the device
driver ultimately makes a non-interruptable system call to transfer the
data directly from our mem-aligned 128 MB buffer to disk.</font>
<li value=2><font size=2 face="sans-serif">The GC thread sends a tgkill(SIGUSR1)
to all threads</font>
<li value=3><font size=2 face="sans-serif">The GC thread waits on mutex
#1 (presumably waiting on all the threads to signal it that it can begin
GC)</font>
<li value=4><font size=2 face="sans-serif">The Java thread wakes mutex
#1 (presumably signaling the GC it is ready to go)</font>
<li value=5><font size=2 face="sans-serif">The Java thread waits on mutex
#2 (presumably waiting on GC to finish)</font>
<li value=6><font size=2 face="sans-serif">The GC thread wakes mutex #2
(presumably telling the Java thread it can resume processing)</font></ol>
<br><font size=2 face="sans-serif">We're seeing times between #3 &
#4 that are proportional to the amount of time spent in the pwrite(). We
also see some overhead between #5  that is proportional to the number
of Java threads we have (currently between 30 & 40 that we've created
not counting the JVMs).</font>
<br>
<br><font size=2 face="sans-serif">Unfortunately, the JRockit logging only
reveals the actual time GC takes (#4 - #5). Hopefully, HotSpot's
logging includes the total time (#2 - #6).</font>
<br>
<br><font size=2 face="sans-serif">I'm pursuing these questions with Oracle/BEA.
Again, I'm just trying get a feel for HotSpot's behavior in comparison.
While we're using JRockit today, HotSpot will be our ultimate platform.</font>
<br>
<br>
<br><font size=2 face="sans-serif">One alternate solution that has been
suggested is infrequently calling GC explicitly within our code during
special times when we know we can afford to take the hit. We would
even accept a greater hit than normal if we could avoid being impacted
during critical times. Everything I've ever read says to not do
this, but I'm curious why in this case this is a bad idea. Note that
we're using the concurrent GC, so I'm not even sure if System.gc() supports
this.</font>
<br>
<br>
<br><font size=2 face="sans-serif">Thanks again!</font>
<br>
<br><font size=2 face="sans-serif"><br>
Mark Maxey<br>
Raytheon, Garland<br>
580/2/P22-1<br>
(972)205-5760<br>
Mark_R_Maxey@Raytheon.com</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>"Y. Srinivas Ramakrishna"
<Y.S.Ramakrishna@Sun.COM></b> </font>
<br><font size=1 face="sans-serif">Sent by: Y.S.Ramakrishna@Sun.COM</font>
<p><font size=1 face="sans-serif">04/14/2009 10:19 AM</font>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">Mark R Maxey <Mark_R_Maxey@raytheon.com></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">hotspot-gc-dev@openjdk.java.net</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: Garbage Collection Pauses &
Non-interruptable System Calls</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><tt><font size=2>Hello Mark --<br>
<br>
I am assuming your threads doing DMA are actually executing native code
(or<br>
waiting for signals in native code). Threads in native code do not
need to<br>
synchronize \in any manner with GC while they are executing native code.<br>
It is only the transitions to and from native mode (from Java code) that
<br>
require<br>
synchronization. Roughly speaking, the JVM fences off those native<br>
threads so that, in the event that they need to re-enter the JVM or<br>
access the Java heap, they will be suspended until a GC/safepoint that<br>
is in progress is completed.<br>
<br>
Thus, I do not believe you need to fear that a long-running DMA call would<br>
cause GC's to be delayed (which I understand is your main concern
below).<br>
<br>
Have you actually seen cases where this is happening? If so, what <br>
version of the JDK<br>
are you running?<br>
<br>
thanks.<br>
-- ramki<br>
<br>
Mark R Maxey wrote:<br>
> Hello,<br>
><br>
> I have a problem I was hoping with which I need some advice.<br>
><br>
> We wrote a custom JNI library for file I/O that sits underneath the
Java <br>
> NIO FileChannel. One of our driving requirements is highly performant
<br>
> file I/O. We achieved this by doing DMA I/O from large direct
memory <br>
> aligned buffers. The JNI is very trivial - it just takes a buffer
and <br>
> performs the appropriate system call based on the parameters given
to it. <br>
> 100% of the logic for calculating offsets, buffer management, etc.
is all <br>
> in our implementation of java.nio.FileChannel.<br>
><br>
> Here's our problem: We have requirements to respond to some
messages in <br>
> as little as 250 ms. During this time, we're doing file writes
of 128 MB <br>
> that take around 200 ms. When GC kicks in, it tries to pause
all threads. <br>
> Because the DMA write is non-interruptable, GC waits for the
I/O to <br>
> complete before being able to pause the thread & run. That
means that GC <br>
> can take well over 200 ms putting us in grave danger of missing our
<br>
> timelines. Worse, there is always the chance the write will
hang due to a <br>
> bad filesystem. We've seen this cause the JVM to hang indefinitely
<br>
> forcing us to cycle the process.<br>
><br>
> Unless we find a solution that allows GC to continue while doing this
I/O, <br>
> we will convert all the code to C++. While that might solve
our timeline <br>
> for that particular process, we have many less performance critical
<br>
> processes that use our JNI FileChannel libraries that would hang if
a <br>
> filesystem goes bad.<br>
><br>
> We've tweaked the file system device timeouts down to a minimum, but
they <br>
> are still very high (on the order of several seconds to minutes).
It <br>
> would be nice if the JVM had a similar timeout for pausing threads,
i.e., <br>
> where the pause times out after X number of milliseconds. We'd
be willing <br>
> to sacrifice a larger heap size and postpone GC in the hopes that
the next <br>
> time it ran GC, we wouldn't be in the middle of a non-interruptable
system <br>
> call.<br>
><br>
> The only solution being batted around here is pushing the system calls
out <br>
> of Java threads and into native threads. The JNI call would
push the info <br>
> for the I/O call onto a native C++ queue where a small number of native
<br>
> threads (3?) would pull the data off the queue and perform the actual
<br>
> system call. The trick is finding an implementation where the
Java <br>
> thread blocked waiting on a response from the native thread is <br>
> interruptible. All this assumes GC doesn't try to pause native
threads. <br>
> We thought about using pthreads, but were concerned about its signal
<br>
> interaction with the JVM. So, we're leaning towards using pipes
to push <br>
> data from one thread to another.<br>
><br>
> If you have any suggestions or advice, we are desperate for your wisdom.<br>
><br>
> Thanks!<br>
><br>
><br>
> Mark Maxey<br>
> Raytheon, Garland<br>
> 580/2/P22-1<br>
> (972)205-5760<br>
> Mark_R_Maxey@Raytheon.com<br>
><br>
> <br>
<br>
</font></tt>
<br>
<br>
<br>