<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<font face="Helvetica, Arial, sans-serif">Mark-<br>
<br>
You've probably already gotten this suggestion, but another alternative
to look at is Sun's real-time system.<br>
<br>
<a class="moz-txt-link-freetext" href="http://java.sun.com/javase/technologies/realtime/index.jsp">http://java.sun.com/javase/technologies/realtime/index.jsp</a><br>
<br>
Right now, we offer a Java 5 compatible release that provides a number
of means of managing the impact of GC in an application. It does trade
off throughput, but that may not be a barrier in your specific case.
It also provides a threading model with fine grained priorities that
may be more to your liking It offers a priority inversion protocol as
part of the implementation too.<br>
<br>
-John<br>
</font><br>
Paul Hohensee wrote:
<blockquote cite="mid:49E508DE.6060300@sun.com" type="cite">The url for
"Hotspot Runtime Overview of JNI" didn't come through for me.
<br>
<br>
In any case, as Ramki noted, Hotspot lets native code called from Java
run free
<br>
during GC pauses, unless said native code calls back into the jvm for
some reason,
<br>
including just returning back to Java from native. Hotspot does _not_
try to
<br>
pause threads executing native code, nor does Hotspot use signal
mechanisms
<br>
to block threads executing Java code so GC can happen. Hotspot uses a
polling
<br>
mechanism instead because signal delivery mechanisms on Unix and Linux
are
<br>
unreliable. I doubt you'll be able to reproduce the JRockit issue with
Hotspot.
<br>
You might have other problems, but not that one.
<br>
<br>
I suggest forwarding your questions to the JRockit team at Oracle.
Though I suspect
<br>
some of them are on this list too. :)
<br>
<br>
Please try Hotspot before you give up on Java. From your description,
you should
<br>
use the CMS (concurrent mark-sweep) collector. See also the GC
performance info
<br>
accessible from
<br>
<br>
<a class="moz-txt-link-freetext" href="http://java.sun.com/performance">http://java.sun.com/performance</a>
<br>
<br>
and Jon Masamitsu's blog
<br>
<br>
<a class="moz-txt-link-freetext" href="http://blogs.sun.com/jonthecollector/">http://blogs.sun.com/jonthecollector/</a>
<br>
<br>
Paul
<br>
<br>
Mark R Maxey wrote:
<br>
<blockquote type="cite"><br>
After reading the last paragraph of HotSpot Runtime Overview of JNI
more closely, I understand more. I think we're almost on the same
page. The problem seems to be that all threads are suspended until the
Java thread returns from the native call.
<br>
<br>
<br>
Mark Maxey
<br>
Raytheon, Garland
<br>
580/2/P22-1
<br>
(972)205-5760
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Mark_R_Maxey@Raytheon.com">Mark_R_Maxey@Raytheon.com</a>
<br>
<br>
<br>
*Mark R Maxey/US/Raytheon*
<br>
<br>
04/14/2009 12:37 PM
<br>
<br>
<br>
To
<br>
"Y. Srinivas Ramakrishna" <a class="moz-txt-link-rfc2396E" href="mailto:Y.S.Ramakrishna@Sun.COM"><Y.S.Ramakrishna@Sun.COM></a>
<br>
cc
<br>
<a class="moz-txt-link-abbreviated" href="mailto:hotspot-gc-dev@openjdk.java.net">hotspot-gc-dev@openjdk.java.net</a>, <a class="moz-txt-link-abbreviated" href="mailto:Y.S.Ramakrishna@Sun.COM">Y.S.Ramakrishna@Sun.COM</a>, Andrew M
Dungan/US/Raytheon@MAIL, David A Lilly/RCS/Raytheon/US@MAIL, Mark R
Maxey/US/Raytheon@MAIL
<br>
Subject
<br>
Re: Garbage Collection Pauses & Non-interruptable System
CallsLink
<Notes://MK2-MSG05/86256EF3005F851B/38D46BF5E8F08834852564B500129B2C/E1F59319D41C5C4886257598005422BD>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Thank you for your reply. The speed and depth of your response is
encouraging.
<br>
<br>
Let me confess something I should have done up-front. The behavior
we're seeing is using JDK 5 via JRockit R27.6. We're in the process of
reproducing these problems under HotSpot JDK 6 Update 12, though it'll
be a few days before we can do so. The reason I'm pinging this forum
is to research in advance what differences we might expect between the
two JVMs.
<br>
<br>
Let me describe exactly what we're seeing as provided by doing an
strace on the process:
<br>
<br>
1. A Java thread calls a native C code that ultimately calls a
<br>
pwrite(). We suspect that the device driver ultimately makes a
<br>
non-interruptable system call to transfer the data directly from
<br>
our mem-aligned 128 MB buffer to disk.
<br>
2. The GC thread sends a tgkill(SIGUSR1) to all threads
<br>
3. The GC thread waits on mutex #1 (presumably waiting on all the
<br>
threads to signal it that it can begin GC)
<br>
4. The Java thread wakes mutex #1 (presumably signaling the GC it
<br>
is ready to go)
<br>
5. The Java thread waits on mutex #2 (presumably waiting on GC to
<br>
finish)
<br>
6. The GC thread wakes mutex #2 (presumably telling the Java thread
<br>
it can resume processing)
<br>
<br>
<br>
We're seeing times between #3 & #4 that are proportional to the
amount of time spent in the pwrite(). We also see some overhead
between #5  that is proportional to the number of Java threads
we have (currently between 30 & 40 that we've created not counting
the JVMs).
<br>
<br>
Unfortunately, the JRockit logging only reveals the actual time GC
takes (#4 - #5). Hopefully, HotSpot's logging includes the total time
(#2 - #6).
<br>
<br>
I'm pursuing these questions with Oracle/BEA. Again, I'm just trying
get a feel for HotSpot's behavior in comparison. While we're using
JRockit today, HotSpot will be our ultimate platform.
<br>
<br>
<br>
One alternate solution that has been suggested is infrequently calling
GC explicitly within our code during special times when we know we can
afford to take the hit. We would even accept a greater hit than normal
if we could avoid being impacted during critical times. Everything
I've ever read says to not do this, but I'm curious why in this case
this is a bad idea. Note that we're using the concurrent GC, so I'm
not even sure if System.gc() supports this.
<br>
<br>
<br>
Thanks again!
<br>
<br>
<br>
Mark Maxey
<br>
Raytheon, Garland
<br>
580/2/P22-1
<br>
(972)205-5760
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Mark_R_Maxey@Raytheon.com">Mark_R_Maxey@Raytheon.com</a>
<br>
<br>
<br>
*"Y. Srinivas Ramakrishna" <a class="moz-txt-link-rfc2396E" href="mailto:Y.S.Ramakrishna@Sun.COM"><Y.S.Ramakrishna@Sun.COM></a>*
<br>
Sent by: <a class="moz-txt-link-abbreviated" href="mailto:Y.S.Ramakrishna@Sun.COM">Y.S.Ramakrishna@Sun.COM</a>
<br>
<br>
04/14/2009 10:19 AM
<br>
<br>
<br>
To
<br>
Mark R Maxey <a class="moz-txt-link-rfc2396E" href="mailto:Mark_R_Maxey@raytheon.com"><Mark_R_Maxey@raytheon.com></a>
<br>
cc
<br>
<a class="moz-txt-link-abbreviated" href="mailto:hotspot-gc-dev@openjdk.java.net">hotspot-gc-dev@openjdk.java.net</a>
<br>
Subject
<br>
Re: Garbage Collection Pauses & Non-interruptable System Calls
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Hello Mark --
<br>
<br>
I am assuming your threads doing DMA are actually executing native code
(or
<br>
waiting for signals in native code). Threads in native code do not
need to
<br>
synchronize \in any manner with GC while they are executing native
code.
<br>
It is only the transitions to and from native mode (from Java code)
that
<br>
require
<br>
synchronization. Roughly speaking, the JVM fences off those native
<br>
threads so that, in the event that they need to re-enter the JVM or
<br>
access the Java heap, they will be suspended until a GC/safepoint that
<br>
is in progress is completed.
<br>
<br>
Thus, I do not believe you need to fear that a long-running DMA call
would
<br>
cause GC's to be delayed (which I understand is your main concern
below).
<br>
<br>
Have you actually seen cases where this is happening? If so, what
<br>
version of the JDK
<br>
are you running?
<br>
<br>
thanks.
<br>
-- ramki
<br>
<br>
Mark R Maxey wrote:
<br>
> Hello,
<br>
>
<br>
> I have a problem I was hoping with which I need some advice.
<br>
>
<br>
> We wrote a custom JNI library for file I/O that sits underneath
the Java
<br>
> NIO FileChannel. One of our driving requirements is highly
performant
<br>
> file I/O. We achieved this by doing DMA I/O from large direct
memory
<br>
> aligned buffers. The JNI is very trivial - it just takes a buffer
and
<br>
> performs the appropriate system call based on the parameters given
to it.
<br>
> 100% of the logic for calculating offsets, buffer management, etc.
is all
<br>
> in our implementation of java.nio.FileChannel.
<br>
>
<br>
> Here's our problem: We have requirements to respond to some
messages in
<br>
> as little as 250 ms. During this time, we're doing file writes of
128 MB
<br>
> that take around 200 ms. When GC kicks in, it tries to pause all
threads.
<br>
> Because the DMA write is non-interruptable, GC waits for the I/O
to
<br>
> complete before being able to pause the thread & run. That
means that GC
<br>
> can take well over 200 ms putting us in grave danger of missing
our
<br>
> timelines. Worse, there is always the chance the write will hang
due to a
<br>
> bad filesystem. We've seen this cause the JVM to hang
indefinitely
<br>
> forcing us to cycle the process.
<br>
>
<br>
> Unless we find a solution that allows GC to continue while doing
this I/O,
<br>
> we will convert all the code to C++. While that might solve our
timeline
<br>
> for that particular process, we have many less performance
critical
<br>
> processes that use our JNI FileChannel libraries that would hang
if a
<br>
> filesystem goes bad.
<br>
>
<br>
> We've tweaked the file system device timeouts down to a minimum,
but they
<br>
> are still very high (on the order of several seconds to minutes).
It
<br>
> would be nice if the JVM had a similar timeout for pausing
threads, i.e.,
<br>
> where the pause times out after X number of milliseconds. We'd be
willing
<br>
> to sacrifice a larger heap size and postpone GC in the hopes that
the next
<br>
> time it ran GC, we wouldn't be in the middle of a
non-interruptable system
<br>
> call.
<br>
>
<br>
> The only solution being batted around here is pushing the system
calls out
<br>
> of Java threads and into native threads. The JNI call would push
the info
<br>
> for the I/O call onto a native C++ queue where a small number of
native
<br>
> threads (3?) would pull the data off the queue and perform the
actual
<br>
> system call. The trick is finding an implementation where the
Java
<br>
> thread blocked waiting on a response from the native thread is
<br>
> interruptible. All this assumes GC doesn't try to pause native
threads.
<br>
> We thought about using pthreads, but were concerned about its
signal
<br>
> interaction with the JVM. So, we're leaning towards using pipes
to push
<br>
> data from one thread to another.
<br>
>
<br>
> If you have any suggestions or advice, we are desperate for your
wisdom.
<br>
>
<br>
> Thanks!
<br>
>
<br>
>
<br>
> Mark Maxey
<br>
> Raytheon, Garland
<br>
> 580/2/P22-1
<br>
> (972)205-5760
<br>
> <a class="moz-txt-link-abbreviated" href="mailto:Mark_R_Maxey@Raytheon.com">Mark_R_Maxey@Raytheon.com</a>
<br>
>
<br>
> <br>
<br>
<br>
<br>
<br>
</blockquote>
</blockquote>
</body>
</html>