GC_Locker turning CMS into a stop-the-world collector
Chi Ho Kwok
chkwok at digibites.nl
Sat Jan 16 17:06:44 PST 2010
Hmm, where to begin...
Today, I started tracing a weird, periodic performance problem with a
jetty based server. The app uses quite a bit of memory, so it's
running with a CMS collector and a 16G heap. We've noticed some weird,
random 20-30 seconds latency, which occurs about once an hour, so I've
added some diagnostics and most importantly, a way to call Thread.
getAllStackTraces() remotely - and captured a live trace of the 'bug'.
So I've got almost every thread (~60) on the system blocked on
at java.nio.Bits.copyToByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:191)
at sun.nio.ch.IOUtil.read(IOUtil.java:209)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:131)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
And a few other threads are blocked via different paths on copyTo/FromByteArray.
What's happening?!
Looking through the code, tracing from copyToByteArray to
jni_GetPrimitiveArrayCritical, GC_locker::lock_critical(thread) in
jni.cpp:2546 looks like the cause of all evil. But the bug doesn't
trigger on every GC, nope, so this alone won't cause everything to get
stuck. We have a minor collection every few seconds and a major one
every minute, but the problem is more rare, it occurs about once per
hour, but can be absent for up to half a day when you're lucky. But it
has the annoying tendency to occur when it's unusually busy.
To understand what the heck is happening, I've opened
gcLocker.cpp/hpp/.inline.hpp, and tried to find the exact condition
how this can happen. lock_critical() has two paths - a fast path when
there's enough memory and needs_gc() returns false, or a slow path
which *blocks the thread* on JNICritical_lock->wait() if needs_gc() or
_doing_gc is true. It's the same for unlock_critical, if needs_gc() is
set, it calls the _slow() version.
All the puzzle pieces are here now, all I needed to do is bring them
all together in a scenario:
1. A thread enters a critical section, calls lock_critical(), uses the fast path
2. needs_gc() goes from false to true, because old generation
utilization is above the collection threshold. CMS background
collection won't start, yet, because GC_Locker's _jni_lock_count is >
0.*
3. Thread exits critical section. needs_gc() is true, so it uses
jni_unlock_slow()
4. In jni_unlock_slow(), this is the last thread out. That will set
_doing_gc to true, and block all future attempts on lock_critical()
until that flag is cleared **
5. Do a full collection by calling
Universe::heap()->collect(GCCause::_gc_locker). This take "forever"
***
6. Set _doing_gc to false again via clear_needs_gc()
7. Rejoice! JNICritical_lock->notify_all() is called! And watch the
load go to 30+ because the server has been doing nothing for ages.
*: speculation. Didn't actually find the line calling set_needs_gc in
vm/gc_implementation/*, but I assume it's set from there, somewhere.
**: I've seen a few threads that were blocked on a copyToByteArray in
a MappedByteBuffer that has been .load()-ed, so it must be a lock. A
simple memcpy cannot take this long.
***: speculation. Didn't actually measure the time, but because every
single thread in the system is trapped in JNICritical_lock->wait(), I
assume this part takes forever.
So the conclusion is: using a channel to handle HTTP requests combined
with a large heap is suicidal at the moment. Every channel.read() call
will block when GC_Locker initiates a CMS collection, blocking every
single request until the foreground collection is done. That's because
SocketChannelImpl.read() calls copyToByteArray internally, and it will
block during a GC_Locker initiated CMS-collection.
I'm just a lowly java programmer, so I've no idea how to fix this. My
workaround now is just to stop using a SelectChannelConnector and
switch back to a ServerSocketConnector in Jetty, praying that the
other nio calls I have in the code (MappedByteBuffer.read() is
semi-frequently used) won't turn CMS into a very slow, almost serial
stop the world collector.
I kinda doubt that this is intended, so, should I file this as a bug?
I've (mis)filed it already as #1695140, but the analysis is wrong
there, blaming it on "some kind of spinlock?", and as I thought it was
unrelated to gc, no jvm arguments were attached. Well, it's a lock,
but it's quite a bit more complex than I assumed.
Any help would be appreciated. I've got no experience at all in
hacking jvm's, but I can try to create a test case to trigger this
situation manually and try different builds of the jvm.
Attached is the full stack trace of all threads during the 'lag
spike'. Note that the stack dump is triggered over http, using java.io
/ streams; that one kept working while all requests on a channel were
blocked. "InstrumentedConnector" is just simple a subclass of the
default jetty SelectChannelConnector, with extra stats to help
figuring out what's happening.
Chi Ho Kwok
-------------- next part --------------
Format is: list of threads with the same stack trace, followed by the actual stack trace
2010-01-16 18:34:03.102240 - 7.39908218384 ms
{'priority': 1, 'state': 'RUNNABLE', 'name': '1468041408 at qtp-1986936160-28'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '962687369 at qtp-1986936160-30'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1506392522 at qtp-1986936160-52'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1728081269 at qtp-1986936160-3'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1352230939 at qtp-1986936160-39'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1663906309 at qtp-1986936160-40'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '411509632 at qtp-1986936160-31'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '148376547 at qtp-1986936160-2'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '413536612 at qtp-1986936160-17'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '761700907 at qtp-1986936160-44'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1222033772 at qtp-1986936160-6'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1361235382 at qtp-1986936160-13'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '405642246 at qtp-1986936160-34'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '623839641 at qtp-1986936160-14'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '870010735 at qtp-1986936160-7'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1027818036 at qtp-1986936160-1'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '981863753 at qtp-1986936160-24'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '555551311 at qtp-1986936160-18'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1526644999 at qtp-1986936160-29'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '267273800 at qtp-1986936160-37'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1476473244 at qtp-1986936160-58'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '172647384 at qtp-1986936160-20'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1357862146 at qtp-1986936160-10'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '2773808 at qtp-1986936160-5'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '2030814365 at qtp-1986936160-51'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1838022392 at qtp-1986936160-4'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1551156138 at qtp-1986936160-23'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1169983611 at qtp-1986936160-43'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '2018812712 at qtp-1986936160-56'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1209719856 at qtp-1986936160-41'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1032121412 at qtp-1986936160-38'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1813126941 at qtp-1986936160-26'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1144967167 at qtp-1986936160-15'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1765421434 at qtp-1986936160-47'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '43086831 at qtp-1986936160-21'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '703447155 at qtp-1986936160-25'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1382393727 at qtp-1986936160-16'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1393665909 at qtp-1986936160-19'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '618846953 at qtp-1986936160-11'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '821556544 at qtp-1986936160-9'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '2026789660 at qtp-1986936160-54'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '281555666 at qtp-1986936160-42'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '9814147 at qtp-1986936160-33'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1295986757 at qtp-1986936160-0'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1221696456 at qtp-1986936160-48'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '303731508 at qtp-1986936160-61'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '949026880 at qtp-1986936160-27'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '961725657 at qtp-1986936160-22'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '900409598 at qtp-1986936160-53'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1145518399 at qtp-1986936160-45'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '305035296 at qtp-1986936160-60'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1594958326 at qtp-1986936160-8'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1752918153 at qtp-1986936160-35'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1034573819 at qtp-1986936160-46'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1377877625 at qtp-1986936160-36'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1535043768 at qtp-1986936160-50'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1702714666 at qtp-1986936160-32'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1649966228 at qtp-1986936160-59'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '852031224 at qtp-1986936160-12'}
at java.nio.Bits.copyToByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:191)
at sun.nio.ch.IOUtil.read(IOUtil.java:209)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:131)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:290)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 5, 'state': 'TIMED_WAITING', 'name': 'Timer-0'}
{'priority': 5, 'state': 'TIMED_WAITING', 'name': 'Timer-1'}
at java.lang.Object.wait(Object.java:?)
at java.util.TimerThread.mainLoop(Timer.java:509)
at java.util.TimerThread.run(Timer.java:462)
{'priority': 5, 'state': 'RUNNABLE', 'name': '1720122918 at qtp-1986936160-66 - Acceptor0 InstrumentedConnector at 0.0.0.0:1247'}
{'priority': 5, 'state': 'RUNNABLE', 'name': '1833701635 at qtp-1986936160-64 - Acceptor0 InstrumentedConnector at 0.0.0.0:1249'}
{'priority': 5, 'state': 'RUNNABLE', 'name': '569616903 at qtp-1986936160-68 - Acceptor0 InstrumentedConnector at 0.0.0.0:1245'}
{'priority': 5, 'state': 'RUNNABLE', 'name': '2078955121 at qtp-1986936160-67 - Acceptor0 InstrumentedConnector at 0.0.0.0:1246'}
{'priority': 5, 'state': 'RUNNABLE', 'name': '391717236 at qtp-1986936160-65 - Acceptor0 InstrumentedConnector at 0.0.0.0:1248'}
at sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:?)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:459)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 1, 'state': 'RUNNABLE', 'name': '1949571424 at qtp-1986936160-55'}
{'priority': 1, 'state': 'RUNNABLE', 'name': '1743299062 at qtp-1986936160-57'}
at java.nio.Bits.copyToByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
at com.wol3.server.model.loader.Loader.getUnzippedByteBuffer(Loader.java:174)
at com.wol3.server.model.loader.Loader.load0(Loader.java:90)
at com.wol3.server.model.loader.Loader.load(Loader.java:68)
at com.wol3.server.model.loader.Loader.load(Loader.java:197)
at com.wol3.server.model.loader.CachedLoader.getCombatLog(CachedLoader.java:95)
at com.wol3.server.http.handler.DataAPIServlet.doGet(DataAPIServlet.java:46)
at com.wol3.server.http.handler.DataAPIServlet.doPost(DataAPIServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 5, 'state': 'WAITING', 'name': 'main'}
at java.lang.Object.wait(Object.java:?)
at java.lang.Object.wait(Object.java:485)
at org.mortbay.thread.QueuedThreadPool.join(QueuedThreadPool.java:298)
at org.mortbay.jetty.Server.join(Server.java:332)
at com.wol3.server.http.HttpServer.join(HttpServer.java:140)
at com.wol3.server.http.HttpServer.main(HttpServer.java:146)
{'priority': 5, 'state': 'TIMED_WAITING', 'name': 'CachedLoader-GC2-daemon'}
at java.lang.Thread.sleep(Thread.java:?)
at com.wol3.server.model.loader.CachedLoader$1.run(CachedLoader.java:35)
at java.lang.Thread.run(Thread.java:619)
{'priority': 10, 'state': 'WAITING', 'name': 'Reference Handler'}
at java.lang.Object.wait(Object.java:?)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
{'priority': 9, 'state': 'RUNNABLE', 'name': 'Signal Dispatcher'}
{'priority': 5, 'state': 'TIMED_WAITING', 'name': 'Thread-74'}
at java.lang.Thread.sleep(Thread.java:?)
at com.wol3.server.model.loader.CachedLoader$2.run(CachedLoader.java:59)
at java.lang.Thread.run(Thread.java:619)
{'priority': 8, 'state': 'WAITING', 'name': 'Finalizer'}
at java.lang.Object.wait(Object.java:?)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
{'priority': 10, 'state': 'RUNNABLE', 'name': '808460461 at qtp-274617771-0'}
at java.lang.Thread.dumpThreads(Thread.java:?)
at java.lang.Thread.getAllStackTraces(Thread.java:1487)
at com.wol3.server.http.handler.SystemServlet.doGet(SystemServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:915)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 1, 'state': 'RUNNABLE', 'name': '458505352 at qtp-1986936160-63'}
at java.nio.Bits.copyToByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:224)
at com.wol3.server.data.ChunkedBCLReader$2.read(ChunkedBCLReader.java:119)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:221)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at com.wol3.server.data.ChunkedBCLReader.inflateBuffer(ChunkedBCLReader.java:100)
at com.wol3.server.data.ChunkedBCLReader.readFrom(ChunkedBCLReader.java:66)
at com.wol3.server.model.loader.Loader.load0(Loader.java:96)
at com.wol3.server.model.loader.Loader.load(Loader.java:68)
at com.wol3.server.model.loader.Loader.load(Loader.java:197)
at com.wol3.server.model.loader.CachedLoader.getCombatLog(CachedLoader.java:95)
at com.wol3.server.http.handler.DataAPIServlet.doGet(DataAPIServlet.java:46)
at com.wol3.server.http.handler.DataAPIServlet.doPost(DataAPIServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 1, 'state': 'RUNNABLE', 'name': '1298336441 at qtp-1986936160-49'}
at java.nio.Bits.copyFromByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:314)
at org.mortbay.io.nio.DirectNIOBuffer.poke(DirectNIOBuffer.java:201)
at org.mortbay.io.nio.DirectNIOBuffer.poke(DirectNIOBuffer.java:141)
at org.mortbay.io.AbstractBuffer.put(AbstractBuffer.java:448)
at org.mortbay.jetty.HttpGenerator.addContent(HttpGenerator.java:148)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:644)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:109)
at org.mortbay.jetty.AbstractGenerator$OutputWriter.write(AbstractGenerator.java:903)
at java.io.PrintWriter.write(PrintWriter.java:382)
at com.wol3.server.http.handler.DataAPIHandler.writeResponse(DataAPIHandler.java:680)
at com.wol3.server.http.handler.DataAPIServlet.doGet(DataAPIServlet.java:70)
at com.wol3.server.http.handler.DataAPIServlet.doPost(DataAPIServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 10, 'state': 'RUNNABLE', 'name': '1806344089 at qtp-274617771-1 - Acceptor0 SocketConnector at 0.0.0.0:1250'}
at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:?)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:99)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
{'priority': 1, 'state': 'RUNNABLE', 'name': '1614281502 at qtp-1986936160-62'}
at java.nio.Bits.copyFromByteArray(Bits.java:?)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:314)
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:290)
at sun.nio.ch.IOUtil.write(IOUtil.java:70)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:203)
at com.wol3.server.model.postprocessing.ChartDataExporter.exportDefault(ChartDataExporter.java:291)
at com.wol3.server.model.loader.Loader.load0(Loader.java:126)
at com.wol3.server.model.loader.Loader.load(Loader.java:68)
at com.wol3.server.model.loader.Loader.load(Loader.java:197)
at com.wol3.server.model.loader.CachedLoader.getCombatLog(CachedLoader.java:95)
at com.wol3.server.http.handler.DataAPIServlet.doGet(DataAPIServlet.java:46)
at com.wol3.server.http.handler.DataAPIServlet.doPost(DataAPIServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:536)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:930)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
More information about the hotspot-gc-use
mailing list