Strange bug inside jetty at shenandoah/jdk8u

[Kirill A. Korinsky]

Hey,

I made very simple test case: https://github.com/catap/jetty-shenandoah-error <https://github.com/catap/jetty-shenandoah-error>

It is durty but it is good enough.

How to reproduce:
The first terminal:
> mvn clean package
> docker-compose up --build
The second terminal:
> ab -k -c 1 -n 10000 -p request_body http://localhost:8080/test/


and you should have a lot of errors likes:
> server_1  | 2017-12-20 16:09:56.287:WARN:oejh.HttpParser:qtp1706377736-41: parse exception: java.lang.IndexOutOfBoundsException: 20 for HttpChannelOverHttp at 1138214c{r=4,c=false,a=IDLE,uri=null}
> server_1  | java.lang.IndexOutOfBoundsException: 20
> server_1  | 	at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> server_1  | 	at org.eclipse.jetty.http.HttpParser.parseLine(HttpParser.java:767)
> server_1  | 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1310)
> server_1  | 	at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:353)
> server_1  | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:236)
> server_1  | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> server_1  | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> server_1  | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> server_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> server_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> server_1  | 	at java.lang.Thread.run(Thread.java:748)


If you remove -XX:+UseShenandoahGC from java arguments (at Dockerfile) it works fine. If you remove -XX:ShenandoahGCHeuristics=aggressive it will request a lot of requests to reproduce it.

--
wbr, Kirill


> On 20 Dec 2017, at 12:27, Aleksey Shipilev <shade at redhat.com> wrote:
> 
> On 12/20/2017 12:12 AM, Kirill A. Korinsky wrote:
>> Well,
>> 
>> Simpler case includes decreasing heap to 1Gb.
>> 
>> I attached two file:
>> - errors.log with jetty errors
>> - gc.log with gc.log
>> 
>> It run at fedora 27 openjdk-1.8.0 without fast debug.
>> 
>> Same setting but without Shenandoah doest reproduce this.
>> 
>> If I increase heap size this issue is making rare. Since 6Gb heap I can't reproduce it. Maybe it help.
> 
> GC logs and application logs are useless to debug the failure like this.
> 
> We need the actual VM error or other kind of hs_err-generating smoking gun. Or, at least the MCVE
> that fails and debuggable, so we can take a look deeper here. I assume something like stripped down
> embedded Jetty with trivial response handler fails? Tried to do this [1], and Shenandoah/8u runs
> fine -- what is missing there? Since you seem to have the application that fails reliably, could you
> strip it down to bare essentials that still fails?
> 
> Thanks,
> -Aleksey
> 
> [1] https://github.com/shipilev/jetty-test
>