Strange bug inside jetty at shenandoah/jdk8u

[Kirill A. Korinsky]

Hey,

I simplified my test case to
> public class DemoServer {
>     public static void main(String[] args) throws Exception {
>         Server server = new Server(80);
>         server.start();
>         server.join();
>     }
> }

and it generates 3-4 errors for 500 requests.

This code at same JVM without Shenandoah processed 1 million requests without this error.

--
wbr, Kirill


> On 20 Dec 2017, at 20:16, Kirill A. Korinsky <kirill at korins.ky> wrote:
> 
> Hey,
> 
> I made very simple test case: https://github.com/catap/jetty-shenandoah-error <https://github.com/catap/jetty-shenandoah-error>
> 
> It is durty but it is good enough.
> 
> How to reproduce:
> The first terminal:
>> mvn clean package
>> docker-compose up --build
> The second terminal:
>> ab -k -c 1 -n 10000 -p request_body http://localhost:8080/test/ <http://localhost:8080/test/>
> 
> and you should have a lot of errors likes:
>> server_1  | 2017-12-20 16:09:56.287:WARN:oejh.HttpParser:qtp1706377736-41: parse exception: java.lang.IndexOutOfBoundsException: 20 for HttpChannelOverHttp at 1138214c{r=4,c=false,a=IDLE,uri=null}
>> server_1  | java.lang.IndexOutOfBoundsException: 20
>> server_1  | 	at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
>> server_1  | 	at org.eclipse.jetty.http.HttpParser.parseLine(HttpParser.java:767)
>> server_1  | 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1310)
>> server_1  | 	at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:353)
>> server_1  | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:236)
>> server_1  | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>> server_1  | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>> server_1  | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
>> server_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
>> server_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
>> server_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
>> server_1  | 	at java.lang.Thread.run(Thread.java:748)
> 
> 
> If you remove -XX:+UseShenandoahGC from java arguments (at Dockerfile) it works fine. If you remove -XX:ShenandoahGCHeuristics=aggressive it will request a lot of requests to reproduce it.
> 
> --
> wbr, Kirill
> 
> 
>> On 20 Dec 2017, at 12:27, Aleksey Shipilev <shade at redhat.com <mailto:shade at redhat.com>> wrote:
>> 
>> On 12/20/2017 12:12 AM, Kirill A. Korinsky wrote:
>>> Well,
>>> 
>>> Simpler case includes decreasing heap to 1Gb.
>>> 
>>> I attached two file:
>>> - errors.log with jetty errors
>>> - gc.log with gc.log
>>> 
>>> It run at fedora 27 openjdk-1.8.0 without fast debug.
>>> 
>>> Same setting but without Shenandoah doest reproduce this.
>>> 
>>> If I increase heap size this issue is making rare. Since 6Gb heap I can't reproduce it. Maybe it help.
>> 
>> GC logs and application logs are useless to debug the failure like this.
>> 
>> We need the actual VM error or other kind of hs_err-generating smoking gun. Or, at least the MCVE
>> that fails and debuggable, so we can take a look deeper here. I assume something like stripped down
>> embedded Jetty with trivial response handler fails? Tried to do this [1], and Shenandoah/8u runs
>> fine -- what is missing there? Since you seem to have the application that fails reliably, could you
>> strip it down to bare essentials that still fails?
>> 
>> Thanks,
>> -Aleksey
>> 
>> [1] https://github.com/shipilev/jetty-test <https://github.com/shipilev/jetty-test>
>> 
>