Submitting bugs in loom branch

Sat Sep 21 19:46:26 UTC 2019

Hi,

A little update from my side on this. I made some tweaks in the code, the
most important, I guess, is increasing the number of executor's threads
<https://github.com/jigga/loom-academy/blob/master/workshop-service/src/main/java/loom/workshop/SimpleExecutor.java#L25>
(I was unable to reproduce the issue with single-threaded executor).

Moreover, I updated tomcat's SocketProcessorBase run method implementation
<https://github.com/jigga/tomcat/blob/fibers/java/org/apache/tomcat/util/net/SocketProcessorBase.java>
as
the previous one was rather naive.

With all these changes, the workshop-service VM crashes almost every time
it is hit with the attack script.

I added some more hs_err*
<https://github.com/jigga/loom-academy/tree/master/workshop-service> files
to the repo.

Arek

On Tue, Sep 17, 2019 at 8:29 PM Arkadiusz Gasiński <jigga at jigga.pl> wrote:

> Well, ok then, will try to describe here how to reproduce the issue.
>
> To give you some context, I wanted to test how Fiber friendly Tomcat is
> (actually spring boot app running with embedded Tomcat). Turned out it is
> not that friendly when it comes to Fibers as all HTTP request processing is
> done within the synchronized block. There's this
> NioEndpoint$SocketProcessor
> <https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/NioEndpoint.java>
> class that extends from SocketProcessorBase
> <https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/SocketProcessorBase.java>
> (Runnable), whose whole run method is within synchronized block, which, if
> I understand correctly, pins http threads.
>
> So I forked tomcat and made the following change
> <https://github.com/jigga/tomcat/commit/3bc1becdf50f20580fdf3333181e318f5baed59b>
> in the SocketProcessorBase class, built it locally and it appears that it
> did the trick - Tomcat became Fiber friendly :)
>
> So how did I tested it...
>
> I have this sample github project <https://github.com/jigga/loom-academy>
> I created some time ago to experiment a bit with Fibers. It consists of 3
> simple microservices: discovery-service, echo-service and workshop-service.
> The idea is to have more realistic setup than just an endpoint that sleeps
> the current request serving thread for some time, so in this case you call
> the workshop-service (either http://localhost:8080/loom/test - request is
> processed in Fiber or http://localhost:8081/loom/test - request is
> processed in Thread), which in turn uses discovery-service to look up
> echo-service (it starts on some random port) and then calls its
> /delay/{timeout} endpoint, where timeout is between 0 and 1000 ms. The
> echo-service echoes back the request with the given delay.
>
> Before you actually checkout this project, you first need to clone my
> Tomcat's fork and build it locally, by calling "*ant deploy*" and then "*ant
> embed*". You then need to copy the following jars from *output/embed* dir
> to your local maven repo. You also need to append the version number
> (9.0.27) to these jars as the build does not do it.
>
>    - tomcat-embed-core.jar, rename to *tomcat-embed-core-9.0.27.jar* and
>    copy to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.27
>    (or wherever your local maven repo is)
>    - tomcat-embed-el.jar, rename to *tomcat-embed-el-9.0.27.jar* and copy
>    to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-el/9.0.27
>    - tomcat-embed-websocket.jar, rename to
>    *tomcat-embed-websocket-9.0.27.jar* and copy
>    to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-websocket/9.0.27
>
> This sample project of mine is a gradle project, so you also need to make
> sure that gradle is configured to use loom build, I have the following in
> my cat ~/.gradle/gradle.properties:
>
>
> org.gradle.java.home=/Library/Java/JavaVirtualMachines/jdk-14.jdk/Contents/Home
>
> where obviously jdk-14.jdk is a loom build.
>
> You should be good to checkout the project now, and once it's done, you
> should start all 3 services, preferably in separate terminals like so:
>
> ./gradlew discovery-service:bootRun
> ./gradlew echo-service:bootRun
> ./gradlew workshop-service:bootRun
>
> Once started, you can then throw some load at workshop-service using
> provided attack.sh script (for it to work you actually need to have Vegeta
> <https://github.com/tsenart/vegeta> installed locally). You call the
> script with either fibers or threads argument. As already mentioned above,
> this service has 2 connectors, namely:
>
>    - 8080 - request processed in Fiber
>    - 8081 - request processed in Thread
>
> Both connectors are configured with single threaded executors (
> SimpleExecutor
> <https://github.com/jigga/loom-academy/blob/master/workshop-service/src/main/java/loom/workshop/SimpleExecutor.java>
> class).
>
> The single-threaded fiber connector works with 100% success rate with the
> rate up to about 375 requests per second on my machine. The thread based
> connector is obviously waaay less performant.
>
> jiggbook-1:loom-academy jigga$ ./attack.sh fibers
>> Requests      [total, rate]            45000, 375.00
>> Duration      [total, attack, wait]    2m0.863797902s, 2m0.000299s,
>> 863.498902ms
>> Latencies     [mean, 50, 95, 99, max]  505.184513ms, 503.86826ms,
>> 957.762176ms, 996.938898ms, 1.056817859s
>> Bytes In      [total, mean]            10165188, 225.89
>> Bytes Out     [total, mean]            0, 0.00
>> Success       [ratio]                  100.00%
>> Status Codes  [code:count]             200:45000
>> Error Set:
>
>
> So the issues I'm actually facing are that the VM in which
> workshop-service is running occasionally crashes. It does not happen on
> every run, but happened several times today. I included two hs_err* files
> <https://github.com/jigga/loom-academy/tree/master/workshop-service> in
> the repo. I believe that this may be GC-related issue as the Current thread
> is always GCTaskThread.
>
> I updated and built the jdk loom on the 2nd of September, have not rebuilt
> since then.
>
> Hope that you'll be able to reproduce it given the above info.
>
> If you need additional info, just let me know.
>
> Thanks,
> Arek
>
> On Tue, Sep 17, 2019 at 12:03 PM Rickard Bäckman <
> rickard.backman at oracle.com> wrote:
>
>> Feel free to report them here right now.
>> If you can provide a reproducer it's great!
>>
>> Thank you!
>>
>> /R
>>
>> On 09/17, Arkadiusz Gasiński wrote:
>> > Hi,
>> >
>> > Where do I submit bug reports related to loom-JVM crashes?
>> >
>> > The JVM prints this when it crashes:
>> >
>> > #
>> > # If you would like to submit a bug report, please visit:
>> > #   http://bugreport.java.com/bugreport/crash.jsp
>> > #
>> >
>> > But I just wonder if it's not too early for loom crashes to be reported
>> > there?
>> >
>> > Thanks,
>> > Arek
>>
>