Submitting bugs in loom branch

[Arkadiusz Gasiński]

Well, ok then, will try to describe here how to reproduce the issue.

To give you some context, I wanted to test how Fiber friendly Tomcat is
(actually spring boot app running with embedded Tomcat). Turned out it is
not that friendly when it comes to Fibers as all HTTP request processing is
done within the synchronized block. There's this NioEndpoint$SocketProcessor
<https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/NioEndpoint.java>
class that extends from SocketProcessorBase
<https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/SocketProcessorBase.java>
(Runnable), whose whole run method is within synchronized block, which, if
I understand correctly, pins http threads.

So I forked tomcat and made the following change
<https://github.com/jigga/tomcat/commit/3bc1becdf50f20580fdf3333181e318f5baed59b>
in the SocketProcessorBase class, built it locally and it appears that it
did the trick - Tomcat became Fiber friendly :)

So how did I tested it...

I have this sample github project <https://github.com/jigga/loom-academy> I
created some time ago to experiment a bit with Fibers. It consists of 3
simple microservices: discovery-service, echo-service and workshop-service.
The idea is to have more realistic setup than just an endpoint that sleeps
the current request serving thread for some time, so in this case you call
the workshop-service (either http://localhost:8080/loom/test - request is
processed in Fiber or http://localhost:8081/loom/test - request is
processed in Thread), which in turn uses discovery-service to look up
echo-service (it starts on some random port) and then calls its
/delay/{timeout} endpoint, where timeout is between 0 and 1000 ms. The
echo-service echoes back the request with the given delay.

Before you actually checkout this project, you first need to clone my
Tomcat's fork and build it locally, by calling "*ant deploy*" and then "*ant
embed*". You then need to copy the following jars from *output/embed* dir
to your local maven repo. You also need to append the version number
(9.0.27) to these jars as the build does not do it.

   - tomcat-embed-core.jar, rename to *tomcat-embed-core-9.0.27.jar* and
   copy to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.27
   (or wherever your local maven repo is)
   - tomcat-embed-el.jar, rename to *tomcat-embed-el-9.0.27.jar* and copy
   to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-el/9.0.27
   - tomcat-embed-websocket.jar, rename to
   *tomcat-embed-websocket-9.0.27.jar* and copy
   to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-websocket/9.0.27

This sample project of mine is a gradle project, so you also need to make
sure that gradle is configured to use loom build, I have the following in
my cat ~/.gradle/gradle.properties:

org.gradle.java.home=/Library/Java/JavaVirtualMachines/jdk-14.jdk/Contents/Home

where obviously jdk-14.jdk is a loom build.

You should be good to checkout the project now, and once it's done, you
should start all 3 services, preferably in separate terminals like so:

./gradlew discovery-service:bootRun
./gradlew echo-service:bootRun
./gradlew workshop-service:bootRun

Once started, you can then throw some load at workshop-service using
provided attack.sh script (for it to work you actually need to have Vegeta
<https://github.com/tsenart/vegeta> installed locally). You call the script
with either fibers or threads argument. As already mentioned above, this
service has 2 connectors, namely:

   - 8080 - request processed in Fiber
   - 8081 - request processed in Thread

Both connectors are configured with single threaded executors (
SimpleExecutor
<https://github.com/jigga/loom-academy/blob/master/workshop-service/src/main/java/loom/workshop/SimpleExecutor.java>
class).

The single-threaded fiber connector works with 100% success rate with the
rate up to about 375 requests per second on my machine. The thread based
connector is obviously waaay less performant.

jiggbook-1:loom-academy jigga$ ./attack.sh fibers
> Requests      [total, rate]            45000, 375.00
> Duration      [total, attack, wait]    2m0.863797902s, 2m0.000299s,
> 863.498902ms
> Latencies     [mean, 50, 95, 99, max]  505.184513ms, 503.86826ms,
> 957.762176ms, 996.938898ms, 1.056817859s
> Bytes In      [total, mean]            10165188, 225.89
> Bytes Out     [total, mean]            0, 0.00
> Success       [ratio]                  100.00%
> Status Codes  [code:count]             200:45000
> Error Set:


So the issues I'm actually facing are that the VM in which workshop-service
is running occasionally crashes. It does not happen on every run, but
happened several times today. I included two hs_err* files
<https://github.com/jigga/loom-academy/tree/master/workshop-service> in the
repo. I believe that this may be GC-related issue as the Current thread is
always GCTaskThread.

I updated and built the jdk loom on the 2nd of September, have not rebuilt
since then.

Hope that you'll be able to reproduce it given the above info.

If you need additional info, just let me know.

Thanks,
Arek

On Tue, Sep 17, 2019 at 12:03 PM Rickard Bäckman <rickard.backman at oracle.com>
wrote:

> Feel free to report them here right now.
> If you can provide a reproducer it's great!
>
> Thank you!
>
> /R
>
> On 09/17, Arkadiusz Gasiński wrote:
> > Hi,
> >
> > Where do I submit bug reports related to loom-JVM crashes?
> >
> > The JVM prints this when it crashes:
> >
> > #
> > # If you would like to submit a bug report, please visit:
> > #   http://bugreport.java.com/bugreport/crash.jsp
> > #
> >
> > But I just wonder if it's not too early for loom crashes to be reported
> > there?
> >
> > Thanks,
> > Arek
>