Submitting bugs in loom branch
Ron Pressler
ron.pressler at oracle.com
Sat Oct 5 19:11:14 UTC 2019
Hi.
We appreciate these bug reports, and would encourage you to submit more if you find them
(and simple reproductions are especially appreciated), but the implementation is undergoing
some major changes, so we may prioritise some new work over looking into bugs.
So while we are collecting the reports, they may not be addressed in a timely manner.
Thank you again — for reporting issues as well as understanding.
Ron
On 5 October 2019 at 19:58:42, Arkadiusz Gasiński (jigga at jigga.pl) wrote:
Hi,
Has anyone had a chance to look into that? I do realize that the setup
required to reproduce the issue is a bit burdensome and I'll try to prepare
something simpler in the coming days, but essentially at the moment the VM
in which the load-tested service runs crashes every time when the request
rate is around 10 per second and above. Interestingly, with the latest
changes in the fibers branch, the current thread in the generated
hs_err_pid* files is now *fiber-carrier-thread-X* rather than a GC thread.
Thanks,
Arek
On Sat, Sep 21, 2019 at 10:46 PM Arkadiusz Gasiński <jigga at jigga.pl> wrote:
> Hi,
>
> A little update from my side on this. I made some tweaks in the code, the
> most important, I guess, is increasing the number of executor's threads
> <https://github.com/jigga/loom-academy/blob/master/workshop-service/src/main/java/loom/workshop/SimpleExecutor.java#L25>
> (I was unable to reproduce the issue with single-threaded executor).
>
> Moreover, I updated tomcat's SocketProcessorBase run method implementation
> <https://github.com/jigga/tomcat/blob/fibers/java/org/apache/tomcat/util/net/SocketProcessorBase.java> as
> the previous one was rather naive.
>
> With all these changes, the workshop-service VM crashes almost every time
> it is hit with the attack script.
>
> I added some more hs_err*
> <https://github.com/jigga/loom-academy/tree/master/workshop-service>
> files to the repo.
>
> Arek
>
>
> On Tue, Sep 17, 2019 at 8:29 PM Arkadiusz Gasiński <jigga at jigga.pl> wrote:
>
>> Well, ok then, will try to describe here how to reproduce the issue.
>>
>> To give you some context, I wanted to test how Fiber friendly Tomcat is
>> (actually spring boot app running with embedded Tomcat). Turned out it is
>> not that friendly when it comes to Fibers as all HTTP request processing is
>> done within the synchronized block. There's this
>> NioEndpoint$SocketProcessor
>> <https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/NioEndpoint.java>
>> class that extends from SocketProcessorBase
>> <https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/SocketProcessorBase.java>
>> (Runnable), whose whole run method is within synchronized block, which, if
>> I understand correctly, pins http threads.
>>
>> So I forked tomcat and made the following change
>> <https://github.com/jigga/tomcat/commit/3bc1becdf50f20580fdf3333181e318f5baed59b>
>> in the SocketProcessorBase class, built it locally and it appears that it
>> did the trick - Tomcat became Fiber friendly :)
>>
>> So how did I tested it...
>>
>> I have this sample github project <https://github.com/jigga/loom-academy>
>> I created some time ago to experiment a bit with Fibers. It consists of 3
>> simple microservices: discovery-service, echo-service and workshop-service.
>> The idea is to have more realistic setup than just an endpoint that sleeps
>> the current request serving thread for some time, so in this case you call
>> the workshop-service (either http://localhost:8080/loom/test - request
>> is processed in Fiber or http://localhost:8081/loom/test - request is
>> processed in Thread), which in turn uses discovery-service to look up
>> echo-service (it starts on some random port) and then calls its
>> /delay/{timeout} endpoint, where timeout is between 0 and 1000 ms. The
>> echo-service echoes back the request with the given delay.
>>
>> Before you actually checkout this project, you first need to clone my
>> Tomcat's fork and build it locally, by calling "*ant deploy*" and then "*ant
>> embed*". You then need to copy the following jars from *output/embed* dir
>> to your local maven repo. You also need to append the version number
>> (9.0.27) to these jars as the build does not do it.
>>
>> - tomcat-embed-core.jar, rename to *tomcat-embed-core-9.0.27.jar* and
>> copy to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.27
>> (or wherever your local maven repo is)
>> - tomcat-embed-el.jar, rename to *tomcat-embed-el-9.0.27.jar* and
>> copy to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-el/9.0.27
>> - tomcat-embed-websocket.jar, rename to
>> *tomcat-embed-websocket-9.0.27.jar* and copy
>> to ~/.m2/repository/org/apache/tomcat/embed/tomcat-embed-websocket/9.0.27
>>
>> This sample project of mine is a gradle project, so you also need to make
>> sure that gradle is configured to use loom build, I have the following in
>> my cat ~/.gradle/gradle.properties:
>>
>>
>> org.gradle.java.home=/Library/Java/JavaVirtualMachines/jdk-14.jdk/Contents/Home
>>
>> where obviously jdk-14.jdk is a loom build.
>>
>> You should be good to checkout the project now, and once it's done, you
>> should start all 3 services, preferably in separate terminals like so:
>>
>> ./gradlew discovery-service:bootRun
>> ./gradlew echo-service:bootRun
>> ./gradlew workshop-service:bootRun
>>
>> Once started, you can then throw some load at workshop-service using
>> provided attack.sh script (for it to work you actually need to have
>> Vegeta <https://github.com/tsenart/vegeta> installed locally). You call
>> the script with either fibers or threads argument. As already mentioned
>> above, this service has 2 connectors, namely:
>>
>> - 8080 - request processed in Fiber
>> - 8081 - request processed in Thread
>>
>> Both connectors are configured with single threaded executors (
>> SimpleExecutor
>> <https://github.com/jigga/loom-academy/blob/master/workshop-service/src/main/java/loom/workshop/SimpleExecutor.java>
>> class).
>>
>> The single-threaded fiber connector works with 100% success rate with the
>> rate up to about 375 requests per second on my machine. The thread based
>> connector is obviously waaay less performant.
>>
>> jiggbook-1:loom-academy jigga$ ./attack.sh fibers
>>> Requests [total, rate] 45000, 375.00
>>> Duration [total, attack, wait] 2m0.863797902s, 2m0.000299s,
>>> 863.498902ms
>>> Latencies [mean, 50, 95, 99, max] 505.184513ms, 503.86826ms,
>>> 957.762176ms, 996.938898ms, 1.056817859s
>>> Bytes In [total, mean] 10165188, 225.89
>>> Bytes Out [total, mean] 0, 0.00
>>> Success [ratio] 100.00%
>>> Status Codes [code:count] 200:45000
>>> Error Set:
>>
>>
>> So the issues I'm actually facing are that the VM in which
>> workshop-service is running occasionally crashes. It does not happen on
>> every run, but happened several times today. I included two hs_err* files
>> <https://github.com/jigga/loom-academy/tree/master/workshop-service> in
>> the repo. I believe that this may be GC-related issue as the Current thread
>> is always GCTaskThread.
>>
>> I updated and built the jdk loom on the 2nd of September, have not
>> rebuilt since then.
>>
>> Hope that you'll be able to reproduce it given the above info.
>>
>> If you need additional info, just let me know.
>>
>> Thanks,
>> Arek
>>
>> On Tue, Sep 17, 2019 at 12:03 PM Rickard Bäckman <
>> rickard.backman at oracle.com> wrote:
>>
>>> Feel free to report them here right now.
>>> If you can provide a reproducer it's great!
>>>
>>> Thank you!
>>>
>>> /R
>>>
>>> On 09/17, Arkadiusz Gasiński wrote:
>>> > Hi,
>>> >
>>> > Where do I submit bug reports related to loom-JVM crashes?
>>> >
>>> > The JVM prints this when it crashes:
>>> >
>>> > #
>>> > # If you would like to submit a bug report, please visit:
>>> > # http://bugreport.java.com/bugreport/crash.jsp
>>> > #
>>> >
>>> > But I just wonder if it's not too early for loom crashes to be reported
>>> > there?
>>> >
>>> > Thanks,
>>> > Arek
>>>
>>
More information about the loom-dev
mailing list