Hotspot crash compiling sun.invoke.util.ValueConversions::identity

[Eirik Bjørsnøs]

Vladimir,

The problem does indeed seem to be fixed by 8u40. While the problem is
now consistently reproducible on u31, I cannot reproduce it on u40.

Skybar does add a ClassFileTransformer which instruments byte code for
line coverage analysis. (However we never retransform already loaded
classes).

I think we'll consider this problem solved for now on our side.

Thanks,
Eirik.

On Tue, Mar 3, 2015 at 7:23 PM, Vladimir Ivanov
<vladimir.x.ivanov at oracle.com> wrote:
> Sorry, it turns out the bug is confidential.
>
> Here's the fix (part of 8u40):
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/aff6ccb506cb
>
> I'd suggest you to try 8u40 early access binaries [1] to check that the
> problem goes away.
>
> Best regards,
> Vladimir Ivanov
>
> [1] https://jdk8.java.net/download.html
>
>
> On 3/3/15 9:18 PM, Vladimir Ivanov wrote:
>>
>> Eirik,
>>
>> Thanks a lot for the reproducer!
>>
>> With fastdebug binaries it fails with the following error:
>>     #  assert(false) failed: DEBUG MESSAGE: i2c adapter must return to
>> an interpreter frame
>>
>> I wasn't able to reproduce the problem neither with 8u40 nor 9.
>>
>> It should be JDK-8056154 [1].
>>
>> Do you do class redefinition in the agent?
>>
>> Best regards,
>> Vladimir Ivanov
>>
>> [1] https://bugs.openjdk.java.net/browse/JDK-8056154
>>
>> On 3/2/15 10:22 PM, Eirik Bjørsnøs wrote:
>>>
>>> This seems to reproduce the issue reliably on 1.8.0_31 on my Mac OS X
>>> Maverics:
>>>
>>> # TERMINAL A:
>>>
>>> git clone https://github.com/WinterTechForum/skybar.git
>>> git clone https://github.com/WinterTechForum/skybar-demo.git
>>> cd skybar
>>> git checkout 7eef1b8
>>> ./gradlew shadowJar
>>> cd ../skybar-demo/
>>> mvn clean install
>>> java -Dskybar.includedPackage=com/skybar
>>> -javaagent:../skybar/build/libs/skybar-1.0-SNAPSHOT-all.jar -jar
>>> target/skybar-demo-1.0-SNAPSHOT-jetty-console.war --headless
>>>
>>> # Terminal B:
>>>
>>> curl "http://localhost:8080/message?[1-100000]"
>>>
>>> (Let run for a minute or two)
>>>
>>> # Terminal C:
>>> curl "http://localhost:4321/coverage.json?[1-100000]"
>>>
>>> On Mon, Mar 2, 2015 at 4:26 PM, Vladimir Ivanov
>>> <vladimir.x.ivanov at oracle.com> wrote:
>>>>
>>>> Thanks, Eirik!
>>>>
>>>>> I have an hs_err file ready with a corresponding core dump file
>>>>> (176MB).
>>>>>
>>>>> I'll zip these up and send a download link to you off-list.
>>>>>
>>>>> Let me know if there is a better way to manage these files. Should we
>>>>> have a Jira issue to track progress?
>>>>
>>>>
>>>> Yes, I plan to file one once I make sure it's not a duplicate.
>>>>
>>>> Best regards,
>>>> Vladimir Ivanov
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Eirik.
>>>>>
>>>>> On Mon, Mar 2, 2015 at 4:01 PM, Vladimir Ivanov
>>>>> <vladimir.x.ivanov at oracle.com> wrote:
>>>>>>
>>>>>>
>>>>>> Eirik,
>>>>>>
>>>>>> Thanks for the report!
>>>>>>
>>>>>> As a first step, I'd like to see corresponding hotspot crash log
>>>>>> (hs_err_pid*.log).
>>>>>>
>>>>>> It seems you stepped on a bug in invokedynamic implementation (lambdas
>>>>>> are
>>>>>> translated into indy calls on bytecode level) which happens when it is
>>>>>> initialized too early in VM lifecycle (e.g. java agent premain).
>>>>>>
>>>>>> I'd be grateful for a stand-alone test case (focus on premain method
>>>>>> logic)
>>>>>> or steps how to reproduce it myself.
>>>>>>
>>>>>> Best regards,
>>>>>> Vladimir Ivanov
>>>>>>
>>>>>> On 3/2/15 4:10 AM, Eirik Bjørsnøs wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ladies and gentlemen,
>>>>>>>
>>>>>>> Last week, some of the world's brightest, most capable and all round
>>>>>>> good looking Java developers gathered in Crested Butte, Colorado for
>>>>>>> the Winter Tech Forum (WTF) 2015 [1].
>>>>>>>
>>>>>>> Wednesday was Hackathon Day, and a group of us decided to embark the
>>>>>>> adventure of creating a live Java code coverage engine called Skybar
>>>>>>> [2].
>>>>>>>
>>>>>>> We made good progress and by the afternoon we knew we'd have
>>>>>>> something
>>>>>>> good to show at the 8pm showcase. So we confidently headed over to
>>>>>>> Ryce Asian Bistro [3] at 17:30 for a nice team dinner.
>>>>>>>
>>>>>>> Luckily we went easy on the saké, because when we came back to
>>>>>>> finalise our demo, we found that Hotspot had crashed:
>>>>>>>
>>>>>>> # A fatal error has been detected by the Java Runtime Environment:
>>>>>>> #
>>>>>>> #  SIGBUS (0xa) at pc=0x000000012a6d61e6, pid=10738, tid=18691
>>>>>>> [..]
>>>>>>> Compiled method (c2)  152453 1542       4
>>>>>>> sun.invoke.util.ValueConversions::identity (2 bytes)
>>>>>>>
>>>>>>> Shock, horror! Our demo was just minutes away, would we be able to
>>>>>>> present? Our team immediately went into debugging mode:
>>>>>>>
>>>>>>> Could we reproduce without using our -javaagent? No. Was the crash
>>>>>>> consistently reproducible? Kind of, but we had to stress test our
>>>>>>> code. Was there a any consistency in the hs_err reports? Yes, it
>>>>>>> seemed to always involve lambda expressions in out agent code.
>>>>>>>
>>>>>>> The team decided we had to remove all lambdas from our agent code in
>>>>>>> the hope that would improve the situation. And it did! At 7:50 PM, we
>>>>>>> had a stable build and ran out to make it to the presentations.
>>>>>>>
>>>>>>> Skybar was very well received. Our demo worked. Nothing crashed.
>>>>>>> Applause and shouts of "Ship it!" was heard from the audience. It was
>>>>>>> good times. We headed over to the Princess Wine Bar [4] for some
>>>>>>> celebratory drinks.
>>>>>>>
>>>>>>> The team is continuing our work on Skybar. But we miss our lamdas,
>>>>>>> and
>>>>>>> we want them back! To achieve this, we need help from the Hotspot
>>>>>>> team. On our side, we are willing to  work with you try to make
>>>>>>> reproducible test cases and collect any other useful information.
>>>>>>>
>>>>>>> What would be the best next step in saving Lambas on Agents? We would
>>>>>>> love to help out fixing this. Should we file an issue somewhere and
>>>>>>> upload our hs_err file(s)?
>>>>>>>
>>>>>>> On behalf of the Skybar team -
>>>>>>> Eirik,
>>>>>>>
>>>>>>> [1] http://www.mindviewinc.com/Conferences/JavaPosseRoundup/
>>>>>>> [2] https://github.com/WinterTechForum/skybar
>>>>>>> [3] http://www.ryceasianbistro.com
>>>>>>> [4] https://www.facebook.com/princesswinebar
>>>>>>>
>>>>>>
>>>>
>