Graal and Clojure

Mon Apr 7 06:20:02 UTC 2014

Good news - I reapplied the -XX:-UseHSAILDeoptimization flag (last time 
I tried it the code had not yet made it into my jdk8/graal build) and a 
lot of the instability mentioned in my last mail disappeared - 
particularly in the simulator. Many more tests also passed :-)

http://ouroboros.dyndns-free.com/ci/job/clumatra-hardware-acceleration/
http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-simulated-acceleration/

Jules

On 06/04/14 15:22, Jules Gosnell wrote:
> So, Guys,
>
> Here is the promised update on Clumatra - an effort to get Clojure up
> and running on top of Graal/Okra/HSA.
>
> - I now have a working Ubuntu/AMD HSA box.
>
> - I have a Jenkins nightly build of JDK8 / Graal :
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/graal/
>
> - if useful, you can download the latest linux/64 tgz here (it is 300mb
> - I'm going to look at shrinking it:
>
> http://ouroboros.dyndns-free.com/clumatra/jdk1.8.0-graal.tgz
>
> - I have builds of Clumatra without, with simulated and with hardware
> based HSA :
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/
>
> - The clumatra testsuite is growing - results have been a bit unstable
> recently as I learnt which tests completely take out or hang the
> jdk8/graal jvm and which can be trusted to succeed / fail in a well
> behaved fashion:
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/
>
>
> I am working on a reflection/macro based solution, which will
> automagically create tests for all methods of all classes that I want
> tests for. When I bring this online in a couple of weeks the number of
> tests should jump dramatically.
>
> Please feel free to inspect test results etc on my Jenkins and let me
> know whether they are helpful and whether they can be made more useful
> in any way. currently some of them seem to indicate problems in the okra
> backend. This is when graal seems to run happily but I get back the
> wrong result from my test:
>
> e.g.
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_unchecked_inc_test/
>
>
> I run the test via graal/okra twice and once in pure clojure - so you
> see three results. The last one is the expected one, the first two the
> ones returned from the GPU.
>
> Other tests let me know that graal is not yet implementing various
> features of jvm bytecode - e.g.
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_my_static_inc_test/
>
>
> Unfortunately the test report does not seem to be capturing the HSAIL
> logging that you can see in the console output... My guess is that it is
> captured by redirecting clojure's usual output stream, but this is not
> the same one as being used by Graal...
>
> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/console
>
>
>
> That's enough for now. I'll post with more progress as and when.
>
>
>
> Jules
>
>
>
> On 31/03/14 23:40, Caspole, Eric wrote:
>> Hi Jules,
>> I updated everything and got Clumatra running on the real hardware today.
>>
>> In project.clj, I used this:
>>
>>    :jvm-opts ["-server" "-ea" "-esa" "-Xms1g" "-Xmx1g"
>>               "-verbosegc" "-G:Log=CodeGen"
>>               "-XX:+GPUOffload" "-XX:+TraceGPUInteraction"]
>>
>> Then make sure your /path/to/okra/dist/bin is in the PATH and
>> LD_LIBRARY_PATH.
>>
>> When it is using the simulator, you will see this:
>> [HSAIL] using
>> _OKRA_SIM_LIB_PATH_=/tmp/okraresource.dir_2595501614013009358/libokra_x86_64.so
>>
>>
>> With hardware or external okra you will see this:
>>
>> [HSAIL] library is libokra_x86_64.so
>>
>> Regards,
>> Eric
>>
>>
>> ________________________________________
>> From: Jules Gosnell [jules_gosnell at yahoo.com]
>> Sent: Thursday, March 20, 2014 5:27 PM
>> To: Caspole, Eric; graal-dev at openjdk.java.net;
>> sumatra-dev at openjdk.java.net
>> Subject: Re: Graal and Clojure
>>
>> Eric,
>>
>> I just checked support to Clumatra for running individual unit tests
>> from the CLI - should be enough to get you set up with your debugger.
>>
>> All details at the bottom of the README.
>>
>> https://github.com/JulesGosnell/clumatra
>>
>> I'm afraid you will need to install maven if you don't already have it -
>> for some reason lein does not seem to have a test-jar target.
>>
>> It is still a little rough around the edges, but better than nothing :-)
>>
>> I also just bought my HSA h/w - should be up and running next week
>> sometime. Did you figure out how to enable native-mode Okra ?
>>
>> catch you later,
>>
>>
>> Jules
>>
>>
>>
>> On 17/03/14 22:20, Eric Caspole wrote:
>>> Hi everybody,
>>> I was setting up a new HSA system today and I installed Clumatra and got
>>> it working under the simulator. I have a real HSA system not too
>>> different from this:
>>>
>>> http://code.google.com/p/aparapi/wiki/SettingUpLinuxHSAMachineForAparapi
>>>
>>> In the clojure/lein/maven based system there is so much harness I can't
>>> immediately see how to switch in the real hardware version of Okra
>>> instead of the simulator Okra that gets installed into maven and
>>> exploded in /tmp when the tests run.
>>>
>>> Jules, could you show how to bypass the harness etc to switch the Okra?
>>> Can I just run 1 test with a simple java command line? This makes it a
>>> lot easier to use a debugger which I will definitely want to do.
>>>
>>> Otherwise this is a cool project and we already found 1 or 2 issues we
>>> can fix in HSAIL Graal related to your clojure tests.
>>> Regards,
>>> Eric
>>>
>>>
>>>
>>> On 03/17/2014 06:43 AM, Doug Simon wrote:
>>>> Hi Julian,
>>>>
>>>> In terms of what to test, I’ll leave it up to the Sumatra developers
>>>> at AMD and Oracle to comment on what may be interesting.
>>>>
>>>> As far as I know, no one in the Graal team is that familiar with
>>>> Clojure. This actually makes the value of having Clojure tests
>>>> integrated into our gate system somewhat problematic -
>>>> debugging/fixing failures may be difficult. I’d suggest you report
>>>> failures your discover on the sumatra-dev and graal-dev lists and we
>>>> can resolve them that way for now.
>>>>
>>>> In addition, for Graal we’re always interested in benchmark suites for
>>>> JVM hosted languages. Can you recommend any well known and trusted
>>>> Clojure benchmarks we should consider?
>>>>
>>>> -Doug
>>>>
>>>> On Mar 17, 2014, at 12:35 AM, Jules Gosnell <jules_gosnell at yahoo.com>
>>>> wrote:
>>>>
>>>>> Doug,
>>>>>
>>>>> Sorry it has taken me a while to get back - I've been busy playing
>>>>> with code.
>>>>>
>>>>> I currently have a small clojure testsuite up and running. I plan to
>>>>> broaden this and then think about providing it in a junit-compatible
>>>>> way to your project.
>>>>>
>>>>> I still haven't made up my mind exactly what I should be testing. It
>>>>> seems unnecessary to duplicate all your tests in Clojure, since you
>>>>> are already guarding against regression in these areas - I think I
>>>>> will start out with a fairly general set of tests as I probe for
>>>>> particular problem areas for Clojure, into which I may dive for more
>>>>> detail - time will tell... if you have any ideas, please let me know.
>>>>>
>>>>> I've checked my stuff in at github, if anyone is interested - here is
>>>>> the project:
>>>>>
>>>>> https://github.com/JulesGosnell/clumatra
>>>>>
>>>>> here is some clojure that I reverse engineered from GraalKernelTester:
>>>>>
>>>>> https://github.com/JulesGosnell/clumatra/blob/master/src/clumatra/core.clj
>>>>>
>>>>>
>>>>>
>>>>> and here are the first successful tests:
>>>>>
>>>>> https://github.com/JulesGosnell/clumatra/blob/master/test/clumatra/core_test.clj
>>>>>
>>>>>
>>>>>
>>>>> not very pretty yet - I am still feeling my way.
>>>>>
>>>>> cheers
>>>>>
>>>>>
>>>>> Jules
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 06/03/14 10:24, Doug Simon wrote:
>>>>>> Hi Julian,
>>>>>>
>>>>>> This looks very interesting and will be an good alternative testing
>>>>>> vector for the Sumatra work as it matures. If Clojure tests can
>>>>>> somehow be made to run from Junit, then I’m sure we can try
>>>>>> integrating it into our testing.
>>>>>>
>>>>>> -Doug
>>>>>>
>>>>>> On Mar 6, 2014, at 12:56 AM, Julian Gosnell
>>>>>> <jules_gosnell at yahoo.com> wrote:
>>>>>>
>>>>>>> Guys,
>>>>>>>
>>>>>>> I just built the Java8 / Graal / Okra stack and managed to run some
>>>>>>> very simple Clojure copying the contents of one int array into
>>>>>>> another one on Okra.
>>>>>>>
>>>>>>> https://groups.google.com/forum/#!topic/clojure/JpjK__NTR5Y
>>>>>>>
>>>>>>>
>>>>>>> I find the ramifications of this very exciting :-)
>>>>>>>
>>>>>>> I understand that fn-ality is limited at the moment - but I am keen
>>>>>>> to keep testing and to help ensure early visibility of Clojure
>>>>>>> related issues to both communities - perhaps even the submission of
>>>>>>> a Clojure testsuite if Graal developers thought that useful.
>>>>>>>
>>>>>>> I'd be very interested in your thoughts on Graal / Clojure.
>>>>>>>
>>>>>>> regards,
>>>>>>>
>>>>>>>
>>>>>>> Jules
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>