Graal and Clojure

Sun Apr 6 14:22:19 UTC 2014

So, Guys,

Here is the promised update on Clumatra - an effort to get Clojure up 
and running on top of Graal/Okra/HSA.

- I now have a working Ubuntu/AMD HSA box.

- I have a Jenkins nightly build of JDK8 / Graal :

http://ouroboros.dyndns-free.com/ci/view/clumatra/job/graal/

- if useful, you can download the latest linux/64 tgz here (it is 300mb 
- I'm going to look at shrinking it:

http://ouroboros.dyndns-free.com/clumatra/jdk1.8.0-graal.tgz

- I have builds of Clumatra without, with simulated and with hardware 
based HSA :

http://ouroboros.dyndns-free.com/ci/view/clumatra/

- The clumatra testsuite is growing - results have been a bit unstable 
recently as I learnt which tests completely take out or hang the 
jdk8/graal jvm and which can be trusted to succeed / fail in a well 
behaved fashion:

http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/

I am working on a reflection/macro based solution, which will 
automagically create tests for all methods of all classes that I want 
tests for. When I bring this online in a couple of weeks the number of 
tests should jump dramatically.

Please feel free to inspect test results etc on my Jenkins and let me 
know whether they are helpful and whether they can be made more useful 
in any way. currently some of them seem to indicate problems in the okra 
backend. This is when graal seems to run happily but I get back the 
wrong result from my test:

e.g.

http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_unchecked_inc_test/

I run the test via graal/okra twice and once in pure clojure - so you 
see three results. The last one is the expected one, the first two the 
ones returned from the GPU.

Other tests let me know that graal is not yet implementing various 
features of jvm bytecode - e.g.

http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_my_static_inc_test/

Unfortunately the test report does not seem to be capturing the HSAIL 
logging that you can see in the console output... My guess is that it is 
captured by redirecting clojure's usual output stream, but this is not 
the same one as being used by Graal...

http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/console

That's enough for now. I'll post with more progress as and when.

Jules

On 31/03/14 23:40, Caspole, Eric wrote:
> Hi Jules,
> I updated everything and got Clumatra running on the real hardware today.
>
> In project.clj, I used this:
>
>    :jvm-opts ["-server" "-ea" "-esa" "-Xms1g" "-Xmx1g"
>               "-verbosegc" "-G:Log=CodeGen"
>               "-XX:+GPUOffload" "-XX:+TraceGPUInteraction"]
>
> Then make sure your /path/to/okra/dist/bin is in the PATH and LD_LIBRARY_PATH.
>
> When it is using the simulator, you will see this:
> [HSAIL] using _OKRA_SIM_LIB_PATH_=/tmp/okraresource.dir_2595501614013009358/libokra_x86_64.so
>
> With hardware or external okra you will see this:
>
> [HSAIL] library is libokra_x86_64.so
>
> Regards,
> Eric
>
>
> ________________________________________
> From: Jules Gosnell [jules_gosnell at yahoo.com]
> Sent: Thursday, March 20, 2014 5:27 PM
> To: Caspole, Eric; graal-dev at openjdk.java.net; sumatra-dev at openjdk.java.net
> Subject: Re: Graal and Clojure
>
> Eric,
>
> I just checked support to Clumatra for running individual unit tests
> from the CLI - should be enough to get you set up with your debugger.
>
> All details at the bottom of the README.
>
> https://github.com/JulesGosnell/clumatra
>
> I'm afraid you will need to install maven if you don't already have it -
> for some reason lein does not seem to have a test-jar target.
>
> It is still a little rough around the edges, but better than nothing :-)
>
> I also just bought my HSA h/w - should be up and running next week
> sometime. Did you figure out how to enable native-mode Okra ?
>
> catch you later,
>
>
> Jules
>
>
>
> On 17/03/14 22:20, Eric Caspole wrote:
>> Hi everybody,
>> I was setting up a new HSA system today and I installed Clumatra and got
>> it working under the simulator. I have a real HSA system not too
>> different from this:
>>
>> http://code.google.com/p/aparapi/wiki/SettingUpLinuxHSAMachineForAparapi
>>
>> In the clojure/lein/maven based system there is so much harness I can't
>> immediately see how to switch in the real hardware version of Okra
>> instead of the simulator Okra that gets installed into maven and
>> exploded in /tmp when the tests run.
>>
>> Jules, could you show how to bypass the harness etc to switch the Okra?
>> Can I just run 1 test with a simple java command line? This makes it a
>> lot easier to use a debugger which I will definitely want to do.
>>
>> Otherwise this is a cool project and we already found 1 or 2 issues we
>> can fix in HSAIL Graal related to your clojure tests.
>> Regards,
>> Eric
>>
>>
>>
>> On 03/17/2014 06:43 AM, Doug Simon wrote:
>>> Hi Julian,
>>>
>>> In terms of what to test, I’ll leave it up to the Sumatra developers
>>> at AMD and Oracle to comment on what may be interesting.
>>>
>>> As far as I know, no one in the Graal team is that familiar with
>>> Clojure. This actually makes the value of having Clojure tests
>>> integrated into our gate system somewhat problematic -
>>> debugging/fixing failures may be difficult. I’d suggest you report
>>> failures your discover on the sumatra-dev and graal-dev lists and we
>>> can resolve them that way for now.
>>>
>>> In addition, for Graal we’re always interested in benchmark suites for
>>> JVM hosted languages. Can you recommend any well known and trusted
>>> Clojure benchmarks we should consider?
>>>
>>> -Doug
>>>
>>> On Mar 17, 2014, at 12:35 AM, Jules Gosnell <jules_gosnell at yahoo.com>
>>> wrote:
>>>
>>>> Doug,
>>>>
>>>> Sorry it has taken me a while to get back - I've been busy playing
>>>> with code.
>>>>
>>>> I currently have a small clojure testsuite up and running. I plan to
>>>> broaden this and then think about providing it in a junit-compatible
>>>> way to your project.
>>>>
>>>> I still haven't made up my mind exactly what I should be testing. It
>>>> seems unnecessary to duplicate all your tests in Clojure, since you
>>>> are already guarding against regression in these areas - I think I
>>>> will start out with a fairly general set of tests as I probe for
>>>> particular problem areas for Clojure, into which I may dive for more
>>>> detail - time will tell... if you have any ideas, please let me know.
>>>>
>>>> I've checked my stuff in at github, if anyone is interested - here is
>>>> the project:
>>>>
>>>> https://github.com/JulesGosnell/clumatra
>>>>
>>>> here is some clojure that I reverse engineered from GraalKernelTester:
>>>>
>>>> https://github.com/JulesGosnell/clumatra/blob/master/src/clumatra/core.clj
>>>>
>>>>
>>>> and here are the first successful tests:
>>>>
>>>> https://github.com/JulesGosnell/clumatra/blob/master/test/clumatra/core_test.clj
>>>>
>>>>
>>>> not very pretty yet - I am still feeling my way.
>>>>
>>>> cheers
>>>>
>>>>
>>>> Jules
>>>>
>>>>
>>>>
>>>>
>>>> On 06/03/14 10:24, Doug Simon wrote:
>>>>> Hi Julian,
>>>>>
>>>>> This looks very interesting and will be an good alternative testing
>>>>> vector for the Sumatra work as it matures. If Clojure tests can
>>>>> somehow be made to run from Junit, then I’m sure we can try
>>>>> integrating it into our testing.
>>>>>
>>>>> -Doug
>>>>>
>>>>> On Mar 6, 2014, at 12:56 AM, Julian Gosnell
>>>>> <jules_gosnell at yahoo.com> wrote:
>>>>>
>>>>>> Guys,
>>>>>>
>>>>>> I just built the Java8 / Graal / Okra stack and managed to run some
>>>>>> very simple Clojure copying the contents of one int array into
>>>>>> another one on Okra.
>>>>>>
>>>>>> https://groups.google.com/forum/#!topic/clojure/JpjK__NTR5Y
>>>>>>
>>>>>>
>>>>>> I find the ramifications of this very exciting :-)
>>>>>>
>>>>>> I understand that fn-ality is limited at the moment - but I am keen
>>>>>> to keep testing and to help ensure early visibility of Clojure
>>>>>> related issues to both communities - perhaps even the submission of
>>>>>> a Clojure testsuite if Graal developers thought that useful.
>>>>>>
>>>>>> I'd be very interested in your thoughts on Graal / Clojure.
>>>>>>
>>>>>> regards,
>>>>>>
>>>>>>
>>>>>> Jules
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>
>
>