Graal and Clojure

Mon Apr 21 20:59:24 UTC 2014

Thanks for your interest, Thomas,

Progress is currently in the form of tests - currently around 500 - most 
are generated by a huge and hairy clojure macro !

These are run on a nightly build of graal/jdk8 against a pure clojure 
control, okra simulator and okra with HSA hardware.

The tests are totally clojure-centric - I didn't see any point in 
replicating the graal unit test suite.

I currently have a small number of tests which actually crash the okra 
simulator. I can provide more detail if anyone is interested,...

The tests exercise some of the java components of the clojure runtime.

In order to start testing actual Clojure code I need support for 
everything required to compile a Clojure Function. Some part[s] of this 
is/are currrently still "unimplemented". I would be very interested in 
pursuing this with an interested Graal developer.

Meanwhile, I will keep extending my testsuite and hoping to see more and 
more tests passing as graal/okra development continues.

Thanks guys,

Jules

P.S.

I would repost all the relevant links, but apparently the dyndns service 
that I use tends to cause my posts to be filtered...You can find them 
back up this thread.

On 16/04/14 12:13, Thomas Wuerthinger wrote:
> Cool! It is very exciting to see so much progress with Clojure running on the GPU via Graal. - thomas
>
> On 07 Apr 2014, at 08:20, Jules Gosnell <jules_gosnell at yahoo.com> wrote:
>
>> Good news - I reapplied the -XX:-UseHSAILDeoptimization flag (last time I tried it the code had not yet made it into my jdk8/graal build) and a lot of the instability mentioned in my last mail disappeared - particularly in the simulator. Many more tests also passed :-)
>>
>> http://ouroboros.dyndns-free.com/ci/job/clumatra-hardware-acceleration/
>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-simulated-acceleration/
>>
>>
>> Jules
>>
>> On 06/04/14 15:22, Jules Gosnell wrote:
>>> So, Guys,
>>>
>>> Here is the promised update on Clumatra - an effort to get Clojure up
>>> and running on top of Graal/Okra/HSA.
>>>
>>> - I now have a working Ubuntu/AMD HSA box.
>>>
>>> - I have a Jenkins nightly build of JDK8 / Graal :
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/graal/
>>>
>>> - if useful, you can download the latest linux/64 tgz here (it is 300mb
>>> - I'm going to look at shrinking it:
>>>
>>> http://ouroboros.dyndns-free.com/clumatra/jdk1.8.0-graal.tgz
>>>
>>> - I have builds of Clumatra without, with simulated and with hardware
>>> based HSA :
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/
>>>
>>> - The clumatra testsuite is growing - results have been a bit unstable
>>> recently as I learnt which tests completely take out or hang the
>>> jdk8/graal jvm and which can be trusted to succeed / fail in a well
>>> behaved fashion:
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/
>>>
>>>
>>> I am working on a reflection/macro based solution, which will
>>> automagically create tests for all methods of all classes that I want
>>> tests for. When I bring this online in a couple of weeks the number of
>>> tests should jump dramatically.
>>>
>>> Please feel free to inspect test results etc on my Jenkins and let me
>>> know whether they are helpful and whether they can be made more useful
>>> in any way. currently some of them seem to indicate problems in the okra
>>> backend. This is when graal seems to run happily but I get back the
>>> wrong result from my test:
>>>
>>> e.g.
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_unchecked_inc_test/
>>>
>>>
>>> I run the test via graal/okra twice and once in pure clojure - so you
>>> see three results. The last one is the expected one, the first two the
>>> ones returned from the GPU.
>>>
>>> Other tests let me know that graal is not yet implementing various
>>> features of jvm bytecode - e.g.
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/testReport/clumatra/core-test/long_my_static_inc_test/
>>>
>>>
>>> Unfortunately the test report does not seem to be capturing the HSAIL
>>> logging that you can see in the console output... My guess is that it is
>>> captured by redirecting clojure's usual output stream, but this is not
>>> the same one as being used by Graal...
>>>
>>> http://ouroboros.dyndns-free.com/ci/view/clumatra/job/clumatra-hardware-acceleration/83/console
>>>
>>>
>>>
>>> That's enough for now. I'll post with more progress as and when.
>>>
>>>
>>>
>>> Jules
>>>
>>>
>>>
>>> On 31/03/14 23:40, Caspole, Eric wrote:
>>>> Hi Jules,
>>>> I updated everything and got Clumatra running on the real hardware today.
>>>>
>>>> In project.clj, I used this:
>>>>
>>>>    :jvm-opts ["-server" "-ea" "-esa" "-Xms1g" "-Xmx1g"
>>>>               "-verbosegc" "-G:Log=CodeGen"
>>>>               "-XX:+GPUOffload" "-XX:+TraceGPUInteraction"]
>>>>
>>>> Then make sure your /path/to/okra/dist/bin is in the PATH and
>>>> LD_LIBRARY_PATH.
>>>>
>>>> When it is using the simulator, you will see this:
>>>> [HSAIL] using
>>>> _OKRA_SIM_LIB_PATH_=/tmp/okraresource.dir_2595501614013009358/libokra_x86_64.so
>>>>
>>>>
>>>> With hardware or external okra you will see this:
>>>>
>>>> [HSAIL] library is libokra_x86_64.so
>>>>
>>>> Regards,
>>>> Eric
>>>>
>>>>
>>>> ________________________________________
>>>> From: Jules Gosnell [jules_gosnell at yahoo.com]
>>>> Sent: Thursday, March 20, 2014 5:27 PM
>>>> To: Caspole, Eric; graal-dev at openjdk.java.net;
>>>> sumatra-dev at openjdk.java.net
>>>> Subject: Re: Graal and Clojure
>>>>
>>>> Eric,
>>>>
>>>> I just checked support to Clumatra for running individual unit tests
>>>> from the CLI - should be enough to get you set up with your debugger.
>>>>
>>>> All details at the bottom of the README.
>>>>
>>>> https://github.com/JulesGosnell/clumatra
>>>>
>>>> I'm afraid you will need to install maven if you don't already have it -
>>>> for some reason lein does not seem to have a test-jar target.
>>>>
>>>> It is still a little rough around the edges, but better than nothing :-)
>>>>
>>>> I also just bought my HSA h/w - should be up and running next week
>>>> sometime. Did you figure out how to enable native-mode Okra ?
>>>>
>>>> catch you later,
>>>>
>>>>
>>>> Jules
>>>>
>>>>
>>>>
>>>> On 17/03/14 22:20, Eric Caspole wrote:
>>>>> Hi everybody,
>>>>> I was setting up a new HSA system today and I installed Clumatra and got
>>>>> it working under the simulator. I have a real HSA system not too
>>>>> different from this:
>>>>>
>>>>> http://code.google.com/p/aparapi/wiki/SettingUpLinuxHSAMachineForAparapi
>>>>>
>>>>> In the clojure/lein/maven based system there is so much harness I can't
>>>>> immediately see how to switch in the real hardware version of Okra
>>>>> instead of the simulator Okra that gets installed into maven and
>>>>> exploded in /tmp when the tests run.
>>>>>
>>>>> Jules, could you show how to bypass the harness etc to switch the Okra?
>>>>> Can I just run 1 test with a simple java command line? This makes it a
>>>>> lot easier to use a debugger which I will definitely want to do.
>>>>>
>>>>> Otherwise this is a cool project and we already found 1 or 2 issues we
>>>>> can fix in HSAIL Graal related to your clojure tests.
>>>>> Regards,
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On 03/17/2014 06:43 AM, Doug Simon wrote:
>>>>>> Hi Julian,
>>>>>>
>>>>>> In terms of what to test, I’ll leave it up to the Sumatra developers
>>>>>> at AMD and Oracle to comment on what may be interesting.
>>>>>>
>>>>>> As far as I know, no one in the Graal team is that familiar with
>>>>>> Clojure. This actually makes the value of having Clojure tests
>>>>>> integrated into our gate system somewhat problematic -
>>>>>> debugging/fixing failures may be difficult. I’d suggest you report
>>>>>> failures your discover on the sumatra-dev and graal-dev lists and we
>>>>>> can resolve them that way for now.
>>>>>>
>>>>>> In addition, for Graal we’re always interested in benchmark suites for
>>>>>> JVM hosted languages. Can you recommend any well known and trusted
>>>>>> Clojure benchmarks we should consider?
>>>>>>
>>>>>> -Doug
>>>>>>
>>>>>> On Mar 17, 2014, at 12:35 AM, Jules Gosnell <jules_gosnell at yahoo.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Doug,
>>>>>>>
>>>>>>> Sorry it has taken me a while to get back - I've been busy playing
>>>>>>> with code.
>>>>>>>
>>>>>>> I currently have a small clojure testsuite up and running. I plan to
>>>>>>> broaden this and then think about providing it in a junit-compatible
>>>>>>> way to your project.
>>>>>>>
>>>>>>> I still haven't made up my mind exactly what I should be testing. It
>>>>>>> seems unnecessary to duplicate all your tests in Clojure, since you
>>>>>>> are already guarding against regression in these areas - I think I
>>>>>>> will start out with a fairly general set of tests as I probe for
>>>>>>> particular problem areas for Clojure, into which I may dive for more
>>>>>>> detail - time will tell... if you have any ideas, please let me know.
>>>>>>>
>>>>>>> I've checked my stuff in at github, if anyone is interested - here is
>>>>>>> the project:
>>>>>>>
>>>>>>> https://github.com/JulesGosnell/clumatra
>>>>>>>
>>>>>>> here is some clojure that I reverse engineered from GraalKernelTester:
>>>>>>>
>>>>>>> https://github.com/JulesGosnell/clumatra/blob/master/src/clumatra/core.clj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> and here are the first successful tests:
>>>>>>>
>>>>>>> https://github.com/JulesGosnell/clumatra/blob/master/test/clumatra/core_test.clj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> not very pretty yet - I am still feeling my way.
>>>>>>>
>>>>>>> cheers
>>>>>>>
>>>>>>>
>>>>>>> Jules
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 06/03/14 10:24, Doug Simon wrote:
>>>>>>>> Hi Julian,
>>>>>>>>
>>>>>>>> This looks very interesting and will be an good alternative testing
>>>>>>>> vector for the Sumatra work as it matures. If Clojure tests can
>>>>>>>> somehow be made to run from Junit, then I’m sure we can try
>>>>>>>> integrating it into our testing.
>>>>>>>>
>>>>>>>> -Doug
>>>>>>>>
>>>>>>>> On Mar 6, 2014, at 12:56 AM, Julian Gosnell
>>>>>>>> <jules_gosnell at yahoo.com> wrote:
>>>>>>>>
>>>>>>>>> Guys,
>>>>>>>>>
>>>>>>>>> I just built the Java8 / Graal / Okra stack and managed to run some
>>>>>>>>> very simple Clojure copying the contents of one int array into
>>>>>>>>> another one on Okra.
>>>>>>>>>
>>>>>>>>> https://groups.google.com/forum/#!topic/clojure/JpjK__NTR5Y
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I find the ramifications of this very exciting :-)
>>>>>>>>>
>>>>>>>>> I understand that fn-ality is limited at the moment - but I am keen
>>>>>>>>> to keep testing and to help ensure early visibility of Clojure
>>>>>>>>> related issues to both communities - perhaps even the submission of
>>>>>>>>> a Clojure testsuite if Graal developers thought that useful.
>>>>>>>>>
>>>>>>>>> I'd be very interested in your thoughts on Graal / Clojure.
>>>>>>>>>
>>>>>>>>> regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jules
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>