hsail gate commands -- webrev
Doug Simon
doug.simon at oracle.com
Wed Jan 22 07:57:09 PST 2014
It would also be good if the generated temp_hsa.hsail and temp_hsa.o files were deleted by default.
On Jan 22, 2014, at 12:57 PM, Doug Simon <doug.simon at oracle.com> wrote:
> This all seems to work - nice job. Just a few requests/comments before I integrate.
>
> HSAILCompilationResult.java:
>
> 60 // this test of okraLib existence is to allow the loading of the
> 61 // native library from the jar file if it is not already on the path
> 62 private static boolean okraLibExists = OkraUtil.okraLibExists();
>
> The above assignment produces a “value of field is not used” warning in Eclipse. It seems that all you want here is the side-effect of the call to OkraUtil.okraLibExists(). If that’s the case, just put the call in a static initializer block and delete the field:
>
> static {
> OkraUtil.okraLibExists();
> }
>
> However, I tried just commenting out the line altogether and the unit tests seemed to still run fine. Which begs the question, do you really need this here?
>
> Can you also please update (and re-upload) okra-with-sim.jar to include the Java sources (just like okra-1.2.jar had). This helps with debugging the code in Eclipse.
>
> -Doug
>
> On Jan 21, 2014, at 6:07 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
>
>> All --
>>
>> I have uploaded a webrev to allow use of the hsail simulator without setting up any PATH or LD_LIBRARY_PATH variables to make it easier to run the hsail junit tests as part of the graal gate:
>>
>> http://cr.openjdk.java.net/~tdeneau/graal-webrevs/webrev-lazy-native-hsail-library-open/webrev/
>>
>> * The main change was in gpu_hsail.cpp to lazily load the native library,
>> (waiting until it is really needed). It always tries the original LD_LIBRARY_PATH
>> as a first try, and only if that doesn't work, it will try
>> the environment variable _OKRA_SIM_LIB_PATH_ (which will have been set up by the
>> java side which will have copied the native library and hsailasm to a temporary
>> directory if necessary)
>>
>> * HSAILCompilationResult, added a call the the JNI side to get it to load the native library
>> from the jar file if it needs to.
>>
>> * mx/projects changed to use the newly built okra-with-sim.jar which has been uploaded to
>> my cr.openjdk.java.net directory (the old okra.jar came from there as well).
>> This jar file contains the prebuilt binaries but can also be built from sources
>> following the instructions at
>> https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-Simulator
>>
>> To test:
>> * make sure not only that the okra/dist/bin directory
>> is removed from PATH and LD_LIBRARY_PATH but also that it is not one of the directories
>> listed under "ldconfig -p"
>>
>>
>> The changes to the okra.jar file and the native library are not part of this webrev, but
>> if interested, these changes can be viewed as a pull request on the github directory
>> https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-Simulator/pull/11
>>
>> * Again, the native library loading as performed by the OkraContext class
>> first tries the original PATH and LD_LIBRARY_PATH strategy before reverting
>> to using a temporary directory. It looks in the jar file that OkraContext.class
>> was loaded from and copies anything in resources/okra to a temporary directory.
>> Then it tries to load the native library from there.
>>
>> * the native library itself now contains an on_load function
>> with attribute((constructor)) which gets executed as the .so file gets loaded.
>> It does the following:
>> * finds out where it was loaded from (using dladdr)
>> * appends that directory to PATH and LD_LIBRARY_PATH (this allows
>> a later invoke of hsailasm to work)
>> * sets an environment variable _OKRA_SIM_LIB_PATH_. This env var
>> is used later by the JVM when it wants to load the native library
>> and discover its C-based entry points.
>>
>> -- Tom
>>
>>
>>> -----Original Message-----
>>> From: Tom Rodriguez [mailto:tom.rodriguez at oracle.com]
>>> Sent: Wednesday, January 15, 2014 4:15 PM
>>> To: Douglas Simon
>>> Cc: Deneau, Tom; graal-dev at openjdk.java.net
>>> Subject: Re: hsail gate commands
>>>
>>> dladdr can also be used to figure it out directly.
>>>
>>> tom
>>>
>>> On Jan 15, 2014, at 2:07 PM, Doug Simon <doug.simon at oracle.com> wrote:
>>>
>>>>
>>>> On Jan 15, 2014, at 10:46 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
>>>>
>>>>> Doug --
>>>>>
>>>>> how does the JNI_OnLoad know what path it was loaded from?
>>>>> or would the java side have to pass this information down?
>>>>
>>>> The Java side has to pass this down. One way to do it would be to
>>> define a static String field in some class in the Okra Java library and
>>> assign it the path before calling System.load(). Inside JNI_OnLoad, the
>>> JNI interface can be used to find the class and read the field.
>>>>
>>>> -Doug
>>>>
>>>>>> -----Original Message-----
>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>> Sent: Wednesday, January 15, 2014 3:11 PM
>>>>>> To: Deneau, Tom
>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>> Subject: Re: hsail gate commands
>>>>>>
>>>>>> Hi Tom,
>>>>>>
>>>>>> A quick glance through the HSAIL C++ layer seems to indicate that it
>>>>>> should be able to lazily initialize the HSAIL library linkage
>>> (currently
>>>>>> done in gnu::Hsail::probe_linkage()). This means it should be
>>> deferrable
>>>>>> until after the Java part of the Okra library has had a chance to
>>>>>> extract the native library file (and hsailasm executable) into the
>>> file
>>>>>> system. Then you would call System.load() to load the library.
>>> There's
>>>>>> still the problem of how to communicate the absolute path of the
>>> native
>>>>>> library to the os::dll_load call. There may be a more elegant
>>> solution,
>>>>>> but I'd suggest using the JNI_OnLoad[1] mechanism. Inside the
>>> JNI_OnLoad
>>>>>> function in the Okra native library, you can set an environment
>>> variable
>>>>>> (e.g., OKRA_DLL_PATH) which gnu::Hsail::probe_linkage can then use.
>>> Note
>>>>>> that the environment variable has to be set this way as there's no
>>> way
>>>>>> to set an environment variable from Java.
>>>>>>
>>>>>> By the time you finish all this hackery, you may start thinking that
>>>>>> simply porting the HSAIL simulator/toolchain to Java might be easier
>>> ;-)
>>>>>>
>>>>>> [1]
>>>>>>
>>> http://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/invocation.html#
>>>>>> JNI_OnLoad
>>>>>>
>>>>>> On Jan 15, 2014, at 9:05 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
>>>>>>
>>>>>>> Doug --
>>>>>>>
>>>>>>> One complication I saw here.
>>>>>>> The native library file is used both from the java side (using
>>>>>>> loadLibrary) and from the hotspot side (using os::dll_load followed
>>> by
>>>>>> looking up the specific functions it needs).
>>>>>>> And the hotspot attempt to load happens first (from gpu::init()).
>>>>>>>
>>>>>>> So anything the java side would have done with resources in the
>>> .jar
>>>>>>> file and temp directories wouldn't have been done yet.
>>>>>>>
>>>>>>> -- Tom
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>>>> Sent: Tuesday, January 14, 2014 3:27 AM
>>>>>>>> To: Deneau, Tom
>>>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>>>> Subject: Re: hsail gate commands
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 13, 2014, at 8:52 PM, Deneau, Tom <tom.deneau at amd.com>
>>> wrote:
>>>>>>>>
>>>>>>>>> Doug --
>>>>>>>>>
>>>>>>>>> Using the techniques in
>>>>>>>>> http://frommyplayground.com/how-to-load-native-jni-library-from-
>>> jar/
>>>>>>>>> that you sent, would it be acceptable to leave hsailasm as a
>>>>>>>>> separate
>>>>>>>> executable which gets unjarred into the system temporary directory
>>>>>>>> and then executed from there using an absolute pathname?
>>>>>>>>
>>>>>>>> Sounds feasible and reasonable to me.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- Tom
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>>>>>> Sent: Sunday, January 12, 2014 2:29 PM
>>>>>>>>>> To: Deneau, Tom
>>>>>>>>>> Subject: Re: hsail gate commands
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Jan 9, 2014, at 9:50 PM, Deneau, Tom <tom.deneau at amd.com>
>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Doug --
>>>>>>>>>>>
>>>>>>>>>>> We'll take a look at your suggestion.
>>>>>>>>>>
>>>>>>>>>> Great.
>>>>>>>>>>
>>>>>>>>>> BTW, you guys may want to bring
>>>>>>>>>>
>>> https://wiki.openjdk.java.net/display/Sumatra/The+HSAIL+Simulator
>>>>>>>>>> more up to date at some point. At least you can remove the
>>> "(webrev
>>>>>>>>>> under review)" qualifications.
>>>>>>>>>>
>>>>>>>>>> -Doug
>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>>>>>>>> Sent: Thursday, January 09, 2014 1:38 PM
>>>>>>>>>>>> To: Deneau, Tom
>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>>>>>>>> Subject: Re: hsail gate commands
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Jan 9, 2014, at 6:51 PM, Tom Rodriguez
>>>>>>>>>>>> <tom.rodriguez at oracle.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It would still be great if there would be some way to
>>>>>>>>>>>>>>>> incorporate
>>>>>>>>>>>> the hsail simulator into the gate...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A good start would be if you could provide us with a patch
>>> for
>>>>>>>>>>>>>>> `mx',
>>>>>>>>>>>> that adds a command which sets up the HSAIL simulator
>>> properly.
>>>>>>>>>>>> Currently our gate is running on Linux, but this might change
>>> in
>>>>>>>>>>>> the future, so keep in mind that this should potentially work
>>> on
>>>>>>>>>>>> all platforms (Linux, MacOSX, Windows).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Our gate server is basically executing `mx gate' to
>>> validate
>>>>>>>>>>>> changes. That includes building, bootstrap tests, unittest,
>>> etc.
>>>>>>>>>>>> If that command was successful on our gate server, the changes
>>>>>>>>>>>> are accepted and pushed to the repository. Having the `mx
>>> gate'
>>>>>>>>>>>> command, one can reproduce the gate process on his/her local
>>>>>>>> machine.
>>>>>>>>>>>> Therefore it would be important that one can easily and
>>> reliable
>>>>>>>>>>>> set up the HSAIL environment if it is part of the gate
>>> process.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Given the dependencies for building the simulator[1] and the
>>>>>>>>>>>>>> fact it
>>>>>>>>>>>> only currently runs on linux, I don't think it will be that
>>> easy
>>>>>>>>>>>> to come up with an mx command.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What precompiling an okra distribution for linux that mx can
>>>>>>>>>> download?
>>>>>>>>>>>>
>>>>>>>>>>>> If Okra used a self contained JNI library, that might just
>>> work.
>>>>>>>>>>>> However, the way it is deployed now involves having the
>>> hsailasm
>>>>>>>>>>>> executable on your PATH. Also, LD_LIBRARY_PATH has to be setup
>>> to
>>>>>>>>>>>> find the JNI library itself.
>>>>>>>>>>>>
>>>>>>>>>>>> @AMD guys: How hard would it be to make the Okra JNI be self
>>>>>>>>>>>> contained library and include the hsailasm functionality? If
>>>>>>>>>>>> that's possible, you could bundle the JNI library inside the
>>> Okra
>>>>>>>>>>>> jar using a technique like
>>>>>>>>>>>> http://frommyplayground.com/how-to-load-native-jni-library-
>>> from-j
>>>>>>>>>>>> ar
>>>>>>>>>>>> /
>>>>>>>>>> to avoid having to set up PATH or LD_LIBRARY_PATH.
>>>>>>>>>>>>
>>>>>>>>>>>> -Doug
>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]
>>>>>>>>>>>>>> https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-
>>> Simula
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> r#
>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>> ra-interface-to-hsail-simulator
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>> From: graal-dev-bounces at openjdk.java.net [mailto:graal-
>>> dev-
>>>>>>>>>>>>>>>>> bounces at openjdk.java.net] On Behalf Of Deneau, Tom
>>>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 11:24 AM
>>>>>>>>>>>>>>>>> To: Doug Simon
>>>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>>>>>>>>>>>>> Subject: RE: hsail gate commands
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> OK, I see why there is no error when running mx --vm
>>> server
>>>>>>>>>>>>>>>>> unittest hsail
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At some point (I don't recall why, maybe at Oracle's
>>>>>>>>>>>>>>>>> request??), we put in some code in KernelTester that if
>>> it
>>>>>>>>>>>>>>>>> could not find the okra simulator files would just
>>> silently
>>>>>>>>>>>>>>>>> not run the tests (which I guess counts as a pass).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> No the hsail simulator is not a java app at all.
>>>>>>>>>>>>>>>>> The page at
>>>>>>>>>>>>>>>>>
>>> https://wiki.openjdk.java.net/display/Sumatra/The+HSAIL+Simu
>>>>>>>>>>>>>>>>> la to r should describe how to build it and what
>>> environment
>>>>>>>>>>>>>>>>> variables to set up to use it...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -- Tom
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 11:02 AM
>>>>>>>>>>>>>>>>>> To: Deneau, Tom
>>>>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>> Subject: Re: hsail gate commands
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Dec 16, 2013, at 5:48 PM, Deneau, Tom
>>>>>>>>>>>>>>>>>> <tom.deneau at amd.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Doug --
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I see.
>>>>>>>>>>>>>>>>>>> I don't understand why the tests would run without
>>> error
>>>>>>>>>>>>>>>>>>> if the simulator and associated assembler is missing
>>>>>> but...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What error are you seeing?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Is there a way we can get the HSAIL simulator into the
>>>>>>>>>>>>>>>>>>> gate
>>>>>>>>>>>>>>>>>> infrastructure?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is it a pure Java app yet? That would certainly make it
>>>>>>>>>> trivial.
>>>>>>>>>>>>>>>>>> In any case, I'll defer to Bernhard or Gilles to answer
>>>>>>>>>>>>>>>>>> this since they are the maintainers of this
>>> infrastructure.
>>>>>>>>>>>>>>>>>> Can you please send instructions on how to
>>>>>>>>>>>>>>>>>> install/use/configure the
>>>>>>>>>>>> simulator.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Doug
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>>>>>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 10:36 AM
>>>>>>>>>>>>>>>>>>>> To: Deneau, Tom
>>>>>>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>>>> Subject: Re: hsail gate commands
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Dec 16, 2013, at 5:21 PM, Deneau, Tom
>>>>>>>>>>>>>>>>>>>> <tom.deneau at amd.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Doug --
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I noticed in syncing with the trunk as of last Friday
>>>>>>>>>>>>>>>>>>>>> that all of our
>>>>>>>>>>>>>>>>>>>> HSAIL test cases broke. The cause was some imperfect
>>>>>>>>>>>>>>>>>>>> code in our HSAILAssembler for compare instructions
>>> which
>>>>>>>>>>>>>>>>>>>> when presented with an unordered compare could
>>> generate
>>>>>>>>>>>>>>>>>>>> code that would not assemble, and was easy to fix.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But I was wondering how this made it past the gate.
>>>>>>>>>>>>>>>>>>>>> Can you describe what gate commands are used
>>> regarding
>>>>>>>>>> hsail?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We simply run 'mx gate' which will include running all
>>>>>>>>>>>>>>>>>>>> the HSAIL unit tests.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On my Mac with the latest bits:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> $ mx --vm server unittest hsail executing junit tests
>>>>>>>> now...
>>>>>>>>>>>>>>>>>>>> (107 test classes) JUnit version 4.8
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>> ........................................................................
>>>>>>>>>>>>>>>>>>>> ...........I................I......
>>>>>>>>>>>>>>>>>>>> Time: 5.525
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> OK (105 tests)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Keep in mind I (and the gate infrastructure) don't
>>> have
>>>>>>>>>>>>>>>>>>>> the HSAIL simulator.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Doug
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>
More information about the graal-dev
mailing list