hsail gate commands -- webrev
Deneau, Tom
tom.deneau at amd.com
Wed Jan 22 09:03:49 PST 2014
Doug --
This okraLibExists force load of the library is only really useful on the non-junit path, for example, when Sumatra is the client.
I can make the changes you recommend.
-- Tom
> -----Original Message-----
> From: Doug Simon [mailto:doug.simon at oracle.com]
> Sent: Wednesday, January 22, 2014 5:58 AM
> To: Deneau, Tom
> Cc: Tom Rodriguez; graal-dev at openjdk.java.net
> Subject: Re: hsail gate commands -- webrev
>
> This all seems to work - nice job. Just a few requests/comments before I
> integrate.
>
> HSAILCompilationResult.java:
>
> 60 // this test of okraLib existence is to allow the loading of
> the
> 61 // native library from the jar file if it is not already on the
> path
> 62 private static boolean okraLibExists =
> OkraUtil.okraLibExists();
>
> The above assignment produces a "value of field is not used" warning in
> Eclipse. It seems that all you want here is the side-effect of the call
> to OkraUtil.okraLibExists(). If that's the case, just put the call in a
> static initializer block and delete the field:
>
> static {
> OkraUtil.okraLibExists();
> }
>
> However, I tried just commenting out the line altogether and the unit
> tests seemed to still run fine. Which begs the question, do you really
> need this here?
>
> Can you also please update (and re-upload) okra-with-sim.jar to include
> the Java sources (just like okra-1.2.jar had). This helps with debugging
> the code in Eclipse.
>
> -Doug
>
> On Jan 21, 2014, at 6:07 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
>
> > All --
> >
> > I have uploaded a webrev to allow use of the hsail simulator without
> setting up any PATH or LD_LIBRARY_PATH variables to make it easier to
> run the hsail junit tests as part of the graal gate:
> >
> > http://cr.openjdk.java.net/~tdeneau/graal-webrevs/webrev-lazy-native-h
> > sail-library-open/webrev/
> >
> > * The main change was in gpu_hsail.cpp to lazily load the native
> library,
> > (waiting until it is really needed). It always tries the original
> LD_LIBRARY_PATH
> > as a first try, and only if that doesn't work, it will try
> > the environment variable _OKRA_SIM_LIB_PATH_ (which will have been
> set up by the
> > java side which will have copied the native library and hsailasm
> to a temporary
> > directory if necessary)
> >
> > * HSAILCompilationResult, added a call the the JNI side to get it to
> load the native library
> > from the jar file if it needs to.
> >
> > * mx/projects changed to use the newly built okra-with-sim.jar which
> has been uploaded to
> > my cr.openjdk.java.net directory (the old okra.jar came from there
> as well).
> > This jar file contains the prebuilt binaries but can also be built
> from sources
> > following the instructions at
> > https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-Simulator
> >
> > To test:
> > * make sure not only that the okra/dist/bin directory
> > is removed from PATH and LD_LIBRARY_PATH but also that it is not
> one of the directories
> > listed under "ldconfig -p"
> >
> >
> > The changes to the okra.jar file and the native library are not part
> > of this webrev, but if interested, these changes can be viewed as a
> > pull request on the github directory
> > https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-Simulator/pul
> > l/11
> >
> > * Again, the native library loading as performed by the OkraContext
> class
> > first tries the original PATH and LD_LIBRARY_PATH strategy before
> reverting
> > to using a temporary directory. It looks in the jar file that
> OkraContext.class
> > was loaded from and copies anything in resources/okra to a
> temporary directory.
> > Then it tries to load the native library from there.
> >
> > * the native library itself now contains an on_load function
> > with attribute((constructor)) which gets executed as the .so file
> gets loaded.
> > It does the following:
> > * finds out where it was loaded from (using dladdr)
> > * appends that directory to PATH and LD_LIBRARY_PATH (this
> allows
> > a later invoke of hsailasm to work)
> > * sets an environment variable _OKRA_SIM_LIB_PATH_. This env
> var
> > is used later by the JVM when it wants to load the native
> library
> > and discover its C-based entry points.
> >
> > -- Tom
> >
> >
> >> -----Original Message-----
> >> From: Tom Rodriguez [mailto:tom.rodriguez at oracle.com]
> >> Sent: Wednesday, January 15, 2014 4:15 PM
> >> To: Douglas Simon
> >> Cc: Deneau, Tom; graal-dev at openjdk.java.net
> >> Subject: Re: hsail gate commands
> >>
> >> dladdr can also be used to figure it out directly.
> >>
> >> tom
> >>
> >> On Jan 15, 2014, at 2:07 PM, Doug Simon <doug.simon at oracle.com>
> wrote:
> >>
> >>>
> >>> On Jan 15, 2014, at 10:46 PM, Deneau, Tom <tom.deneau at amd.com>
> wrote:
> >>>
> >>>> Doug --
> >>>>
> >>>> how does the JNI_OnLoad know what path it was loaded from?
> >>>> or would the java side have to pass this information down?
> >>>
> >>> The Java side has to pass this down. One way to do it would be to
> >> define a static String field in some class in the Okra Java library
> >> and assign it the path before calling System.load(). Inside
> >> JNI_OnLoad, the JNI interface can be used to find the class and read
> the field.
> >>>
> >>> -Doug
> >>>
> >>>>> -----Original Message-----
> >>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>> Sent: Wednesday, January 15, 2014 3:11 PM
> >>>>> To: Deneau, Tom
> >>>>> Cc: graal-dev at openjdk.java.net
> >>>>> Subject: Re: hsail gate commands
> >>>>>
> >>>>> Hi Tom,
> >>>>>
> >>>>> A quick glance through the HSAIL C++ layer seems to indicate that
> >>>>> it should be able to lazily initialize the HSAIL library linkage
> >> (currently
> >>>>> done in gnu::Hsail::probe_linkage()). This means it should be
> >> deferrable
> >>>>> until after the Java part of the Okra library has had a chance to
> >>>>> extract the native library file (and hsailasm executable) into the
> >> file
> >>>>> system. Then you would call System.load() to load the library.
> >> There's
> >>>>> still the problem of how to communicate the absolute path of the
> >> native
> >>>>> library to the os::dll_load call. There may be a more elegant
> >> solution,
> >>>>> but I'd suggest using the JNI_OnLoad[1] mechanism. Inside the
> >> JNI_OnLoad
> >>>>> function in the Okra native library, you can set an environment
> >> variable
> >>>>> (e.g., OKRA_DLL_PATH) which gnu::Hsail::probe_linkage can then
> use.
> >> Note
> >>>>> that the environment variable has to be set this way as there's no
> >> way
> >>>>> to set an environment variable from Java.
> >>>>>
> >>>>> By the time you finish all this hackery, you may start thinking
> >>>>> that simply porting the HSAIL simulator/toolchain to Java might be
> >>>>> easier
> >> ;-)
> >>>>>
> >>>>> [1]
> >>>>>
> >> http://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/invocation.ht
> >> ml#
> >>>>> JNI_OnLoad
> >>>>>
> >>>>> On Jan 15, 2014, at 9:05 PM, Deneau, Tom <tom.deneau at amd.com>
> wrote:
> >>>>>
> >>>>>> Doug --
> >>>>>>
> >>>>>> One complication I saw here.
> >>>>>> The native library file is used both from the java side (using
> >>>>>> loadLibrary) and from the hotspot side (using os::dll_load
> >>>>>> followed
> >> by
> >>>>> looking up the specific functions it needs).
> >>>>>> And the hotspot attempt to load happens first (from gpu::init()).
> >>>>>>
> >>>>>> So anything the java side would have done with resources in the
> >> .jar
> >>>>>> file and temp directories wouldn't have been done yet.
> >>>>>>
> >>>>>> -- Tom
> >>>>>>
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>>>> Sent: Tuesday, January 14, 2014 3:27 AM
> >>>>>>> To: Deneau, Tom
> >>>>>>> Cc: graal-dev at openjdk.java.net
> >>>>>>> Subject: Re: hsail gate commands
> >>>>>>>
> >>>>>>>
> >>>>>>> On Jan 13, 2014, at 8:52 PM, Deneau, Tom <tom.deneau at amd.com>
> >> wrote:
> >>>>>>>
> >>>>>>>> Doug --
> >>>>>>>>
> >>>>>>>> Using the techniques in
> >>>>>>>> http://frommyplayground.com/how-to-load-native-jni-library-from
> >>>>>>>> -
> >> jar/
> >>>>>>>> that you sent, would it be acceptable to leave hsailasm as a
> >>>>>>>> separate
> >>>>>>> executable which gets unjarred into the system temporary
> >>>>>>> directory and then executed from there using an absolute
> pathname?
> >>>>>>>
> >>>>>>> Sounds feasible and reasonable to me.
> >>>>>>>
> >>>>>>>>
> >>>>>>>> -- Tom
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>>>>>> Sent: Sunday, January 12, 2014 2:29 PM
> >>>>>>>>> To: Deneau, Tom
> >>>>>>>>> Subject: Re: hsail gate commands
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Jan 9, 2014, at 9:50 PM, Deneau, Tom <tom.deneau at amd.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> Doug --
> >>>>>>>>>>
> >>>>>>>>>> We'll take a look at your suggestion.
> >>>>>>>>>
> >>>>>>>>> Great.
> >>>>>>>>>
> >>>>>>>>> BTW, you guys may want to bring
> >>>>>>>>>
> >> https://wiki.openjdk.java.net/display/Sumatra/The+HSAIL+Simulator
> >>>>>>>>> more up to date at some point. At least you can remove the
> >> "(webrev
> >>>>>>>>> under review)" qualifications.
> >>>>>>>>>
> >>>>>>>>> -Doug
> >>>>>>>>>
> >>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>>>>>>>> Sent: Thursday, January 09, 2014 1:38 PM
> >>>>>>>>>>> To: Deneau, Tom
> >>>>>>>>>>> Cc: graal-dev at openjdk.java.net
> >>>>>>>>>>> Subject: Re: hsail gate commands
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Jan 9, 2014, at 6:51 PM, Tom Rodriguez
> >>>>>>>>>>> <tom.rodriguez at oracle.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It would still be great if there would be some way to
> >>>>>>>>>>>>>>> incorporate
> >>>>>>>>>>> the hsail simulator into the gate...
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> A good start would be if you could provide us with a
> >>>>>>>>>>>>>> patch
> >> for
> >>>>>>>>>>>>>> `mx',
> >>>>>>>>>>> that adds a command which sets up the HSAIL simulator
> >> properly.
> >>>>>>>>>>> Currently our gate is running on Linux, but this might
> >>>>>>>>>>> change
> >> in
> >>>>>>>>>>> the future, so keep in mind that this should potentially
> >>>>>>>>>>> work
> >> on
> >>>>>>>>>>> all platforms (Linux, MacOSX, Windows).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Our gate server is basically executing `mx gate' to
> >> validate
> >>>>>>>>>>> changes. That includes building, bootstrap tests, unittest,
> >> etc.
> >>>>>>>>>>> If that command was successful on our gate server, the
> >>>>>>>>>>> changes are accepted and pushed to the repository. Having
> >>>>>>>>>>> the `mx
> >> gate'
> >>>>>>>>>>> command, one can reproduce the gate process on his/her local
> >>>>>>> machine.
> >>>>>>>>>>> Therefore it would be important that one can easily and
> >> reliable
> >>>>>>>>>>> set up the HSAIL environment if it is part of the gate
> >> process.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Given the dependencies for building the simulator[1] and
> >>>>>>>>>>>>> the fact it
> >>>>>>>>>>> only currently runs on linux, I don't think it will be that
> >> easy
> >>>>>>>>>>> to come up with an mx command.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What precompiling an okra distribution for linux that mx
> >>>>>>>>>>>> can
> >>>>>>>>> download?
> >>>>>>>>>>>
> >>>>>>>>>>> If Okra used a self contained JNI library, that might just
> >> work.
> >>>>>>>>>>> However, the way it is deployed now involves having the
> >> hsailasm
> >>>>>>>>>>> executable on your PATH. Also, LD_LIBRARY_PATH has to be
> >>>>>>>>>>> setup
> >> to
> >>>>>>>>>>> find the JNI library itself.
> >>>>>>>>>>>
> >>>>>>>>>>> @AMD guys: How hard would it be to make the Okra JNI be self
> >>>>>>>>>>> contained library and include the hsailasm functionality? If
> >>>>>>>>>>> that's possible, you could bundle the JNI library inside the
> >> Okra
> >>>>>>>>>>> jar using a technique like
> >>>>>>>>>>> http://frommyplayground.com/how-to-load-native-jni-library-
> >> from-j
> >>>>>>>>>>> ar
> >>>>>>>>>>> /
> >>>>>>>>> to avoid having to set up PATH or LD_LIBRARY_PATH.
> >>>>>>>>>>>
> >>>>>>>>>>> -Doug
> >>>>>>>>>>>
> >>>>>>>>>>>>> [1]
> >>>>>>>>>>>>> https://github.com/HSAFoundation/Okra-Interface-to-HSAIL-
> >> Simula
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> r#
> >>>>>>>>>>>>> ok
> >>>>>>>>>>>>> ra-interface-to-hsail-simulator
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>> From: graal-dev-bounces at openjdk.java.net [mailto:graal-
> >> dev-
> >>>>>>>>>>>>>>>> bounces at openjdk.java.net] On Behalf Of Deneau, Tom
> >>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 11:24 AM
> >>>>>>>>>>>>>>>> To: Doug Simon
> >>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
> >>>>>>>>>>>>>>>> Subject: RE: hsail gate commands
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> OK, I see why there is no error when running mx --vm
> >> server
> >>>>>>>>>>>>>>>> unittest hsail
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> At some point (I don't recall why, maybe at Oracle's
> >>>>>>>>>>>>>>>> request??), we put in some code in KernelTester that if
> >> it
> >>>>>>>>>>>>>>>> could not find the okra simulator files would just
> >> silently
> >>>>>>>>>>>>>>>> not run the tests (which I guess counts as a pass).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> No the hsail simulator is not a java app at all.
> >>>>>>>>>>>>>>>> The page at
> >>>>>>>>>>>>>>>>
> >> https://wiki.openjdk.java.net/display/Sumatra/The+HSAIL+Simu
> >>>>>>>>>>>>>>>> la to r should describe how to build it and what
> >> environment
> >>>>>>>>>>>>>>>> variables to set up to use it...
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -- Tom
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 11:02 AM
> >>>>>>>>>>>>>>>>> To: Deneau, Tom
> >>>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
> >>>>>>>>>>>>>>>>> Subject: Re: hsail gate commands
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Dec 16, 2013, at 5:48 PM, Deneau, Tom
> >>>>>>>>>>>>>>>>> <tom.deneau at amd.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Doug --
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I see.
> >>>>>>>>>>>>>>>>>> I don't understand why the tests would run without
> >> error
> >>>>>>>>>>>>>>>>>> if the simulator and associated assembler is missing
> >>>>> but...
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What error are you seeing?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is there a way we can get the HSAIL simulator into
> >>>>>>>>>>>>>>>>>> the gate
> >>>>>>>>>>>>>>>>> infrastructure?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is it a pure Java app yet? That would certainly make
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>> trivial.
> >>>>>>>>>>>>>>>>> In any case, I'll defer to Bernhard or Gilles to
> >>>>>>>>>>>>>>>>> answer this since they are the maintainers of this
> >> infrastructure.
> >>>>>>>>>>>>>>>>> Can you please send instructions on how to
> >>>>>>>>>>>>>>>>> install/use/configure the
> >>>>>>>>>>> simulator.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Doug
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>>>> From: Doug Simon [mailto:doug.simon at oracle.com]
> >>>>>>>>>>>>>>>>>>> Sent: Monday, December 16, 2013 10:36 AM
> >>>>>>>>>>>>>>>>>>> To: Deneau, Tom
> >>>>>>>>>>>>>>>>>>> Cc: graal-dev at openjdk.java.net
> >>>>>>>>>>>>>>>>>>> Subject: Re: hsail gate commands
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Dec 16, 2013, at 5:21 PM, Deneau, Tom
> >>>>>>>>>>>>>>>>>>> <tom.deneau at amd.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Doug --
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I noticed in syncing with the trunk as of last
> >>>>>>>>>>>>>>>>>>>> Friday that all of our
> >>>>>>>>>>>>>>>>>>> HSAIL test cases broke. The cause was some
> >>>>>>>>>>>>>>>>>>> imperfect code in our HSAILAssembler for compare
> >>>>>>>>>>>>>>>>>>> instructions
> >> which
> >>>>>>>>>>>>>>>>>>> when presented with an unordered compare could
> >> generate
> >>>>>>>>>>>>>>>>>>> code that would not assemble, and was easy to fix.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> But I was wondering how this made it past the gate.
> >>>>>>>>>>>>>>>>>>>> Can you describe what gate commands are used
> >> regarding
> >>>>>>>>> hsail?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We simply run 'mx gate' which will include running
> >>>>>>>>>>>>>>>>>>> all the HSAIL unit tests.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On my Mac with the latest bits:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> $ mx --vm server unittest hsail executing junit
> >>>>>>>>>>>>>>>>>>> tests
> >>>>>>> now...
> >>>>>>>>>>>>>>>>>>> (107 test classes) JUnit version 4.8
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>
> ........................................................................
> >>>>>>>>>>>>>>>>>>> ...........I................I......
> >>>>>>>>>>>>>>>>>>> Time: 5.525
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> OK (105 tests)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Keep in mind I (and the gate infrastructure) don't
> >> have
> >>>>>>>>>>>>>>>>>>> the HSAIL simulator.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Doug
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
>
More information about the graal-dev
mailing list