non-foreign-call tlab refill from hsail

[Doug Simon]

On Nov 7, 2013, at 11:21 PM, Deneau, Tom <tom.deneau at amd.com> wrote:

> Are snippets required to inline all their calls?  

Generally speaking, yes.

> Or alternatively is there no way to annotate that a method should not be inlined?

You can use the Snippet.SnippetInliningPolicy class to control inlining during snippet preparation.

-Doug

> -----Original Message-----
> From: Doug Simon [mailto:doug.simon at oracle.com] 
> Sent: Thursday, November 07, 2013 4:15 PM
> To: Deneau, Tom
> Cc: graal-dev at openjdk.java.net; sumatra-dev at openjdk.java.net
> Subject: Re: non-foreign-call tlab refill from hsail
> 
> Because it is slow (well, medium) path code that we don't want to inline at every allocation site.
> 
> On Nov 7, 2013, at 11:13 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
> 
>> So I was trying to understand why the NewInstanceStub.newInstance Stub 
>> code was not just included in the original NewObjectSnippet.allocateInstance snippet.
>> 
>> -- Tom
>> 
>> 
>> -----Original Message-----
>> From: Doug Simon [mailto:doug.simon at oracle.com]
>> Sent: Thursday, November 07, 2013 4:04 PM
>> To: Deneau, Tom
>> Cc: graal-dev at openjdk.java.net; sumatra-dev at openjdk.java.net
>> Subject: Re: non-foreign-call tlab refill from hsail
>> 
>> That is a very correct summary of the way it works!
>> 
>> On Nov 7, 2013, at 10:22 PM, Deneau, Tom <tom.deneau at amd.com> wrote:
>> 
>>> Doug --
>>> 
>>> Trying to see if I understand how these pieces fit together.
>>> 
>>> NewObjectSnippets.allocateInstance makes a call to 
>>> NewInstanceStubCall.call if the current tlab does not have enough 
>>> room.
>>> 
>>> NewInstanceStubCall.call looks up the ForeignCallLinkage and finds 
>>> that it is not a simple foreign call to a specific foreign call 
>>> address (its address is 0) but instead has a stub associated with it.
>>> I think this association came from the call to
>>> 
>>>      link(new NewInstanceStub(providers, target, 
>>> registerStubCall(NEW_INSTANCE, REEXECUTABLE, NOT_LEAF, 
>>> ANY_LOCATION)));
>>> 
>>> in HotSpotHostForeingCallsProvider.java
>>> 
>>> So when we try to finalizeAddress for the ForeignCallLinkage we end 
>>> up compiling this stub.
>>> 
>>> The stub is a SnippetStub implemented with the snippet called 
>>> "newInstance" in NewInstanceStub.java and tries to get a new tlab 
>>> using CAS operations.  If this stub cannot get a new tlab it makes a 
>>> "real" foreign call using
>>>      newInstanceC(NEW_INSTANCE_C, thread(), hub);
>>> 
>>> which ends up going to the graalRuntime::new_instance
>>> 
>>> 
>>> -----Original Message-----
>>> From: Doug Simon [mailto:doug.simon at oracle.com]
>>> Sent: Tuesday, October 22, 2013 4:42 AM
>>> To: Deneau, Tom
>>> Cc: graal-dev at openjdk.java.net; sumatra-dev at openjdk.java.net
>>> Subject: Re: non-foreign-call tlab refill from hsail
>>> 
>>> 
>>> On Oct 22, 2013, at 12:18 AM, "Deneau, Tom" <tom.deneau at amd.com> wrote:
>>> 
>>>> We are experimenting with object (and array) allocation from an HSA 
>>>> device (using graal for
>>>> 
>>>> the HSAIL codegen).  Where we are now:
>>>> 
>>>> 
>>>> 
>>>> * the hsa workitems are using TLABs from "donor threads" who exist
>>>> 
>>>>  just to supply TLABs and don't do any allocation themselves.
>>>> 
>>>> 
>>>> 
>>>> * To reduce the number of donor threads required, a TLAB can be
>>>> 
>>>>  used by more than one workitem, in which case the workitems use
>>>> 
>>>>  HSAIL atomic_add instructions to bump the tlab top pointer.
>>>> 
>>>> 
>>>> 
>>>>   * the HSAIL backend has its own fastpath allocation snippets
>>>> 
>>>>     which generate the HSAIL atomic_add instructions which
>>>> 
>>>>     override the snippets in NewObjectSnippets.java
>>>> 
>>>> 
>>>> 
>>>> Some junit tests have been written and pass which allocate objects, or arrays of primitives, or arrays of objects.
>>>> 
>>>> 
>>>> 
>>>> All the above only works for the fastpath case, i.e., if there is indeed enough space in the donor TLAB.  We realize there are other cases:
>>>> 
>>>> 
>>>> 
>>>> a) not enough space in current TLAB but ability to allocate a new TLAB.
>>>> 
>>>> 
>>>> 
>>>> b) not able to allocate a new TLAB, GC required.
>>>> 
>>>> 
>>>> 
>>>> For only case a) above, we would like to experiment with grabbing 
>>>> the new TLAB from HSAIL without making a "foreign call" to the VM.  
>>>> From the hotspot code, I assume the logic required is what one sees 
>>>> in
>>>> 
>>>> mutableSpace::cas_allocate(size_t size) at least for the non-G1 case.
>>> 
>>> When the NewInstanceStub fails to allocate a new TLAB, it calls out to GraalRuntime::new_instance (in graalRuntime.cpp).
>>> 
>>>> Some of this non-foreign-call allocation logic appears to exist in 
>>>> the Snippet called NewInstanceStub.newInstance (as opposed to 
>>>> NewObjectSnippets.allocateInstance snippet which is what we are 
>>>> currently overriding).  This comments for this snippet say
>>>> 
>>>> "Re-attempts allocation after an initial TLAB allocation failed or
>>>> 
>>>> was skipped (e.g., due to * -XX:-UseTLAB)."
>>>> 
>>>> 
>>>> 
>>>> Is this NewInstanceStub.newInstance snippet actually used anywhere in the current graal framework.
>>> 
>>> Yes, you can see a call to NewInstanceStubCall in NewObjectSnippets.allocateInstance.
>>> 
>>>> Is this a starting point we could use to get a non-foreign-call TLAB refill working?
>>> 
>>> Yes. Note that this call *is* a foreign call (see the javadoc for ForeignCallDescriptor).
>>> 
>>> -Doug
>>> 
>> 
>> 
>> 
> 
> 
>