non-foreign-call tlab refill from hsail

[Deneau, Tom]

Doug --

Trying to see if I understand how these pieces fit together.

NewObjectSnippets.allocateInstance makes a call to
NewInstanceStubCall.call if the current tlab does not have enough
room.

NewInstanceStubCall.call looks up the ForeignCallLinkage and finds
that it is not a simple foreign call to a specific foreign call
address (its address is 0) but instead has a stub associated with it.
I think this association came from the call to

        link(new NewInstanceStub(providers, target, registerStubCall(NEW_INSTANCE, REEXECUTABLE, NOT_LEAF, ANY_LOCATION)));

in HotSpotHostForeingCallsProvider.java

So when we try to finalizeAddress for the ForeignCallLinkage we end up
compiling this stub.

The stub is a SnippetStub implemented with the snippet called
"newInstance" in NewInstanceStub.java and tries to get a new tlab
using CAS operations.  If this stub cannot get a new tlab it makes a
"real" foreign call using
        newInstanceC(NEW_INSTANCE_C, thread(), hub);

which ends up going to the graalRuntime::new_instance


-----Original Message-----
From: Doug Simon [mailto:doug.simon at oracle.com] 
Sent: Tuesday, October 22, 2013 4:42 AM
To: Deneau, Tom
Cc: graal-dev at openjdk.java.net; sumatra-dev at openjdk.java.net
Subject: Re: non-foreign-call tlab refill from hsail


On Oct 22, 2013, at 12:18 AM, "Deneau, Tom" <tom.deneau at amd.com> wrote:

> We are experimenting with object (and array) allocation from an HSA device (using graal for
> 
> the HSAIL codegen).  Where we are now:
> 
> 
> 
>   * the hsa workitems are using TLABs from "donor threads" who exist
> 
>     just to supply TLABs and don't do any allocation themselves.
> 
> 
> 
>   * To reduce the number of donor threads required, a TLAB can be
> 
>     used by more than one workitem, in which case the workitems use
> 
>     HSAIL atomic_add instructions to bump the tlab top pointer.
> 
> 
> 
>      * the HSAIL backend has its own fastpath allocation snippets
> 
>        which generate the HSAIL atomic_add instructions which
> 
>        override the snippets in NewObjectSnippets.java
> 
> 
> 
> Some junit tests have been written and pass which allocate objects, or arrays of primitives, or arrays of objects.
> 
> 
> 
> All the above only works for the fastpath case, i.e., if there is indeed enough space in the donor TLAB.  We realize there are other cases:
> 
> 
> 
>   a) not enough space in current TLAB but ability to allocate a new TLAB.
> 
> 
> 
>   b) not able to allocate a new TLAB, GC required.
> 
> 
> 
> For only case a) above, we would like to experiment with grabbing the new TLAB from HSAIL without making a "foreign call" to the VM.  From the hotspot code, I assume the logic required is what one sees in
> 
>    mutableSpace::cas_allocate(size_t size) at least for the non-G1 case.

When the NewInstanceStub fails to allocate a new TLAB, it calls out to GraalRuntime::new_instance (in graalRuntime.cpp).

> Some of this non-foreign-call allocation logic appears to exist in the Snippet called NewInstanceStub.newInstance (as opposed to NewObjectSnippets.allocateInstance snippet which is what we are currently overriding).  This comments for this snippet say
> 
>   "Re-attempts allocation after an initial TLAB allocation failed or
> 
>   was skipped (e.g., due to * -XX:-UseTLAB)."
> 
> 
> 
> Is this NewInstanceStub.newInstance snippet actually used anywhere in the current graal framework.

Yes, you can see a call to NewInstanceStubCall in NewObjectSnippets.allocateInstance.

>  Is this a starting point we could use to get a non-foreign-call TLAB refill working?

Yes. Note that this call *is* a foreign call (see the javadoc for ForeignCallDescriptor).

-Doug