Review Request: UseNUMAInterleaving

Fri Aug 19 00:18:05 UTC 2011

 Actually no, tapping into reserve() functions won't work. 
I think we should put os::numa_make_global() in os::commit_memory() and os::reserve_memory_special().
The reason is that the mmap inside of commit_memory will override the allocation policy of the underlying segment allocated by reserve_memory().
It would be still ok to use os::commit_memory() to implement os::free_memory() on Linux because the numa-aware allocator we change the policy afterwards anyway (see MutableNUMASpace::bias_region() ), so no problem here.

igor

On Thursday, August 18, 2011 at 5:00 PM, Igor Veresov wrote:

> On Thursday, August 18, 2011 at 3:40 PM, Deneau, Tom wrote:
> > Igor --
> > 
> > For the linux/solaris support for UseNUMAInterleaving, you mentioned
> > we can just call os::numa_make_global at the end of
> > os::reserve_memory. Will this only have an effect for those
> > collectors that do not explicitly already support UseNUMA? What
> > would happen if UseNUMAInterleaving is used with a collector that
> > does support UseNUMA? Which collectors do not explicitly support
> > UseNUMA?
> I don't think anything bad is going to happen the way numa_make_global() is currently implemented on unixes. NUMA-aware collectors will just override this page-allocation policy with something else. So I guess for now we could just turn on UseNUMAInterleaving whenever UseNUMA is specified (but not the other way around) regardless of the collector. We will probably have to change that later when we fully support numa of windows, because numa_make_global() will be much more heavyweight there - you would have to deallocate a chunk and then reallocate it again; but for now let's keep it simple.
> 
> And, btw, I think you need to tap not only os::reserve_memory() but also os::attempt_reserve_memory_at(), and os::reserve_memory_special(). Basically everything that is used to allocate memory.
> 
> igor
> > -- Tom
> > 
> > 
> > > -----Original Message-----
> > > From: Igor Veresov [mailto:igor.veresov at oracle.com]
> > > Sent: Wednesday, August 17, 2011 6:38 PM
> > > To: Deneau, Tom
> > > Cc: hotspot-gc-dev at openjdk.java.net (mailto:hotspot-gc-dev at openjdk.java.net)
> > > Subject: Re: Review Request: UseNUMAInterleaving
> > > 
> > > On Tuesday, August 16, 2011 at 1:14 PM, Deneau, Tom wrote:
> > > > Igor --
> > > > 
> > > > I am back from vacation, starting to address your comments...
> > > > Regarding your comment #1 below.
> > > > 
> > > > You mention "future planned NUMA-aware implementations of GCs". How
> > > > do these future planned NUMA-aware implementations of GCs differ from
> > > > today's NUMA-aware GCs? My understanding of the current GCs use of
> > > > NUMA is that they support numa_global (interleaved) and numa_local
> > > > (memory pinned to one numa node).
> > > > 
> > > > In the currently released JVMs on Windows OSes, neither numa_local nor
> > > > numa_global is implemented. The implementation I proposed in the
> > > > patch maps both numa_local and numa_global requests to numa_global (on
> > > > Windows). The reasons for this were:
> > > > 
> > > >  * it was very difficult (if not impossible) to implement the JVM's
> > > >  current numa_local semantics on Windows
> > > It's hard to realize numa_local semantics if you want to minimize the
> > > number of memory segments per lgroup. If you're prepared (like in your
> > > patch) to have hundreds of thousands of segments, this is not a problem
> > > and it's quite easily implementable. The only problem there would that
> > > such a huge number of segments will penalize page fault handling a lot.
> > > > * in the benchmarks we measured, the extra performance that was
> > > >  left on the table by doing only numa_global and not doing
> > > >  numa_local was only a few percent.
> > > Hm, I have trouble believing that. How did you get such results? What
> > > were the experiments?
> > > 
> > > > Are you saying that in the future numa_local will be supported on
> > > > Windows, and that even then it might still be advantageous to have a
> > > > flag (UseNUMAInterleaving) which instead maps all the regions to
> > > > numa_global? Should this flag be available on all OSes?
> > > Basically yes. And like Ramki said it would be nice to support that on
> > > other OSes, so that we could at least get interleaving for the collectors
> > > that do no explicitly support NUMA. I guess I didn't do that before
> > > because the functionality is equivalent to just saying for example on
> > > Linux "numactl -i all java <flags>", but since you can't do that on
> > > windows (as far as I can see) we could support this flag on unixes as
> > > well. Which is fairly easy to do, you just have to call
> > > os::numa_make_global() for a freshly reserved region.
> > > 
> > > igor
> > > > -- Tom
> > > > 
> > > > > -----Original Message-----
> > > > > From: Igor Veresov [mailto:igor.veresov at oracle.com]
> > > > > Sent: Monday, August 08, 2011 1:43 PM
> > > > > To: hotspot-gc-dev at openjdk.java.net (mailto:hotspot-gc-
> > > dev at openjdk.java.net (mailto:dev at openjdk.java.net)); Deneau, Tom
> > > > > Subject: Re: Review Request: UseNUMAInterleaving
> > > > > 
> > > > > Hi, Tom!
> > > > > 
> > > > > Sorry it took me so long to get to that.
> > > > > 
> > > > > 1. I don't think the new version of flag usage is prudent. The reason
> > > I
> > > > > proposed to introduce a new flag for interleaving is that it would
> > > make
> > > > > life easier in the future when the proper NUMA-aware implementation
> > > of
> > > > > GCs are added (G1 would be the most probable candidate). I would
> > > propose
> > > > > to still have UseNUMAInterleaving flag.
> > > > > 
> > > > > The usage would be as follows:
> > > > > - If UseNUMA is specified on Windows that would turn
> > > UseNUMAInterleaving
> > > > > (for the time being, and that behavior would change in the future).
> > > > > - If UseNUMAInterleaving is specified on the command line, you just
> > > do
> > > > > the interleaving. If you don't add this flag now, you'll have to do
> > > that
> > > > > anyway as soon as NUMA-aware GCs start supporting windows.
> > > > > 
> > > > > 
> > > > > igor
> > > > > 
> > > > > 
> > > > > 
> > > > > On 5/26/11 4:37 PM, Deneau, Tom wrote:
> > > > > > I have incorporated the change suggested by Paul Hohensee to just
> > > use
> > > > > the existing UseNUMA flag rather than introduce a new flag. Please
> > > let me
> > > > > know when you think this will be able to be checked in...
> > > > > > 
> > > > > > The new webrev is at
> > > > > > http://cr.openjdk.java.net/~tdeneau/UseNUMAInterleaving/webrev.02/
> > > > > > 
> > > > > > -- Tom Deneau, AMD