[PATCH] Linux NUMA support for HotSpot

[Andi Kleen]

On Mon, Mar 03, 2008 at 04:25:53PM +0300, Igor Veresov wrote:
> On Monday 03 March 2008 15:57:19 Andi Kleen wrote:
> > On Mon, Mar 03, 2008 at 03:19:13PM +0300, Igor Veresov wrote:
> > > > > all working quite well, actually.
> > > >
> > > > One obvious issue I found in the Solaris code was that it neither
> > > > binds threads to nodes (afaik), nor tries to keep up with a thread
> > > > migrating to another node. It just assumes always the same thread:node
> > > > mapping which surely cannot be correct?
> > >
> > > It is however correct. Solaris assigns a home locality group (a node) to
> > > each lwp. And while lwps can be temporary migrated to a remote group,
> > > page allocation still happens on a home node and the lwp is predominantly
> > > scheduled to run in its home lgroup. For more information you could refer
> > > to
> >
> > Interesting. I tried a similar scheduling algorithm on Linux a long time
> > ago (it was called the "homenode scheduler") and it was a general loss on
> > my testing on smaller AMD systems. But maybe Solaris does it all different.
> >
> > Anyways on Linux that won't work because it doesn't have the concept
> > of a homenode.
> 
> Yes, but it has static memory binding instead, which alleviates this problem.

That would require statically binding the threads too which is by default not
a good idea without explicit user configuration

The reasoning is that not using a CPU is always worse than using 
remote memory at least on systems with reasonable small NUMA factor.

(that is what killed the homenode scheduler too) 

> 
> >
> > The other problems is that it seemed to always assume all the threads
> > will consume the whole system and set up for all nodes, which seemed dodgy.
> 
> You mean the allocator? Actually it is adaptive to the allocation rate on a 
> node, which in effect makes the whole eden space usable for applications with 
> asymmetric per-thread allocation rate. This of course also helps with the 
> case when the number of threads is less than the number of nodes.

It didn't seem to adapt though. Or maybe I'm misremembering the code,
it was some time ago.

-Andi