Review Request: UseNUMAInterleaving

[Deneau, Tom]

David --

The BIOS interleaving option interleaves at a 4K granularity.
The OS is completely unaware of this interleaving so yes it will work with any kernel and it will also work with large pages (The 2M of contiguous physical address covering a large page will be interleaved across nodes at 4K granularity).  Some downsides are:
   * it affects everything running on the system, whereas the numactl -i solution
     (or the proposed UseNUMAInterleaving) affects only that single process
   * not all BIOSes support this option

-- Tom
   

-----Original Message-----
From: David Dabbs [mailto:dmdabbs at gmail.com] 
Sent: Monday, May 16, 2011 4:34 PM
To: 'Igor Veresov'
Cc: Deneau, Tom; hotspot-compiler-dev at openjdk.java.net
Subject: RE: Review Request: UseNUMAInterleaving



> -----Original Message-----
> From: Igor Veresov [mailto:igor.veresov at oracle.com]
> Sent: Monday, May 16, 2011 4:28 PM
> To: David Dabbs
> Cc: 'Deneau, Tom'; hotspot-compiler-dev at openjdk.java.net
> Subject: Re: Review Request: UseNUMAInterleaving
> 
> On 5/16/11 1:04 PM, David Dabbs wrote:
> >
> >
> >> -----Original Message-----
> >> From: Igor Veresov
> >> Sent: Monday, May 16, 2011 2:54 PM
> >> To: David Dabbs
> >> Subject: Re: Review Request: UseNUMAInterleaving
> >>
> >> On 5/16/11 11:21 AM, David Dabbs wrote:
> >>
> >> The 2.6.19 requirement is there because we need the sched_getcpu()
> >> syscall to determine on which CPU we're running.
> >>
> >> I guess it won't be hard to extend UseNUMAInterleaving to work on
> other
> >> OSes, including old Linuxes. You can, however, achieve the same
> effect
> >> by running the VM with "numactl -i all java<your-options>" or by
> >> turning interleaving on in BIOS in hardware.
> >>
> >> igor
> >>
> >
> > Thank you for the info, Igor. So these two options you provided
> should
> > work for an "old Linux" (2.6.18, glibc-2.5-49.el5_5.7)? Are they
> equivalent?
> >
> 
> I don't actually recall what interleaving granularity will the hardware
> option provide. It could be less than a page (which would be the
> minimum
> granularity of any software method). Tom probably knows more about
> that.
> 
> Also, the numactl method won't work if you're using large pages.
> 
> igor
> 


I was afraid LargePages would be a deal-killer. We use a 2M page size. Oh
well.
Thank you.

David


> >
> > Thank you,
> >
> > David
> >
> >
> >
> >>>
> >>>
> >>> Could this flag help Linux systems with kernel<   2.6.19, or is
> that
> >> the
> >>> minimum kernel needed for any JVM NUMA support?
> >>> Unfortunately, we run CentOS 5.5 (2.6.18)
> >>>
> >>> Linux node01.int 2.6.18-194.17.4.el5 #1 SMP Mon Oct 25 15:50:53 EDT
> >> 2010
> >>> x86_64 x86_64 x86_64 GNU/Linux
> >>>
> >>> and so -XX:+UseNUMA does not activate (at least not according to
> >>> PrintFlagsFinal).
> >>>
> >>>>  From http://www.infoq.com/news/2010/01/java6u18
> >>> In the Java HotSpot VM, the NUMA-aware allocator has been
> implemented
> >> to
> >>> provide automatic memory placement optimisations for Java
> >> applications.
> >>> Typically, every processor in the system has a local memory that
> >> provides
> >>> low access latency and high bandwidth, and remote memory that is
> >>> considerably slower to access. The NUMA-aware allocator is
> >> implemented for
> >>> Solaris (>= 9u2) and Linux (kernel>= 2.6.19, glibc>= 2.6.1)
> operating
> >>> systems, and can be turned on for the Parallel Scavenger garbage
> >> collector
> >>> with the -XX:+UseNUMA flag. Parallel Scavenger remains the default
> >> for a
> >>> server-class machine and can also be turned on explicitly by
> >> specifying the
> >>> -XX:+UseParallelGC option. The impact of the change is significant:
> >> When
> >>> evaluated against the SPEC JBB 2005 benchmark on an 8 chip Opteron
> >> machine,
> >>> NUMA-aware systems gave about a 30% (for 32-bit) to 40% (for 64-
> bit)
> >>> increase in performance.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> David
> >>>
> >>>
> >>>
> >>>
> >
> >