Review Request: UseNUMAInterleaving

[Deneau, Tom]

This is fine with me.

The only reason to have a separate flag would be on a platform which supported full UseNUMA and someone really wanted to use only the interleaving part.  However, I can't think of a reason for doing that.

Waiting until others get a chance to review the original, then I will add this to the next webrev submission.

-- Tom

> -----Original Message-----
> From: Paul Hohensee [mailto:paul.hohensee at oracle.com]
> Sent: Monday, May 16, 2011 6:19 PM
> To: Deneau, Tom
> Cc: David Dabbs; Igor Veresov; hotspot-compiler-dev at openjdk.java.net
> Subject: Re: Review Request: UseNUMAInterleaving
> 
> I suggest you just use -XX:+UseNUMA rather than adding a new flag. UseNUMA
> seems generic enough to cover whatever implementation is best on a
> particular platform.
> 
> Paul
> 
> Sent from my iPhone
> 
> On May 16, 2011, at 6:26 PM, "Deneau, Tom" <tom.deneau at amd.com> wrote:
> 
> > David --
> >
> > The BIOS interleaving option interleaves at a 4K granularity.
> > The OS is completely unaware of this interleaving so yes it will work
> with any kernel and it will also work with large pages (The 2M of
> contiguous physical address covering a large page will be interleaved
> across nodes at 4K granularity).  Some downsides are:
> >   * it affects everything running on the system, whereas the numactl -i
> solution
> >     (or the proposed UseNUMAInterleaving) affects only that single
> process
> >   * not all BIOSes support this option
> >
> > -- Tom
> >
> >
> > -----Original Message-----
> > From: David Dabbs [mailto:dmdabbs at gmail.com]
> > Sent: Monday, May 16, 2011 4:34 PM
> > To: 'Igor Veresov'
> > Cc: Deneau, Tom; hotspot-compiler-dev at openjdk.java.net
> > Subject: RE: Review Request: UseNUMAInterleaving
> >
> >
> >
> >> -----Original Message-----
> >> From: Igor Veresov [mailto:igor.veresov at oracle.com]
> >> Sent: Monday, May 16, 2011 4:28 PM
> >> To: David Dabbs
> >> Cc: 'Deneau, Tom'; hotspot-compiler-dev at openjdk.java.net
> >> Subject: Re: Review Request: UseNUMAInterleaving
> >>
> >> On 5/16/11 1:04 PM, David Dabbs wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Igor Veresov
> >>>> Sent: Monday, May 16, 2011 2:54 PM
> >>>> To: David Dabbs
> >>>> Subject: Re: Review Request: UseNUMAInterleaving
> >>>>
> >>>> On 5/16/11 11:21 AM, David Dabbs wrote:
> >>>>
> >>>> The 2.6.19 requirement is there because we need the sched_getcpu()
> >>>> syscall to determine on which CPU we're running.
> >>>>
> >>>> I guess it won't be hard to extend UseNUMAInterleaving to work on
> >> other
> >>>> OSes, including old Linuxes. You can, however, achieve the same
> >> effect
> >>>> by running the VM with "numactl -i all java<your-options>" or by
> >>>> turning interleaving on in BIOS in hardware.
> >>>>
> >>>> igor
> >>>>
> >>>
> >>> Thank you for the info, Igor. So these two options you provided
> >> should
> >>> work for an "old Linux" (2.6.18, glibc-2.5-49.el5_5.7)? Are they
> >> equivalent?
> >>>
> >>
> >> I don't actually recall what interleaving granularity will the hardware
> >> option provide. It could be less than a page (which would be the
> >> minimum
> >> granularity of any software method). Tom probably knows more about
> >> that.
> >>
> >> Also, the numactl method won't work if you're using large pages.
> >>
> >> igor
> >>
> >
> >
> > I was afraid LargePages would be a deal-killer. We use a 2M page size.
> Oh
> > well.
> > Thank you.
> >
> > David
> >
> >
> >>>
> >>> Thank you,
> >>>
> >>> David
> >>>
> >>>
> >>>
> >>>>>
> >>>>>
> >>>>> Could this flag help Linux systems with kernel<   2.6.19, or is
> >> that
> >>>> the
> >>>>> minimum kernel needed for any JVM NUMA support?
> >>>>> Unfortunately, we run CentOS 5.5 (2.6.18)
> >>>>>
> >>>>> Linux node01.int 2.6.18-194.17.4.el5 #1 SMP Mon Oct 25 15:50:53 EDT
> >>>> 2010
> >>>>> x86_64 x86_64 x86_64 GNU/Linux
> >>>>>
> >>>>> and so -XX:+UseNUMA does not activate (at least not according to
> >>>>> PrintFlagsFinal).
> >>>>>
> >>>>>> From http://www.infoq.com/news/2010/01/java6u18
> >>>>> In the Java HotSpot VM, the NUMA-aware allocator has been
> >> implemented
> >>>> to
> >>>>> provide automatic memory placement optimisations for Java
> >>>> applications.
> >>>>> Typically, every processor in the system has a local memory that
> >>>> provides
> >>>>> low access latency and high bandwidth, and remote memory that is
> >>>>> considerably slower to access. The NUMA-aware allocator is
> >>>> implemented for
> >>>>> Solaris (>= 9u2) and Linux (kernel>= 2.6.19, glibc>= 2.6.1)
> >> operating
> >>>>> systems, and can be turned on for the Parallel Scavenger garbage
> >>>> collector
> >>>>> with the -XX:+UseNUMA flag. Parallel Scavenger remains the default
> >>>> for a
> >>>>> server-class machine and can also be turned on explicitly by
> >>>> specifying the
> >>>>> -XX:+UseParallelGC option. The impact of the change is significant:
> >>>> When
> >>>>> evaluated against the SPEC JBB 2005 benchmark on an 8 chip Opteron
> >>>> machine,
> >>>>> NUMA-aware systems gave about a 30% (for 32-bit) to 40% (for 64-
> >> bit)
> >>>>> increase in performance.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> David
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
> >
> >
> >