RFR (L): 8058354: SPECjvm2008-Derby -2.7% performance regression on Solaris-X64 starting with 9-b29

[Thomas Schatzl]

Hi,

On Thu, 2015-01-29 at 13:25 +0100, Stefan Karlsson wrote:
> Hi Thomas,
> 
> On 2015-01-29 11:30, Thomas Schatzl wrote:
> > Hi all,
> >
> >    can I have reviews for the following change that fixes the use of
> > large pages for auxiliary data on G1?
> >
> > In JDK-8038423 there has been a large change in how G1 handles virtual
> > memory, and overlooked that auxiliary data may use large pages. This
> > caused some performance regressions after introducing that build.
> >
> > This change fixes this problem: G1 is now more flexible in using large
> > pages: particularly auxiliary data that often is not sized to multiples
> > of (large) page size suffers from that. By allowing the virtual space
> > implementation to use small pages on the tail (upper) end of the virtual
> > space, everything else can use large pages.
> >
> > There is one limitation to that: the start address of the used virtual
> > spaces must be aligned to large page size to use large pages in
> > auxiliary data. This is to simplify commit and uncommit within the
> > regions, since very small areas in the auxiliary data can map to large
> > areas in the heap (e.g. for the BOT, at 4k page size, one page maps to
> > 2M of memory, 2M pages map to 1G of memory).
> >
> > The problem is, if the start address of such auxiliary data were not
> > aligned to requested page size, we would potentially need to split
> > neighbouring large pages if we tried to uncommit one.
> >
> > I.e. some ascii art showing the problem.
> >
> > AAAAAA AAAAAA AAAAAA  // heap area, each AAAAAA is a single region
> >     |      |      |    // area covered by auxilary pages
> >        1       2       // auxiliary data pages
> >
> > So if auxiliary data pages were unaligned, so that they correspond to
> > uneven multiples of the heap, when uncommitting e.g. the second region
> > (second set of AAAAAA), we would have to split the auxiliary data pages
> > 1 and 2 into smaller ones.
> >
> > That does not seem to be a good tradeoff in complexity, given that the
> > waste is at most one large page in reserved space (and unfortunately,
> > due to the Linux large page implementation also in actually used space).
> >
> > Changes in detail containing some additional fixes:
> >   - page selection corresponds to other collectors, i.e. if some
> > auxiliary data covers at least one large page, try to use large pages if
> > available.
> >   - fix CMBitMap::compute_size() to align to alignment granularity (this
> > has not been a real problem because the actual size is always a multiple
> > of that)
> >   - allow (very restricted) mixed use of small and large pages in the G1
> > heap.
> >   - pass on alignment hints to os::commit()
> >   - some refactoring extracting out the code to reserve auxiliary memory
> > structures
> >
> > With these changes, performance when using large pages is at least as
> > good as before 9b29.
> >
> > Webrev:
> > http://cr.openjdk.java.net/~tschatzl/8058354/webrev/
> >
> > CR:
> > https://bugs.openjdk.java.net/browse/JDK-8058354
> 
> ReservedSpace(size_t) already support the ability to map large pages in 
> the middle of a memory region. Given your example:
> 
> AAAAAA AAAAAA AAAAAA
> 
>     |      |      |
>        1       2
> 
> 1 and 2 will use large pages.

(I think you mean VirtualSpace, not ReservedSpace. VirtualSpace does the
commit/uncommit although on Linux, ReservedSpace also commits memory
sometimes...)

The problem is not getting the initial large pages, the problem is
managing uncommitting of parts of that area in the presence of attempts
of uncommitting random parts.

Maybe that particular example does not show the extent of the problem
for G1: consider that we have thousands of regions, and hundreds of
pages some auxiliary data structure consists of.

The use case is then, uncommit the auxiliary data corresponding to
region 2-5, 30-34, 120. VirtualSpace does not support that at all.

If the areas that auxiliary data map to were not aligned to something
that is a multiple of the page sizes, we could not do that (easily).

> There was a recent change that broke that, but it's supposed to be fixed 
> with:
> changeset:   7656:4321214d5dbc
> parent:      7654:8dfd8b00c7f1
> user:        ehelin
> date:        Fri Jan 16 10:29:12 2015 +0100
> summary:     8066875: VirtualSpace does not use large pages
> 
> Have  you rerun the performance tests with that fix?

Yes. The problem still is that previously G1 did not even try to use
large pages for the auxiliary data (when implementing this originally,
that functionality got lost). Only the heap used large pages. It also
only supported one size for committed pages, which means you could
either use full small or large pages. That latter restriction has been
lifted for tail ends, which happens a lot for card table etc.

E.g. if you use 1025M of heap (for whatever reason), you get a card
table that consists of 1 large page (=2M) and one small page (4k) now
(at least on Solaris x64).

There is an (not-public) link to performance results for a before/after
run in the CR.

It shows that performance for the benchmarks tested is back to at least
9b28 levels. Only Solaris is affected btw.

Thanks,
  Thomas