Idea: aliased heap for checking to-space invariant

[Aleksey Shipilev]

On 01/30/2018 04:07 PM, Aleksey Shipilev wrote:
> So I have been walking and muttering to myself how we cannot mprotect(PROT_READ) the collection set,
> because we have to accept the fwdptr update in the same page. We used to mprotect cset for
> verification, but that code basically mprotect(PROT_WRITE)-ed the page when fwdptr write had
> faulted, restarted the fwdptr update, accepting everything else after that too. Thus it was became
> too racy to be useful. This was the reason for us to ditch that verification part, and instead rely
> on explicit ShenandoahStoreCheck machinery.
> 
> Then it hit me: the memory protection is enforced on virtual pages, not on physical pages, which
> means we can use the aliased heap to accept the fwdptr stores, while normal heap cset is protected
> from writes! I.e. have the normal heap WRITE|READ as usual, have the alias heap WRITE|READ as usual,
> then when cset is selected WRITE-protect the cset, and watch out for failures. The fwdptr updates
> from WB code should instead go via the aliased heap that is WRITE-enabled.
> 
> This gives us several advantages:
>   *) We capture all bad writes mechanically, instead of hoping we covered all ShStoreCheck cases
>   *) The upstream exposure in .ad and platform-specific macro-assemblers goes away
>   *) Roman's work on aliased heaps is not in vain :)
>   *) We don't arrive to the mess with "differently-shaped" pointers to both normal and aliased heap,
> because we never leak aliased heap pointers anywhere: we just use that as the location for the
> fwdptr CAS.
> 
> We can (and probably should) only enable this for verification, so we don't have any ill effects for
> non-verificated modes (which would just do the same thing they do today).

Implemented the prototype of this here:
  http://cr.openjdk.java.net/~shade/shenandoah/aliased-heap-storecheck/webrev.01/

It is rough around the edges, and not yet committable. But, it runs and fails (with deliberately
omitted WBs here and there) predictably, and it passes tier3_gc_shenandoah.

-Aleksey