String-dedup test failures

[Aleksey Shipilev]

On 11/16/2017 10:25 AM, Roman Kennke wrote:
> I think we need to clean up the string dedup table *after marking*. If a byte[] is not reachable
> after marking, we need to clean all references to it, because *it may become trashed* during evac.
> If we want to keep it alive, then we need to mark it. Evac'ing an unmarked byte[] after marking is
> calling for troubles, because we probably access a pointer that is about to be invalidated.

Yes, that's what I was suggesting:

It seems that we want two separate modes: regular, and cleanup. Much like we do with "Parallel
Cleanup" during Final Mark. In regular mode, we mark all StrDedup entries always, maybe even
concurrently during mark? In cleanup mode, we don't mark StrDedup as roots, but at Final Mark we
purge all stale entries from there. Then if any String cannot "revive" the stale unmarked entry, and
have to insert new, reachable one. I think the mechanism above has the nice symmetry with class
unloading, etc., which makes it easier to reason about. Also makes cleanup during pause optional.

-Aleksey