shenandoah c2 compiler crash in jdk8

[Roland Westrelin]

> I think that the qn that needs to be answered is whether there is a `leak`
> in nodes or you just need more of them. Does that make sense? If so, is
> there an option to dump c2 node usage over time?

The assert failure you posted happens at barrier expansion
time. Shenandoah's barriers are initially inserted as macro instructions
in the c2 IR and as a final step in the optimization process, they are
expanded. It seems that what happens is that the c2 IR gets close to its
limit size before barrier expansion and then expansion pushes the IR
size over the limit.

It makes sense that bumping MaxNodeLimit works around the problem but I
don't see that as a robust fix. With a large MaxNodeLimit, C2 would have
more head room to transform the IR before barrier expansion. So nothing
says that in some cases, c2 wouldn't end up at barrier expansion with a
much larger IR that would cause expansion to fail in the same way.

I don't see how a leak of nodes would happen in this scenario. During
expansion, barriers are expanded one by one. There's no creation of new
barriers. I don't expect "node usage over time" to give a picture that's
clear enough of what's going on either.

Roland.