RFR: 8297138: UB leading to crash in Amalloc with optimized builds

[Thomas Stuefe]

On Tue, 29 Nov 2022 03:45:54 GMT, David Holmes <dholmes at openjdk.org> wrote:

> I think we can close this PR in favour of a RFE that simply removes `UseMallocOnly`. It may have been useful whilst VM memory management was in its infancy (it was introduced in 1998) but now it is just a liability. I think we can simply remove it (and not worry if Arenas need better debuggability or error tracking - that would be a separate RFE).

I'm very much for it!

But we may need a way to detect memory stomping across arena allocations. See https://bugs.openjdk.org/browse/JDK-8007475, and hotspot/jtreg/runtime/8007475/StackMapFrameTest.java. I have used UseMallocOnly for similar purposes in the past.

As I wrote before when I did some Arena code massaging last year (8263557, 8263558, etc, see git log) I also started to add per-arena-allocation-canaries. I hit some minor road blocked and then stopped because I was not sure it was even wanted. 

Having canaries in Arenas would also have the advantage of being both faster and leaner than what we do now with C-heap allocations. So we could afford to enable them for more tests.

So if you all agree, I could bookmark some time early in the jdk20 time frame to revive my patch. We could remove UseMallocOnly now, or do it together with the proposed canary change.

-------------

PR: https://git.openjdk.org/jdk/pull/11320