[ZGC] [aarch64] Unable to allocate heap for certain Linux kernel configurations

[Christoph Göttschkes]

Thanks for your early feedback. I also would like to hear if someone 
knows some problems with msync being used like that. I made some more 
tests and until now, everything looks fine, no weird behavior found. 
Regardless of the kind of mapping, msync always succeeds and only 
returns ENOMEM if the page is not mapped, or the address is invalid.

Yes, the patch was self contained, since I basically copied it from a 
self contained test program, only to share it with you and to quickly 
test it within the JVM, to see if the JVM already runs multi threaded 
while the ZGC initialization is done. When I am done with my testing, I 
will create an RFR for the created bug with this technique to get some 
more feedback on the topic.

Thanks,
Christoph

On 2020-08-31 12:47, Stefan Karlsson wrote:
> On 2020-08-31 10:29, Christoph Göttschkes wrote:
>> Hi Stuart.
>>
>> On 2020-08-28 16:56, Stuart Monteith wrote:
>>> Hi,
>>>      That's right - I have been exploring options on this, and I had 
>>> a similar solution at one point to finding the address space size. 
>>> From speaking with people familiar with the arm64 linux kernel, there 
>>> is no good way to query the available address space except for 
>>> probining it and testing what is there. Thinking we could do with a 
>>> general-purpose routine, I experimented with a routine that forks the 
>>> process and probes the address space non-destructively. MAP_FIXED 
>>> implicitly destroys any existing mappings. Of course, ZGC mmaps 
>>> memory at fixed addresses anyhow, so the concern about embedded the 
>>> JVM in your program and destroying existing mappings turned out to be 
>>> moot, as we'd be doing that anyway.
>> After reading your comment, the only other viable solution I came up 
>> with is using a combination of msync and mmap. I didn't fully look 
>> into this yet, but made a small prototype to show you the idea and to 
>> get early feedback. Maybe someone already looked into this and knows 
>> that some edge case doesn't work.
>>
>> General idea:
>> First use msync to check if there is already a mapping for the page. 
>> If msync returns ENOMEM, this either means: there is no mapping yet, 
>> or the address is invalid. After getting ENOMEM, we can use mmap to 
>> try and map the page. If we get back the same address, we know the 
>> address is valid. If the address is different, we know it is invalid. 
>> I also don't want to use MAP_FIXED, but maybe MAP_FIXED_NOREPLACE (as 
>> mentioned by Florian) would be a solution, but it was only introduced 
>> in Linux 4.17, so I guess we would need a fallback solution.
>>
>> I tested this on different Linux aarch64 devices and it seems to work. 
>> You can find the prototype here:
>>
>> https://cr.openjdk.java.net/~cgo/8252500/prototype-webrev.01/
> 
> General approach seems fine to me. Maybe someone more well-versed in 
> msync can chime in?
> 
> Some comments about the patch:
> 
> - Maybe check for EINVAL and/or add an assert to catch the odd error 
> codes in debug builds?
> 
> - If no max bits are found, the subsequent code will underflow. I think 
> returning some kind of lower-limit that works with the code in 
> ZPlatformAddressOffsetBits would be prudent.
> 
> A few things that will make the code look more like HotSpot code:
> 
> - We usually use os::vm_pages_size() instead of sysconf(_SC_PAGE_SIZE).
> 
> - 'sizeof(uintptr_t) * CHAR_BIT' maybe replace with BitsPerWord
> 
> - %zu use SIZE_FORMAT instead
> 
> Thanks,
> StefanK
> 
>>
>> -- Christoph
>>
>