[ZGC] [aarch64] Unable to allocate heap for certain Linux kernel configurations

[Stuart Monteith]

On 02/09/2020 08:12, Christoph Göttschkes wrote:
> Thanks for your early feedback. I also would like to hear if someone knows some problems with msync being used like 
> that. I made some more tests and until now, everything looks fine, no weird behavior found. Regardless of the kind of 
> mapping, msync always succeeds and only returns ENOMEM if the page is not mapped, or the address is invalid.
> 
> Yes, the patch was self contained, since I basically copied it from a self contained test program, only to share it with 
> you and to quickly test it within the JVM, to see if the JVM already runs multi threaded while the ZGC initialization is 
> done. When I am done with my testing, I will create an RFR for the created bug with this technique to get some more 
> feedback on the topic.
> 
> Thanks,
> Christoph
> 
> On 2020-08-31 12:47, Stefan Karlsson wrote:
>> On 2020-08-31 10:29, Christoph Göttschkes wrote:
>>> Hi Stuart.
>>>
>>> On 2020-08-28 16:56, Stuart Monteith wrote:
>>>> Hi,
>>>>      That's right - I have been exploring options on this, and I had a similar solution at one point to finding the 
>>>> address space size. From speaking with people familiar with the arm64 linux kernel, there is no good way to query 
>>>> the available address space except for probining it and testing what is there. Thinking we could do with a 
>>>> general-purpose routine, I experimented with a routine that forks the process and probes the address space 
>>>> non-destructively. MAP_FIXED implicitly destroys any existing mappings. Of course, ZGC mmaps memory at fixed 
>>>> addresses anyhow, so the concern about embedded the JVM in your program and destroying existing mappings turned out 
>>>> to be moot, as we'd be doing that anyway.
>>> After reading your comment, the only other viable solution I came up with is using a combination of msync and mmap. I 
>>> didn't fully look into this yet, but made a small prototype to show you the idea and to get early feedback. Maybe 
>>> someone already looked into this and knows that some edge case doesn't work.
>>>
>>> General idea:
>>> First use msync to check if there is already a mapping for the page. If msync returns ENOMEM, this either means: 
>>> there is no mapping yet, or the address is invalid. After getting ENOMEM, we can use mmap to try and map the page. If 
>>> we get back the same address, we know the address is valid. If the address is different, we know it is invalid. I 
>>> also don't want to use MAP_FIXED, but maybe MAP_FIXED_NOREPLACE (as mentioned by Florian) would be a solution, but it 
>>> was only introduced in Linux 4.17, so I guess we would need a fallback solution.
>>>
>>> I tested this on different Linux aarch64 devices and it seems to work. You can find the prototype here:
>>>
>>> https://cr.openjdk.java.net/~cgo/8252500/prototype-webrev.01/
>>
>> General approach seems fine to me. Maybe someone more well-versed in msync can chime in?
>>
>> Some comments about the patch:
>>
>> - Maybe check for EINVAL and/or add an assert to catch the odd error codes in debug builds?
>>
>> - If no max bits are found, the subsequent code will underflow. I think returning some kind of lower-limit that works 
>> with the code in ZPlatformAddressOffsetBits would be prudent.
>>
>> A few things that will make the code look more like HotSpot code:
>>
>> - We usually use os::vm_pages_size() instead of sysconf(_SC_PAGE_SIZE).
>>
>> - 'sizeof(uintptr_t) * CHAR_BIT' maybe replace with BitsPerWord
>>
>> - %zu use SIZE_FORMAT instead
>>
>> Thanks,
>> StefanK
>>
>>>
>>> -- Christoph
>>>
>>
> 
Hi,
	You approach + Stefan's comments looks reasonable to me . I tested your solution standalone on a Raspbery Pi 4 with 39 
bit VA and on a workstation with 48-bit addressing.

The configurations we support in Linux on arch64 are:
	4KB pages: 39, 48 bits.
	16KB pages: 36, 47, 48 bits
	64KB pages:  42, 48, 52 bits.

Only the 4KB and 64KB configurations are worth considering, I don't believe there are 16KB page configurations out 
there, so you could cut the range down to a 52 to 39 bits search.

For completeness I should mention that this potentially could be cross platform code. 5-level paging was added to x86, 
so they will have 57 bits available. ppc/s390x don't have ZGC ports yet.

My advice would be to stick with 48 bits for now, as there may be extra considerations with the 52/57 bit address spaces.

BR,
	Stuart