[ZGC] [aarch64] Unable to allocate heap for certain Linux kernel configurations

[Stefan Karlsson]

On 2020-08-28 16:56, Stuart Monteith wrote:
> On 28/08/2020 11:42, Stefan Karlsson wrote:
>> On 2020-08-28 11:38, Christoph Göttschkes wrote:
>>> Hi Stefan, thanks for your feedback. Looks like this case isn't as 
>>> exotic as I first thought. May I ask which kind of machine this is? 
>>> Also a small embedded board?
>>
>> It was one of our compiler devs that ran into this. I don't think it 
>> was a small machine, but rather that was configured differently then 
>> other AArch64 machines that we've run on.
>>
>>>
>>> On 2020-08-28 11:04, Stefan Karlsson wrote:
>>>> I think we hit a very similar problem during some internal testing 
>>>> on one machine. I have a patch to workaround that problem:
>>>>
>>>> https://cr.openjdk.java.net/~stefank/prototype/zaarch-va/webrev.01/
>>>
>>> Your patch works with some modifications. In my case, only 39 bits 
>>> are available in the virtual address space. I put that value as 
>>> "va_bits" and it works.
>>
>> OK. Good to know.
>>
>>>
>>>> Unfortunately, this patch only solves the problem on a very specific 
>>>> setup, and I don't think it covers your use-case. Hopefully, someone 
>>>> with enough AArch64 machine config knowledge would be able to extend 
>>>> this patch to also cover all possible combinations.
>>>
>>> I don't think the possible combinations are the problem. [1] shows 
>>> them (sorry, didn't put that link in the last mail) for Linux. I 
>>> think the real problem is detecting this and making the addressing 
>>> scheme adjust itself.
>>>
>>> Maybe there could be a mechanism which tries to allocate memory 
>>> beyond certain addresses to try and detect the number of bits 
>>> available? On my machine, for instance, the ZGC implementation tries 
>>> to allocate memory with different starting addresses, but always gets 
>>> an address back which is way smaller (because of the kernel 
>>> limitations). Maybe, the ZGC implementation could store this 
>>> information (the number of bits in addresses returned by mmap) and 
>>> use this information to try and make another loop, which tries to 
>>> allocate the heap with a reduced number of bits used for the 
>>> addresses. This could also be a HotSpot option, to speed things up 
>>> during startup if one knows that the machine uses a "weird" 
>>> configuration.
>>>
>>>> I think creating a bug report would be a good start. Do you have an 
>>>> openjdk user name? If not I can create a bug report.
>>>
>>> Yes, I can create bug reports. I used my first mail and created one [2].
>>
>> Thanks! We have had some brief discussions with Stuart (CC:ed), who 
>> created the AArch64 port, about this problem. Maybe he has had some 
>> time to think about it, and have some additional insights or ideas.
>>
>> Thanks,
>> StefanK
>>
>>>
>>> -- Christoph
>>>
>>> [1] 
>>> https://urldefense.com/v3/__https://www.kernel.org/doc/Documentation/arm64/memory.txt__;!!GqivPVa7Brio!LfyuMvR1e95FLlQs-8R-nh4-o89GIOgBFv1QU5_Y8OwPMh7ckvRjmrgp0vjyD0UaGUhW$ 
>>> [2] https://bugs.openjdk.java.net/browse/JDK-8252500
>>>
> 
> Hi,
>      That's right - I have been exploring options on this, and I had a 
> similar solution at one point to finding the address space size. From 
> speaking with people familiar with the arm64 linux kernel, there is no 
> good way to query the available address space except for probining it 
> and testing what is there. Thinking we could do with a general-purpose 
> routine, I experimented with a routine that forks the process and probes 
> the address space non-destructively. MAP_FIXED implicitly destroys any 
> existing mappings. Of course, ZGC mmaps memory at fixed addresses 
> anyhow, so the concern about embedded the JVM in your program and 
> destroying existing mappings turned out to be moot, as we'd be doing 
> that anyway.

Maybe I misunderstand this point, but we use fixed addresses when we 
probe the address space, but we don't use MAP_FIXED:

static bool map(uintptr_t start, size_t size) {
   const void* const res = mmap((void*)start, size, PROT_NONE, 
MAP_ANONYMOUS|MAP_PRIVATE|MAP_NORESERVE, -1, 0)

bool ZVirtualMemoryManager::reserve_contiguous_platform(uintptr_t start, 
size_t size) {
   // Reserve address views
   const uintptr_t marked0 = ZAddress::marked0(start);
   const uintptr_t marked1 = ZAddress::marked1(start);
   const uintptr_t remapped = ZAddress::remapped(start);

   if (!map(marked0, size)) {

StefanK

> 
> CCing Monica as the Windows platform might have similar issues.
> 
> 
> Thanks,
>      Stuart
> 
> 
> BR,
>      Stuart