RFR: 8256155: os::Linux Populate all large_page_sizes, select smallest page size in reserve_memory_special_huge_tlbfs* [v23]

[Stefan Johansson]

On Thu, 18 Mar 2021 12:47:40 GMT, Stefan Johansson <sjohanss at openjdk.org> wrote:

>>> This would also give us to to think about a question I haven't made up my mind around. What will `LargePageSizeInBytes` mean after this change? Should the JVM use 1g pages (on a system where 2m i the default) even if `LargePageSizeInBytes` is not set?
>> 
>> I see two valid scenarios:
>> a) either use huge pages as best as possible; remove fine-grained control from user hands. So, if we have 1G pages, prefer those over 2M over 4K. Then, we could completely remove LargePageSizeInBytes. There is no need for this switch anymore.
>> b) or keep the current behavior. In that case, UseLargePageSize means "use at most default huge page size" even if larger pages are available; and LargePageSizeInBytes can be used to override the large page size to use.
>>  
>> (a): Its elegant, and efficiently uses system resources when available. But its an incompatible change, and the VM being grabby means we could end up using pages meant for a different process.
>> (b): downward compatible. In a sense. With Marcus change, we break downward compatibility already: where "UseLargePageSizeInBytes" before meant "use that or nothing", it now means "use that or whatever smaller page sizes you find". 
>> 
>> I prefer (a), honestly.
>> 
>> ..Thomas
>
>> > This would also give us to to think about a question I haven't made up my mind around. What will `LargePageSizeInBytes` mean after this change? Should the JVM use 1g pages (on a system where 2m i the default) even if `LargePageSizeInBytes` is not set?
>> 
>> I see two valid scenarios:
> 
> Me too.
> 
>> a) either use huge pages as best as possible; remove fine-grained control from user hands. So, if we have 1G pages, prefer those over 2M over 4K. Then, we could completely remove LargePageSizeInBytes. There is no need for this switch anymore.
> 
> I agree, preferably we can make it so that the upper layers can use something like `page_size_for_region*` and request a certain page size, but fall back to smaller ones.
> 
>> b) or keep the current behavior. In that case, UseLargePageSize means "use at most default huge page size" even if larger pages are available; and LargePageSizeInBytes can be used to override the large page size to use.
>> 
> 
> So it becomes more like a maximium value right? Or at least this is how I've thought about this second scenario. On a system with both 2M (the default size) and 1G pages available you would have to set `LargePageSizeInBytes=1g` to use the 1G pages, but the 2M could still be used for smaller mappings.
> 
>> (a): Its elegant, and efficiently uses system resources when available. But its an incompatible change, and the VM being grabby means we could end up using pages meant for a different process.
>> (b): downward compatible. In a sense. With Marcus change, we break downward compatibility already: where "UseLargePageSizeInBytes" before meant "use that or nothing", it now means "use that or whatever smaller page sizes you find".
>> 
>> I prefer (a), honestly.
>> 
> I would also prefer (a).

> @kstefanj @tstuefe
> 
>     1. Would (a) [removing fine grained control of large page [Huge Page] sizes with LargePageSizeInBytes and relying on automatic selection of sizes from available] require JEP process?
> 

Not a JEP, but likely a CSR. If we change the meaning of `LargePageSizeInBytes` (or deprecate it) a CSR will be needed.

>     2. Can we put LargePageSizeInBytes flag in DIAGNOSTIC or EXPERIMENTAL and set it otherwise to the largest available page size? This way we have the default (a) and if users want to change page size they can use diagnostic or experimental and use (b). Also this would lower the amount of change needed. I'm a bit concerned with removing LargePageSizeInBytes altogether. I don't know if some use cases would break or need more fine tuned control of page sizes.
> 

Changing the flag to diagnostic or experimental would also require a CSR and I don't see any real benefit of doing that. 

> 
> It appears the comment in the flag supports a solution like [2] above, but we could change change to `max page size or 0 to let VM choose`: "Large page size (0 to let VM choose the page size)"
> 
> Current LargePageSizeInBytes flag code:
> 
> ```
>   product(size_t, LargePageSizeInBytes, 0,                                  \
>           "Large page size (0 to let VM choose the page size)")             \
>           range(0, max_uintx)                                               \
> ```
> 

Reading this I wonder if the easiest way forward would just be to keep it this way. If the flag is set, only that page size is used, if not all configured page sizes can be used. So if someone sets `LargePagesSizeInBytes` only that size will end up in the available page sizes map. I might need to think a bit more about this, but if there is no problems with that approach it would be quite nice.

> I've incorporated #3073 into this change, as a good base to build on, this also seemed to be Stefan's optimal path for this change.

Hopefully I can push #3073 soon, and then you can just merge master to get this code correctly.

-------------

PR: https://git.openjdk.java.net/jdk/pull/1153