RFR: 8305770: os::Linux::available_memory() should refer MemAvailable in /proc/meminfo

[Yasumasa Suenaga]

On Tue, 25 Apr 2023 08:27:44 GMT, Roberto Castañeda Lozano <rcastanedalo at openjdk.org> wrote:

>> Hi hotspot-compiler folks,
>> 
>> I'd like to change `os::Linux::available_memory()` to refer `MemAvailable` in /proc/meminfo on Linux. One of user of this function is JIT compiler. It is used for determine number of compiler threads. After this change, more compiler threads would be started because `MemAvailable` includes not only free memory but also some caches - it means `MemAvailable` is bigger than `MemFree`.
>> 
>> I think this change is not a problem because number of compiler threads is limited by `CICompilerCount`. Do you have any concerns in compiler perspective?
>
> Hi @YaSuenag, just to confirm that this change would not lead to excessive creation/deletion of compiler threads (which can have a significant cost in terms of memory usage, see e.g. [JDK-8302264](https://bugs.openjdk.org/browse/JDK-8302264)), it would be useful to see some measurements about number of created compiler threads over the execution of some applications in a environment configured so that `available_memory / (200*M)` becomes the limiting factor in https://github.com/openjdk/jdk/blob/f968da97a5a5c68c28ad29d13fdfbe3a4adf5ef7/src/hotspot/share/compiler/compileBroker.cpp#L1024-L1027, before and after the change. This could be easily measured e.g. using `-XX:+TraceCompilerThreads`. Do you have (or could produce) such measurements?

@robcasloz Thanks for your reply! The results of measurement with `-XX:+TraceCompilerThreads` is here. They look like no significant difference. Do you have any concerns? and we go ahead this change?

# before


$ head -n 3 /proc/meminfo; ./build/linux-x86_64-server-fastdebug/images/jdk/bin/java -XX:+TraceCompilerThreads --version
MemTotal:        8117076 kB
MemFree:          123228 kB
MemAvailable:    7457284 kB
    184 Added initial compiler thread C2 CompilerThread0
    184 Added initial compiler thread C1 CompilerThread0
openjdk 21-internal 2023-09-19
OpenJDK Runtime Environment (fastdebug build 21-internal-adhoc.ysuenaga.jdk)
OpenJDK 64-Bit Server VM (fastdebug build 21-internal-adhoc.ysuenaga.jdk, mixed mode, sharing)


# after


$ head -n 3 /proc/meminfo; ./build/linux-x86_64-server-fastdebug/images/jdk/bin/java -XX:+TraceCompilerThreads --version
MemTotal:        8117076 kB
MemFree:          118276 kB
MemAvailable:    7625760 kB
    185 Added initial compiler thread C2 CompilerThread0
    185 Added initial compiler thread C1 CompilerThread0
openjdk 21-internal 2023-09-19
OpenJDK Runtime Environment (fastdebug build 21-internal-adhoc.ysuenaga.jdk)
OpenJDK 64-Bit Server VM (fastdebug build 21-internal-adhoc.ysuenaga.jdk, mixed mode, sharing)


They were measured on Fedora 38 on Hyper-V which assigned 4 vCPUs. I consumed memory with following program. It run on background in each measurement.


  const char *fname = "test.dat";
  const unsigned long sz = 8000UL * 1024UL * 1024UL;
  char *mem;
  int fd;

  fd = open(fname, O_CREAT | O_CLOEXEC | O_RDWR, S_IRUSR | S_IWUSR);
  lseek(fd, sz, SEEK_SET);
  write(fd, &sz, 1);

  mem = (char *)mmap(NULL, sz, PROT_WRITE, MAP_SHARED, fd, 0);
  memset(mem, 1, sz);

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13398#issuecomment-1522562257