Increase memory limits for IcedTea

[Florian Weimer]

* Andrew Haley:

> I'm sorry, but you're just being a bit too obscure for me.  What do
> you mean by "it tends to break"?  

vm.overcommit_memory=2 makes sure that the out-of-memory killer in the
kernel does not need to be run, by being extremely conservative about
handing out memory to applications.  Everything which is writeable needs
to be backed by its own chunk of RAM or swap space (unless it's a shared
mapping which is not copy-on-write, of course).  As a result, you run
into out-of-memory errors sooner, but they are much more predictable and
can usually be recovered from.

> If you could give an example to test, with a description of what you
> think should happen and what does happen, then I could talk to the
> kernel team.

The following instructions are for a machine that has got 4 GB of RAM
and 4 GB swap space.  You need to adjust the numbers accordingly if your
setup is different.

You need a simple Java application that runs indefinitely, for instance:

public class Wait {
    static public void main(String[] args) throws Exception {
        System.in.read();
    }
}

Now run

/usr/lib/jvm/java-6-sun/bin/java -XX:MaxPermSize=3g Wait

in three terminals.  The first two invocations will just wait for
input.  The third one will report this error:

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Because of vm.overcommit_memory=2, it does not matter how much of that 3
GB the application uses.  You can only create very few VMs if the
default heap size is a sizable chunk of your available RAM + swap.

How can this be fixed?  Hotspot uses this mmap call to allocate memory:

11408 mmap(NULL, 4255186944, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2aaaaea31000

MAP_NORESERVE apparently doesn't do what it's name suggest (reserve some
address space, but do not commit to kernel to actually provide backing
store for it).  In my experiments, you need to specify PROT_NONE
initially, and use mprotect to allocate backing store as soon as you
need it (when the heap is grown).  It's still advantageous to call mmap
once, with a large chunk, because that guarantees you get a contiguous
heap.

In some cases, it might make sense to ask the kernel to allocate backing
store for the heap (like Hotspot currently does), but I doubt that this
is true for the majority of workloads.