[9] RFR(S): 8130309: need to bailout cleanly if CompiledStaticCall::emit_to_interp_stub fails when codecache is out of space

[Tobias Hartmann]

Hi,

please review the following patch.

https://bugs.openjdk.java.net/browse/JDK-8130309
http://cr.openjdk.java.net/~thartmann/8130309/webrev.00/

Problem:
While C2 is emitting code, an assert is hit because the emitted code size does not correspond to the size of the instruction. The problem is that the code cache is full and therefore the creation of a to-interpreter stub failed. Instead of bailing out we continue and hit the assert.

More precisely, we emit code for a CallStaticJavaDirectNode on aarch64 and emit two stubs (see 'aarch64_enc_java_static_call' in aarch64.ad):
- MacroAssembler::emit_trampoline_stub() -> requires 64 bytes
- CompiledStaticCall::emit_to_interp_stub() -> requires 56 bytes
However, we only have 112 bytes of free space in the stub section of the code buffer and need to expand it. Since the code cache is full, the expansion fails and the corresponding code blob is freed. In CodeBuffer::free_blob() -> CodeBuffer::set_blob() we set addresses to 'badAddress' and therefore hit the assert.

Solution:
Even if we have enough space in the instruction section, we should check for a failed expansion of the stub section and bail out immediately. I added the corresponding check and also removed an unnecessary call to 'start_a_stub' after we already failed. 

Testing:
- Failing test
- JPRT

Thanks,
Tobias