[9] RFR (XS) 8071534: assert(!failing()) failed: Must not have pending failure. Reason is: out of memory

[Vladimir Kozlov]

https://bugs.openjdk.java.net/browse/JDK-8071534

http://cr.openjdk.java.net/~kvn/8071534/webrev

Add missing C->failing() check after Connection graph construction.
Note, there are C->failing() checks before EA starts Connection graph 
construction.

EA uses bytecode analyzer BCEscapeAnalyzer to look into called bit not 
inlined methods. BCEscapeAnalyzer try to resolve symbols referenced in 
bytecode and VM may throw an exception during resolution. For example, 
if it needs to allocate in metaspace which does not have space.

CI cleans exceptions and converts them to compilation failure:

void ciEnv::record_out_of_memory_failure() {
   // If memory is low, we stop compiling methods.
   record_method_not_compilable("out of memory");
}

It is called when there is a pending exception. For example:

// Make a ciSymbol from a C string (implementation).
ciSymbol* ciSymbol::make_impl(const char* s) {
   EXCEPTION_CONTEXT;
   TempNewSymbol sym = SymbolTable::new_symbol(s, THREAD);
   if (HAS_PENDING_EXCEPTION) {
     CLEAR_PENDING_EXCEPTION;
     CURRENT_THREAD_ENV->record_out_of_memory_failure();
     return ciEnv::_unloaded_cisymbol;
   }
   return CURRENT_THREAD_ENV->get_symbol(sym);
}

That is the only explanation I can think of because after 2 days I was 
not able to reproduce the problem. And it unknown which particular CI 
code produced the exception.

The test stresses VM with limited metaspace and I see one metaspace OOM 
exception in hs_err file (in different thread):

Event: 253.978 Thread 0x28069a60 Exception <a 
'java/lang/OutOfMemoryError'> (0x05d93ad8) thrown at 
[hotspot\src\share\vm\memory\metaspace.cpp, line 3597]

Thanks,
Vladimir