RFR: 8283225: [AIX] ClassLoader.c produces incorrect OutOfMemory Exception when length is 0

[Tyler Steele]

On Wed, 16 Mar 2022 14:38:55 GMT, Thomas Stuefe <stuefe at openjdk.org> wrote:

>> As described in the linked issue, NullClassBytesTest fails due an OutOfMemoryError produced on AIX when the test calls defineClass with a byte array of size of 0. The native implementation of defineClass then calls  malloc with a size of 0. On AIX malloc(0) returns NULL, while on other platforms it return a valid address. When NULL is produced by malloc for this reason, ClassLoader.c incorrectly interprets this as a failure due to a lack of memory.
>> 
>> ~~This PR modifies ClassLoader.c to produce an OutOfMemoryError only when `errno == ENOMEM` and to produce a ClassFormatError with the message "ClassLoader internal allocation failure" in all other cases (in which malloc returns NULL).~~ [edit: The above no longer describes the PR's proposed fix. See discussion below]
>> 
>> In addition, I performed some minor tidy-up work in ClassLoader.c by changing instances of `return 0` to `return NULL`, and `if (some_ptr == 0)` to `if (some_ptr == NULL)`. This was done to improve the clarity of the code in ClassLoader.c, but didn't feel worthy of opening a separate issue.
>> 
>> ### Alternatives
>> 
>> It would be possible to address this failure by modifying the test to accept the OutOfMemoryError on AIX. I thought it was a better solution to modify ClassLoader.c to produce an OutOfMemoryError only when the system is actually out of memory.
>> 
>> ### Testing
>> 
>> This change has been tested on AIX and Linux/x86.
>
> Btw, which malloc call was the problematic exactly? Cannot be the one in getUTF, since that one already adds len + 1 and never gets called with a zero length anyway.

Thanks @tstuefe! Your suggestion lead to a better change, so I modified the PR. 

- ClassLoader.c no longer has any reason to throw a ClassFormatError, so that logic is removed. 
- The test no longer needs to recognize a new error message, so that is changed back as well.
- I also alphabetized the header files, because that is the way I am :-)

Note: I couldn't find an implementation of MAX2 in a C-friendly 'header.h' file, so I just used the ternary operator in the two places I needed it.

-------------

PR: https://git.openjdk.java.net/jdk/pull/7829