RFR: 8254073: Tokenizer improvements (revised) [v2]

[Jim Laskey]

On Tue, 6 Oct 2020 14:58:18 GMT, Jim Laskey <jlaskey at openjdk.org> wrote:

>> src/jdk.compiler/share/classes/com/sun/tools/javac/parser/UnicodeReader.java line 245:
>> 
>>> 243:
>>> 244:         for (int i = 0; i < 4; i++) {
>>> 245:             int digit = Character.digit(buffer[index], 16);
>> 
>> This looks suspicious - what if index ends up being bigger than (or equal to)  `buffer.length` ?
>> Maybe we need a test for incomplete unicode sequences at the end of the tokenizer input - e.g. `\u123`
>
> You are correct. Will revise.

Added change to

    int digit = index < length ? Character.digit(buffer[index], 16) : -1;

Also added new test.

-------------

PR: https://git.openjdk.java.net/jdk/pull/525