RFR: JDK-8305100 [REDO] Clean up JavadocTokenizer

[Jim Laskey]

On Fri, 31 Mar 2023 11:10:42 GMT, Jim Laskey <jlaskey at openjdk.org> wrote:

> Previous work had cleaned up the JavacTokenizer, but left JavadocTokenizer alone, mostly due to its perplexing state. This is follow up work to simplify the JavadocTokenizer.
> 
> Redo changes:
> 
> 1. Needed to check for end of data after @deprecated
> 2. Position of newline in offset map was incorrect (out by one).
> 3. Skip whitespace after first line (put offsets out by one).

tier1,tier2,tier3 mach5 run https://mach5.us.oracle.com/mdash/jobs/jlaskey-open-20230330-1940-43993922

Original PR https://github.com/openjdk/jdk/pull/12953

-------------

PR Comment: https://git.openjdk.org/jdk/pull/13265#issuecomment-1491758451
PR Comment: https://git.openjdk.org/jdk/pull/13265#issuecomment-1491759614