RFR: JDK-8305100 [REDO] Clean up JavadocTokenizer

[Jonathan Gibbons]

On Fri, 31 Mar 2023 11:10:42 GMT, Jim Laskey <jlaskey at openjdk.org> wrote:

> Previous work had cleaned up the JavacTokenizer, but left JavadocTokenizer alone, mostly due to its perplexing state. This is follow up work to simplify the JavadocTokenizer.
> 
> Redo changes:
> 
> 1. Needed to check for end of data after @deprecated
> 2. Position of newline in offset map was incorrect (out by one).
> 3. Skip whitespace after first line (put offsets out by one).

src/jdk.compiler/share/classes/com/sun/tools/javac/parser/JavaTokenizer.java line 1357:

> 1355:                                     line.get() == EOI)) {
> 1356:                         deprecatedFlag = true;
> 1357:                     }

As mentioned recently elsewhere, this is generally insufficient, albeit no worse than before.
Generally, we should move away from this code, and eliminate `deprecated Flag` entirely. Deprecation is (or should be) indicated by the `@Deprecated` annotation, with associated documentation given by a plain/regular `@deprecated` tag, as read by `DocCommentParser` or equivalent.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/13265#discussion_r1154867472