RFR: JDK-8303912 - Clean up JavadocTokenizer [v3]

[Jonathan Gibbons]

On Mon, 27 Mar 2023 17:17:47 GMT, Jim Laskey <jlaskey at openjdk.org> wrote:

>> Previous work had cleaned up the JavacTokenizer, but left JavadocTokenizer alone, mostly due to its perplexing state. This is follow up work to simplify the JavadocTokenizer.
>
> Jim Laskey has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Recommended changes

Marked as reviewed by jjg (Reviewer).

-------------

PR Review: https://git.openjdk.org/jdk/pull/12953#pullrequestreview-1359621594