RFR: JDK-8303912 - Clean up JavadocTokenizer [v3]

[Jim Laskey]

> Previous work had cleaned up the JavacTokenizer, but left JavadocTokenizer alone, mostly due to its perplexing state. This is follow up work to simplify the JavadocTokenizer.

Jim Laskey has updated the pull request incrementally with one additional commit since the last revision:

  Recommended changes

-------------

Changes:
  - all: https://git.openjdk.org/jdk/pull/12953/files
  - new: https://git.openjdk.org/jdk/pull/12953/files/e3e31d09..6e3be1eb

Webrevs:
 - full: https://webrevs.openjdk.org/?repo=jdk&pr=12953&range=02
 - incr: https://webrevs.openjdk.org/?repo=jdk&pr=12953&range=01-02

  Stats: 11 lines in 3 files changed: 0 ins; 6 del; 5 mod
  Patch: https://git.openjdk.org/jdk/pull/12953.diff
  Fetch: git fetch https://git.openjdk.org/jdk.git pull/12953/head:pull/12953

PR: https://git.openjdk.org/jdk/pull/12953