RFR: JDK-8298405: Support Markdown in Documentation Comments [v18]

[Jan Lahoda]

On Tue, 6 Feb 2024 01:36:58 GMT, Jonathan Gibbons <jjg at openjdk.org> wrote:

>> Please review a patch to add support for Markdown syntax in documentation comments, as described in the associated JEP.
>> 
>> Notable features:
>> 
>> * support for `///` documentation comments in `JavaTokenizer`
>> * new module `jdk.internal.md` -- a private copy of the `commonmark-java` library
>> * updates to `DocCommentParser` to treat `///` comments as Markdown
>> * updates to the standard doclet to render Markdown comments in HTML
>
> Jonathan Gibbons has updated the pull request incrementally with one additional commit since the last revision:
> 
>   First pass at remove DocCommentTransformer from the public API.
>   
>   It is still declared internally, and installed by default, using the service-provider mechanism.
>   If the standard impl is not available, an identity transformer is used.

src/jdk.internal.md/share/classes/jdk/internal/markdown/MarkdownTransformer.java line 1:

> 1: /*

This transformer seems to break positions of the `RawTextTree`.
For javadoc like:

                        /// Markdown test
                        ///
                        /// @author testAuthor

without this transformer, taking the start and end positions of the `RawTextTree` and taking the text between these positions will lead to: `[Markdown test, testAuthor]`. With this transfomer, it leads to `[Markdown test, Markdown t]`, which is clearly suspicious.

Testcase:

/*
 * Copyright (c) 2024, Oracle and/or its affiliates. All rights reserved.
 * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 *
 * This code is free software; you can redistribute it and/or modify it
 * under the terms of the GNU General Public License version 2 only, as
 * published by the Free Software Foundation.
 *
 * This code is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
 * version 2 for more details (a copy is included in the LICENSE file that
 * accompanied this code).
 *
 * You should have received a copy of the GNU General Public License version
 * 2 along with this work; if not, write to the Free Software Foundation,
 * Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.
 *
 * Please contact Oracle, 500 Oracle Parkway, Redwood Shores, CA 94065 USA
 * or visit www.oracle.com if you need additional information or have any
 * questions.
 */

/*
 * @test
 * @bug 9999999
 * @summary XXX
 * @run main/othervm --limit-modules jdk.compiler MarkdownTransformerPositionTest
 * @run main MarkdownTransformerPositionTest
 */

import com.sun.source.doctree.DocCommentTree;
import com.sun.source.doctree.RawTextTree;
import com.sun.source.tree.*;
import com.sun.source.util.*;

import java.net.URI;
import java.util.*;
import javax.lang.model.element.Element;
import javax.tools.*;


public class MarkdownTransformerPositionTest {

    public static void main(String... args) throws Exception {
        String source = """
                        /// Markdown test
                        ///
                        /// @author testAuthor
                        public class Test {
                        }
                        """;
        JavaCompiler comp = ToolProvider.getSystemJavaCompiler();
        JavacTask task = (JavacTask)comp.getTask(null, null, null, null, null, Arrays.asList(new JavaSource(source)));
        CompilationUnitTree cu = task.parse().iterator().next();
        task.analyze();
        DocTrees trees = DocTrees.instance(task);
        List<String> rawSpans = new ArrayList<>();
        TreePath clazzTP = new TreePath(new TreePath(cu), cu.getTypeDecls().get(0));
        Element clazz = trees.getElement(clazzTP);
        DocCommentTree docComment = trees.getDocCommentTree(clazz);

        new DocTreeScanner<Void, Void>() {
            @Override
            public Void visitRawText(RawTextTree node, Void p) {
                int start = (int) trees.getSourcePositions().getStartPosition(cu, docComment, node);
                int end = (int) trees.getSourcePositions().getEndPosition(cu, docComment, node);
                rawSpans.add(source.substring(start, end));
                return super.visitRawText(node, p);
            }
        }.scan(docComment, null);

        List<String> expectedRawSpans = List.of("Markdown test", "testAuthor");

        if (!expectedRawSpans.equals(rawSpans)) {
            throw new AssertionError("Incorrect raw text spans, should be: " +
                                     expectedRawSpans + ", but is: " + rawSpans);
        }

        System.err.println("Test result: success, boot modules: " + ModuleLayer.boot().modules());
    }

    static class JavaSource extends SimpleJavaFileObject {

        private final String source;

        public JavaSource(String source) {
            super(URI.create("myfo:/Test.java"), JavaFileObject.Kind.SOURCE);
            this.source = source;
        }

        @Override
        public CharSequence getCharContent(boolean ignoreEncodingErrors) {
            return source;
        }
    }
}

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/16388#discussion_r1479306878