Nashorn comment nodes in the syntax tree

[Emilian Bold]

Here's another more nasty test. Note how comments before / after the
AccessNode property are part of the text range and it becomes impossible to
know where the property is located (short of parsing myself that).

Java 1.8.0_66

Script text:
--begin--
hello  .
//some comment
world /* some other comment */ = 42;
--end--

AccessNode text:
--begin--
hello  .
//some comment
world /* some other comment */
--end-

AccessNode base text:
--begin--
hello
--end-

AccessNode property (approximate) text:
--begin--
  .
//some comment
world /* some other comment */
--end-


// @AlwaysFails
    public void testIdentifierComment(){
        System.out.println("Java " + System.getProperty("java.version"));
        System.out.println();


        String text = "hello  .  \n" +
                "//some comment\n" +
                "world /* some other comment */ = 42;";
        Source source = Source.sourceFor("test.js", text); //NOI18N
        Options options = new Options("nashorn"); // NOI18N
        options.process(new String[]{
                "--parse-only=true", // NOI18N
                "--empty-statements=true", // NOI18N
                "--debug-lines=false"}); // NOI18N

        ScriptEnvironment env = new ScriptEnvironment(options, new
PrintWriter(System.out), new PrintWriter(System.err));

        ErrorManager errorManager = new ErrorManager();

        jdk.nashorn.internal.parser.Parser parser = new
jdk.nashorn.internal.parser.Parser(env, source, errorManager);
        FunctionNode root = parser.parse();

        System.out.println("Script text:\n--begin--\n" +
text+"\n--end--\n");

        root.accept(new NodeVisitor<LexicalContext>(new LexicalContext()) {
            @Override
            public boolean enterAccessNode(AccessNode accessNode) {
                String accessNodeText =
text.substring(accessNode.getStart(), accessNode.getFinish());

                String accessNodeBaseText =
text.substring(accessNode.getBase().getStart(),
accessNode.getBase().getFinish());
                String propertyText =
text.substring(accessNode.getBase().getFinish(), accessNode.getFinish());

                System.out.println("AccessNode text:\n--begin--\n" +
accessNodeText+"\n--end-\n");

                System.out.println("AccessNode base text:\n--begin--\n" +
accessNodeBaseText+"\n--end-\n");

                System.out.println("AccessNode property (approximate)
text:\n--begin--\n" + propertyText+"\n--end-\n");

                assertFalse("The accessnode should not swallow the
comment", accessNodeText.contains("some comment"));

                assertFalse("The property should not swallow the comment",
propertyText.contains("some comment"));

                return super.enterAccessNode(accessNode);
            }
        });
    }



--emi

On Fri, Jul 15, 2016 at 6:19 AM, Sundararajan Athijegannathan <
sundararajan.athijegannathan at oracle.com> wrote:

> Hi,
>
> Sorry for the delayed response. I'm yet to check your test. I'll file a
> bug once I reproduce/check this at my end and let you know the bug id.
>
> Thanks,
>
> -Sundar
>
> On 7/12/2016 2:51 PM, Emilian Bold wrote:
>
> Yes, it must be a bug: comments are certainly swallowed in the nodes on my
> JRE. I see this for FunctionNode but also for IdentNode and perhaps more.
>
> It makes getting actual offsets a pain and I would strip-out comments from
> the text before sending it to the parser, but I need those too.
>
> $ uname -a
> Darwin mac.local 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36
> PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
>
> $ java -version
> java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
>
> Java 1.8.0_66
>
> Script text:
> --begin--
> function hello(){
>  return 'world';
> }
>
> //some comment
> hello();
> --end--
>
> [<unknown>] function {U%}hello()
> Function text:
> --begin--
> {
>  return 'world';
> }
>
> //some comment
> --end-
>
>
> junit.framework.AssertionFailedError: The function should not swallow the
> comment
>
>
> Test:
> package com; import jdk.nashorn.internal.ir.FunctionNode; import
> jdk.nashorn.internal.ir.LexicalContext; import
> jdk.nashorn.internal.ir.visitor.NodeVisitor; import
> jdk.nashorn.internal.runtime.ErrorManager; import
> jdk.nashorn.internal.runtime.ScriptEnvironment; import
> jdk.nashorn.internal.runtime.Source; import
> jdk.nashorn.internal.runtime.options.Options; import
> junit.framework.TestCase; import java.io.PrintWriter; public class
> NashornCommentBugTest extends TestCase { public void testComment(){
> System.out.println("Java " + System.getProperty("java.version"));
> System.out.println(); String text = "function hello(){ \n" + " return
> 'world';\n" + "}\n\n" + "" + "//some comment\n" + "hello();"; Source source
> = Source.sourceFor("test.js", text); //NOI18N Options options = new
> Options("nashorn"); // NOI18N options.process(new String[]{
> "--parse-only=true", // NOI18N "--empty-statements=true", // NOI18N
> "--debug-lines=false"}); // NOI18N ScriptEnvironment env = new
> ScriptEnvironment(options, new PrintWriter(System.out), new
> PrintWriter(System.err)); ErrorManager errorManager = new ErrorManager();
> jdk.nashorn.internal.parser.Parser parser = new
> jdk.nashorn.internal.parser.Parser(env, source, errorManager); FunctionNode
> root = parser.parse(); System.out.println("Script text:\n--begin--\n" +
> text+"\n--end--\n"); root.accept(new NodeVisitor<LexicalContext>(new
> LexicalContext()) { @Override public boolean enterFunctionNode(FunctionNode
> functionNode) { if
> (!functionNode.getKind().equals(FunctionNode.Kind.SCRIPT)) {
> System.out.println(functionNode.toString()); String functionText =
> text.substring(functionNode.getStart(), functionNode.getFinish());
> System.out.println("Function text:\n--begin--\n" +
> functionText+"\n--end-\n"); assertFalse("The function should not swallow
> the comment", functionText.contains("some comment")); } return
> super.enterFunctionNode(functionNode); } }); } }
>
>
>
>
>
> --emi
>
> On Mon, Jun 13, 2016 at 6:31 AM, Sundararajan Athijegannathan <
> sundararajan.athijegannathan at oracle.com> wrote:
>
>> start and finish should not include comments around. If it does, it is a
>> bug. Please do file a bug and send us test case here and we'll file a bug
>>
>> Thanks,
>>
>> -Sundar
>>
>> On 6/10/2016 9:05 PM, Emilian Bold wrote:
>>
>> BTW, jdk.nashorn.internal.ir.IdentNode (and other classes) seem to have
>> the habit of "absorbing" the nearby comments and have incorrect
>> start/finish offsets.
>>
>> Is this intentional or a bug?
>>
>> I haven't tested yet but I wonder if it propagates to the TreeAPI (ie.
>> does IdentifierTree also swallow adjacent comments?)
>>
>>
>>
>> --emi
>>
>> On Wed, Apr 27, 2016 at 5:18 PM, Sundararajan Athijegannathan <
>> sundararajan.athijegannathan at oracle.com> wrote:
>>
>>> Hi,
>>>
>>> I filed an enhancement request:
>>> https://bugs.openjdk.java.net/browse/JDK-8155242
>>>
>>> Thanks,
>>>
>>> -Sundar
>>>
>>>
>>> On 4/27/2016 12:40 AM, Emilian Bold wrote:
>>> > Hello,
>>> >
>>> > NetBeans used a modified Rhino that had useful features for an IDE like
>>> > comment nodes and parser-level sanitization.
>>> >
>>> > As far as I know there is no way to get the comments in the Nashorn
>>> Tree
>>> > API (created as part of JEP 236) or in the jdk.nashorn.internal.ir
>>> Node
>>> > subclasses.
>>> >
>>> > Obviously having access to comment contents and comment offsets is very
>>> > useful. An IDE, for example, would provide code folding for the
>>> multiline
>>> > comment.
>>> >
>>> > Am I missing some proper way to get this information?
>>> >
>>> > I believe an extra option for Parser.create() would be nice (plus a
>>> > CommentTree and a visitComment method).
>>> >
>>> > --emi
>>>
>>>
>>
>>
>
>