Java Platform Module System

[Stephan Herrmann]

On 03.05.2017 20:55, Remi Forax wrote:
 > It's context-free because a context free grammar defined its input in term of
 > terminals and the theory do not say how to map a token to a terminal.
 >
 > Jay is right that it requires to use either some specific parser generator
 > like Tatoo [1] the one i've written 10 years ago (because i wanted the tool to
 > help me to extend a grammar easily) or to modify an existing parser generator so
 > the parser can send the production state to the lexer which will enable/disable
 > the automata that recognize the associated keywords .

Just feeding parser state into the Lexer doesn't cut it for Java 9,
because the classification keyword / identifier cannot be made at
the time when the stream passes the Lexer.
Let me remind you of this example:
        module foo { exports transitive
How should the poor lexer recognize in this situation that transitive
is an identifier (sic) (if you complete the text accordingly)?
Aside from specific heuristics
(which are not available to any parser generator),
we only know about this classification after the parser has matched
an entire declaration.
I'm not even sure that theory has a name for this kind of grammar.
Maybe we should speak of a constraint solver rather than a parser.

Stephan

>
> Rémi
>
> [1] http://dl.acm.org/citation.cfm?id=1168057
>
> ----- Mail original -----
>> De: "Alex Buckley" <alex.buckley at oracle.com>
>> À: "Jayaprakash Arthanareeswaran" <jarthana at in.ibm.com>, "Dan Smith" <daniel.smith at oracle.com>, "Brian Goetz"
>> <Brian.Goetz at oracle.com>
>> Cc: jigsaw-dev at openjdk.java.net
>> Envoyé: Mercredi 3 Mai 2017 19:46:54
>> Objet: Re: Java Platform Module System
>
>> On 5/2/2017 3:39 PM, Alex Buckley wrote:
>>> On 5/2/2017 7:07 AM, Jayaprakash Arthanareeswaran wrote:
>>>> Chapter 2 in [1] describes context-free grammars. The addition to "3.9
>>>> Keywords" defines "restricted keywords", which prevent the grammar for
>>>> ModuleDeclaration from being context-free. This prevents compilers from
>>>> using common parser generators, since those typically only support
>>>> context-free grammars. The lexical/syntactic grammar split defined in
>>>> chapter 2 is not of much use for actual implementations of
>>>> module-info.java parsers.
>>>> The spec at least needs to point out that the given grammar for
>>>> ModuleDeclaration is not actually context-free.
>>>
>>> The syntactic grammar in JLS8 was not context-free either; the opening
>>> line of Chapter 2 has been false for years. For JLS9, I will remove the
>>> claim that the lexical and syntactic grammars are context-free, and
>>> perhaps a future JLS can discuss the difficulties in parsing the
>>
>> Jan Lahoda pointed out privately that the syntactic grammar in JLS8 and
>> JLS9 is in fact context-free -- it's just not LL(1). Not being LL(1) is
>> what I should have said the grammar hasn't been for a long time.
>>
>> Alex