Combined grammar changes for binary literals and underscores in literals

[Derek Foster]

Hi, Bruce.

It's good to hear from you again!

Regarding your suggested changes to the grammar in the proposal, I admit to being a bit skeptical that they really add enough value to compensate for the decrease in readability. I tend to agree with Joe's assessment:

>One advantage of the way I wrote up the grammar is that the other 
>appearances of Digits and HexDigits, such as in floating-point literals 
>in section 3.10.2, automatically inherit the underscore policy without 
>having to having leading digit guard productions in all places.  The 
>current implementation does allow underscores in floating-point literals 
>too, which is tested for in the regression tests.

Specifically, I think that although the grammar changes you propose may be "simpler" in terms of having fewer productions, it is also a little bit harder to understand, and easier to make mistakes with. Given that this is a specification rather than an implementation, I think that it is most important in this case to be clear and comprehensible, so that various people can implement it correctly and be confident that they have done so. A compiler vendor can always massage the grammar as necessary in order to lead to more efficient code if desired. I'm not dead-set against the changes you propose -- they just seem somewhat unnecessary to me, and I'm hesitant to make any unnecessary changes at this late stage, since it would be very easy to accidentally insert a mistake into the specification right before it's finalized.

Derek


-----Original Message-----
>From: Joe Darcy <joe.darcy at oracle.com>
>Sent: Apr 13, 2010 4:05 AM
>To: Bruce Chapman <brucechapman at paradise.net.nz>
>Cc: coin-dev <coin-dev at openjdk.java.net>
>Subject: Re: Combined grammar changes for binary literals and underscores	in literals
>
>Welcome back Bruce!
>
>Bruce Chapman wrote:
>> Joe, Derek et al,
>>
>> I am just writing up the combined integer literal omnibus proposal.
>>
>> A couple of questions about this grammar
>>
>>
>> 1)
>>
>> considering
>>
>> DecimalNumeral
>>    0
>>    NonZeroDigit Digits_opt
>>    NonZeroDigit Underscores Digits // New   (1)
>>
>> Underscores
>>    _
>>    Underscores _
>>
>> Digits:
>>    Digit
>>    Digit DigitsAndUnderscores_opt Digit (2)
>>
>>
>> I think we can drop rule (1) above, if rule (2) is changed to
>>
>> DigitsAndUnderscores_opt Digit
>>
>> since there is already a leading NonZeroDigit, so we can start a 
>> "Digits" with an underscore, so we don't need the special rule to allow 
>> for an underscore immediately after the NonZeroDigit. 
>>
>> Agreed?
>>
>> If so we can also make corresponding change for Octal rules then drop 
>> the Underscores production as these are the only places it is used.
>>
>>
>> 2)
>>
>> I am wondering whether it would enhance grokkability of we renamed the 
>> productions
>>
>> HexDigitsAndUnderscores (and other radix friends, and corresponding singular forms)
>>
>>
>> to be (eg)
>> HexMidDigits
>>
>> because it can't be used at either end of the HexDigits, but can be in 
>> the middle.
>>
>> - if we did this the hex rules would change from
>>
>> HexDigits:
>>    HexDigit
>>    HexDigit HexDigitsAndUnderscores_opt HexDigit
>>
>> HexDigitsAndUnderscores:
>>    HexDigitOrUnderscore
>>    HexDigitsAndUnderscores HexDigitOrUnderscore
>>
>> HexDigitOrUnderscore
>>    HexDigit
>>    _
>>
>>
>>
>> to
>>
>> HexDigits:
>>    HexDigit
>>    HexDigit HexMidDigits_opt HexDigit
>>
>> HexMidDigits:
>>    HexMidDigit
>>    HexMidDigits HexMidDigit  (but actually vice versa as previously 
>> discussed off-list with Alex Buckley)
>>
>> HexMidDigit
>>    HexDigit
>>    _
>>
>>
>>
>>
>> Comments?
>>
>>   
>
>One advantage of the way I wrote up the grammar is that the other 
>appearances of Digits and HexDigits, such as in floating-point literals 
>in section 3.10.2, automatically inherit the underscore policy without 
>having to having leading digit guard productions in all places.  The 
>current implementation does allow underscores in floating-point literals 
>too, which is tested for in the regression tests.
>
>-Joe
>
>
>> Bruce
>>
>>
>>
>>
>>
>>
>> Joe Darcy wrote:
>>   
>>> Hello.
>>>
>>> It has been pointed out to me off-list that the grammar as written below 
>>> would disallow two-digit numbers, which is not desired.  This problem is 
>>> corrected by changing the productions like
>>>
>>>     Digit DigitsAndUnderscores Digit
>>>
>>> to be
>>>
>>>     Digit DigitsAndUnderscores_opt Digit
>>>
>>> -Joe
>>>
>>> On 07/15/09 02:22 PM, Joseph D. Darcy wrote:
>>>   
>>>     
>>>> Hello.
>>>>
>>>> Getting around to some grammatical hacking, below is a grammar I think 
>>>> properly captures the desired changes for combining the binary 
>>>> literals and underscores in literals proposals.  This grammar does 
>>>> *not* include support for unsigned literals or autosizing literals.
>>>>
>>>> The basic approach is to redefine the FooDigits productions to be a 
>>>> non-empty sequence of Foo digits and underscores starting and ending 
>>>> with a Foo digit rather than just a non-empty sequence Foo digits.  
>>>> Each FooDigits nonterminal has two rules, one for a single digit and 
>>>> one for two or more digits.  The rules for two or more digits force a 
>>>> true digit, rather than an underscore, as the first and last character.
>>>>
>>>> Changing the FooDigits definitions allows underscores in both integer 
>>>> and floating-point literals.  In more detail:
>>>>
>>>> * Replace the productions Digits, HexDigits, OctalDigits, and  in 
>>>> JLSv3 3.10.1 as shown below
>>>> * Add productions for BinaryDigits
>>>> * Add rules for FooDigitsAndUnderscores and FooDigitOrUnderscore
>>>> * Add productions to DecimalNumeral and OctalNumeral:
>>>> * Add a rule for Underscores
>>>>
>>>> IntegerLiteral:
>>>>        DecimalIntegerLiteral
>>>>        HexIntegerLiteral
>>>>        OctalIntegerLiteral
>>>>        BinaryIntegerLiteral         // New
>>>>
>>>> BinaryIntegerLiteral:
>>>>        BinaryNumeral IntegerTypeSuffix_opt
>>>>
>>>> BinaryNumeral:
>>>>        0 b BinaryDigits
>>>>        0 B BinaryDigits
>>>>
>>>>
>>>> DecimalNumeral
>>>>   0
>>>>   NonZeroDigit Digits_opt
>>>>   NonZeroDigit Underscores Digits // New
>>>>
>>>> Underscores
>>>>    _
>>>>    Underscores _
>>>>
>>>> Digits:
>>>>    Digit
>>>>    Digit DigitsAndUnderscores Digit
>>>>
>>>> DigitsAndUnderscores:
>>>>    DigitOrUnderscore
>>>>    DigitsAndUnderscores DigitOrUnderscore
>>>>
>>>> DigitOrUnderscore
>>>>    Digit
>>>>    _
>>>>
>>>>
>>>> HexDigits:
>>>>    HexDigit
>>>>    HexDigit HexDigitsAndUnderscores HexDigit
>>>>
>>>> HexDigitsAndUnderscores:
>>>>    HexDigitOrUnderscore
>>>>    HexDigitsAndUnderscores HexDigitOrUnderscore
>>>>
>>>> HexDigitOrUnderscore
>>>>    HexDigit
>>>>    _
>>>>
>>>> OctalNumeral:
>>>>    0 OctalDigits
>>>>    0 Underscores OctalDigits // New
>>>>
>>>> OctalDigits:
>>>>    OctalDigit
>>>>    OctalDigit OctalDigitsAndUnderscores OctalDigit
>>>>
>>>> OctalDigitsAndUnderscores:
>>>>    OctalDigitOrUnderscore
>>>>    OctalDigitsAndUnderscores OctalDigitOrUnderscore
>>>>
>>>> OctalDigitOrUnderscore
>>>>    OctalDigit
>>>>    _
>>>>
>>>> BinaryDigits:
>>>>    BinaryDigit
>>>>    BinaryDigit BinaryDigitsAndUnderscores BinaryDigit
>>>>
>>>> BinaryDigitsAndUnderscores:
>>>>    BinaryDigitOrUnderscore
>>>>    BinaryDigitsAndUnderscores BinaryDigitOrUnderscore
>>>>
>>>> BinaryDigitOrUnderscore
>>>>    BinaryDigit
>>>>    _
>>>>
>>>> BinaryDigit: one of
>>>>        0 1
>>>>
>>>> -Joe
>>>>     
>>>>       
>>>
>>>   
>>>     
>>
>>
>>   
>
>