Musings on 8232447: The javadoc parser ends the first sentence of a comment too soon

[Roger Riggs]

Hi,

The first sentence is not just  any old sentence.
It has a very specific role to play in the javadoc both to introduce the 
class, method, feild, etc.
AND to stand independently when used in a summary.
That places a responsibility on the author to craft the sentence for 
those purposes.
The author should review their work in the generated javadoc, the 
summary tables, etc.
before feeling satisified and moving on.

IMHO the first sentence should be short and to the point and not include 
markup or
extra explainatory phrases (such as e.g.).
I don't think the tools should try to be as understanding as
the reader or to compensate for the shortcomings of the author.

$.02, Roger


On 5/13/20 12:20 PM, Jonathan Gibbons wrote:
> Pavel,
>
> Good write up.   You should link to this from 8232447.
>
> -- Jon
>
> On 5/13/20 7:44 AM, Pavel Rappo wrote:
>> The issue:
>>
>>      https://bugs.openjdk.java.net/browse/JDK-8232447
>>
>> The more I think about this issue, the less I feel like solving it. 
>> On the one hand, that problem is more complicated than it looks. On 
>> the other hand, solving that problem doesn’t seem to be that 
>> important since it’s about making our best-effort to improve 
>> presentation. I'm leaning towards a solution that is good-enough 
>> (possibly, the one that we already have) or reconsidering the problem 
>> altogether.
>>
>> Here's what the problem is about. JavaDoc extracts summaries from doc 
>> comments to place them on documentation pages to assist quick scans 
>> by humans (think Table of Contents with descriptive headings). Since 
>> JavaDoc does not understand the meaning of doc comments, to extract a 
>> summary it relies on a convention [^0] that the first sentence of a 
>> doc comment is that doc comment's summary. The problem is that 
>> sometimes JavaDoc gets that first sentence wrong. For example, 
>> according to JavaDoc, the first sentence of this doc comment for 
>> `GraphicsEnvironment.preferProportionalFonts` [^1]
>>
>>> Indicates a preference for proportional over non-proportional (e.g. 
>>> dual-spaced CJK fonts) fonts in the mapping of logical fonts to 
>>> physical fonts. If the default mapping contains fonts for which 
>>> proportional and non-proportional variants exist, then calling this 
>>> method indicates the mapping should use a proportional variant.
>> is
>>
>>> Indicates a preference for proportional over non-proportional (e.g.
>> Now, why does this happen? Unless a more sophisticated mechanism is 
>> requested or the locale's language is not English, JavaDoc uses a 
>> simple "dot-space" algorithm to detect a sentence boundary. That 
>> algorithm scans input from left to right looking for the dot 
>> character followed by a whitespace. While it looks reasonable, in the 
>> above case it is clearly inadequate.
>>
>> At this point, the reader might say: "Pfft. I know how to fix this." 
>> Please bear with me and I'll show you that the problem is actually 
>> multilayered. Not only does it include a sentence segmentation 
>> algorithm [^2], but input that the algorithm is fed with, as well as 
>> structure and quality of doc comments the input is created from.
>>
>> Instead of jumping head-first into augmenting the "dot-space" 
>> algorithm with more heuristics, let's try one more thing. If 
>> instructed to do so or the locale's language is not English, JavaDoc 
>> uses `BreakIterator` [^3]. That `java.text` mechanism is specifically 
>> designed to find various boundaries in text. When `BreakIterator` is 
>> turned on (and after additional tweaking), JavaDoc gets that first 
>> sentence about "proportional fonts" right, however, other issues show 
>> up. Consider the following comment for 
>> `FocusTraversalPolicy.getComponentAfter` [^4]:
>>
>>> Returns the Component that should receive the focus after 
>>> aComponent. aContainer must be a focus cycle root of aComponent or a 
>>> focus traversal policy provider.
>> Here `BreakIterator` thinks that the whole paragraph is a single 
>> sentence. This is because in English sentences begin with capital 
>> letters. I should pause here. This is an important moment. While some 
>> doc comments may indeed have typos, irregularities, or quality 
>> issues, that doc comment about "aComponent" has none of those. It's 
>> genuine and consists of easily recognizable by humans a couple of 
>> sentences that do not, however, strictly abide by the rules of 
>> English Grammar. To me, this (and other experiments with 
>> `BreakIterator` I've done) shows that doc comments are not your 
>> regular prose. Unsurprisingly, even a specialized text tool doesn't 
>> grok it. (Which makes me wonder if that was one of the reasons why 
>> `BreakIterator` is turned off by default.) Add indentation and markup 
>> on top of that and you'll see why the ultimate form that JavaDoc has 
>> to work with is not a string but something like this:
>>
>>      list size = 10
>>       0 = {DCTree$DCStartElement} "<code>"
>>       1 = {DCTree$DCText} "DOMLocator"
>>       2 = {DCTree$DCEndElement} "</code>"
>>       3 = {DCTree$DCText} " is an interface that describes a location 
>> (e.g.\n where an error occurred).\n "
>>       4 = {DCTree$DCStartElement} "<p>"
>>       5 = {DCTree$DCText} "See also the "
>>       6 = {DCTree$DCStartElement} "<a 
>> href='http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407'>"
>>       7 = {DCTree$DCText} "Document Object Model (DOM) Level 3 Core 
>> Specification"
>>       8 = {DCTree$DCEndElement} "</a>"
>>       9 = {DCTree$DCText} "."
>>
>> Continuous text we see on a documentation page [^5] in a browser 
>> comes from a representation such as the above, where the text can be 
>> scattered across various AST nodes. This has interesting 
>> implications. Consider the following doc comment (note the whitespace 
>> after `comment.`):
>>
>>      /** This is the first sentence of this <i>comment. </i> This is 
>> the second sentence. */
>>
>> Both simple "dot-space" algorithm and `BreakIterator` fail to extract 
>> the first sentence here, producing the exact same result consisting 
>> of both sentences. When `.` is moved immediately after the closing 
>> `</i>`, they both extract the first sentence correctly. However, the 
>> HTML output breaks (note the absence of closing `</i>`):
>>
>>      <div class="block">This is the first sentence of this 
>> <i>comment.</div>
>>
>> This is partly because JavaDoc does not interpret HTML. Instead, it 
>> uses a hybrid approach that applies a sentence segmentation algorithm 
>> as an auxiliary step to individual text nodes (not necessarily the 
>> whole text) while maintaining awareness of the surrounding nodes. The 
>> fact that nodes preserve indentation and formatting of the original 
>> doc comment makes things worse, as whitespace is significant in 
>> sentence segmentation. No wonder JavaDoc hardly sees the forest for 
>> the syntax trees! Perhaps, a more careful way of doing that would be 
>> as follows:
>>
>>    1. Interpret markup as text.
>>    2. Apply sentence segmentation to that text to find the first 
>> sentence.
>>    3. Map that first sentence back to markup to accurately extract 
>> the corresponding portion.
>>
>> But even that won't magically solve all the issues as it's not 
>> possible to decompose an arbitrary markup into independent 
>> components. Consider the following doc comment:
>>
>>      /**
>>       * <table class="comment">
>>       *     <tr>
>>       *        <td><i>Is this the first sentence?</i></td>
>>       *        <td>Is this the second sentence?</td>
>>       *     </tr>
>>       *     <tr>...</tr>
>>       *  </table>
>>       ...
>>
>> Even if we find that "first sentence", can we safely extract it from 
>> its table-context? And all this is just the structure layer of the 
>> problem.
>>
>> Next layer is ambiguities. Unless extreme measures are taken those 
>> are only resolvable by a human, sometimes by an expert in the area 
>> the documentation relates to. Using abbreviations such as "etc.", 
>> "e.g.", "i.e.", and "vs." is part of the issue. Early guides [^6] on 
>> JavaDoc advised against using abbreviations. While I can see now one 
>> of the reasons for this advice, people use them anyway. Some might 
>> say that abbreviations can be more succinct and practical. For 
>> instance, "etc." is shorter than "and so on", "and so forth", or "and 
>> so on and so forth", and even pronounced literally as "et cetera" in 
>> speech. Non-standard grammar in abbreviations aggravates the issue. 
>> For instance, is "ie" a misspelt "i.e.", an initialism of Internet 
>> Explorer, or a top-level domain name of The Republic of Ireland? Or 
>> is "etc" is a misspelt "etc." or rather that `/etc` directory from 
>> the UNIX Filesystem Hierarchy Standard? (When scanning OpenJDK repo 
>> for occurrences of "etc." in comments, I found that it can be written 
>> with the number of dots anywhere from 0 to 4. The latter could be 
>> explained as ellipsis `...` followed by a dot `.`, faulty keyboard, 
>> or perhaps a muscle twitch.)
>>
>> The final layer is typos and low-quality comments. What proportion of 
>> doc comment follow that convention about the first sentence? What 
>> proportion of comments respect grammar or have a meaningful 
>> structure? While we shouldn't aim for a solution that rights the 
>> wrongs of bad comments (i.e. Garbage In, Garbage Out), this is 
>> something to keep in mind:
>>
>>      /**
>>       * this function draws the border around each tab
>>       * note that this function does now draw the background of the tab.
>>       * that is done elsewhere
>>       ...
>>       */
>>       protected void paintTabBorder(Graphics g, int tabPlacement, ...
>>
>> There are things we can do to remediate that problem on the doc 
>> comments side of the equation. Reasonable conventions that are 
>> adhered to, better structure of doc comments, or hints. For example, 
>> placing a newline or more than a single whitespace after the first 
>> sentence. Or indicating the summary part of a doc comment with a 
>> relatively new `{@summary}` tag. That said, all of those might have 
>> problems of their own. They are intrusive and require to re-document 
>> the existing code, which is not always possible. In addition to that, 
>> `{@summary}` cannot contain nested markup, which is quite often used 
>> in the summary part. For example
>>
>>      /**
>>       * Returns the runtime class of this {@code Object}. The returned
>>       * {@code Class} object is the object that is locked by {@code
>>       * static synchronized} methods of the represented class.
>>       ...
>>       */
>>       public final native Class<?> getClass();
>>       or
>>
>>      /**
>>       * An ordered collection (also known as a <i>sequence</i>).
>>       ...
>>       */
>>      public interface List<E> extends Collection<E> { ...
>>      Whatever a solution we choose, there's a risk of playing a 
>> whac-a-mole game. Maybe we should aim for a solution that is 
>> good-enough (possibly, the one that we already have) or reconsider 
>> the problem altogether. For instance, do not extract the first 
>> sentence (unless it can be done reliably). Instead, get the first N 
>> characters and indicate continuation (e.g. using ellipsis `...`), or 
>> use the complete doc-comment, whichever is shorter.
>>
>>
>>
>>
>> To sum up, extracting sentences from a text written in a natural 
>> language is anything but trivial and might require human judgement. 
>> When done programmatically, occasional mistakes are inevitable. Doc 
>> comments are barely text. While they have some structure, they also 
>> use formatting, code, and markup. Hence, without pre-processing text 
>> tools might not be applicable. Though JavaDoc could improve its 
>> algorithms and doc comments could be more friendly, what we have 
>> today works surprisingly well on the OpenJDK codebase. If this is not 
>> enough, we could find another way of extracting a summary or 
>> eliminate the need for it completely. That is, change the 
>> presentation in such a way that it won't require summaries.
>>
>> -Pavel
>>
>> [^0]: 
>> https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html#format
>> [^1]: 
>> https://docs.oracle.com/en/java/javase/14/docs/api/java.desktop/java/awt/GraphicsEnvironment.html#preferProportionalFonts()
>> [^2]: https://en.wikipedia.org/wiki/Sentence_boundary_disambiguation
>> [^3]: 
>> https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/text/BreakIterator.html
>> [^4]: 
>> https://docs.oracle.com/en/java/javase/14/docs/api/java.desktop/java/awt/FocusTraversalPolicy.html#getComponentAfter(java.awt.Container,java.awt.Component)
>> [^5]: 
>> https://docs.oracle.com/en/java/javase/14/docs/api/java.xml/org/w3c/dom/DOMLocator.html
>> [^6]: 
>> https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html#styleguide
>>
>