Musings on 8232447: The javadoc parser ends the first sentence of a comment too soon

[Jonathan Gibbons]

Roger,

Generally, I agree with you, and I completely agree with you if we 
regard your
comments as guidelines for Java SE doc comments.  But we also have to
support folk with maybe different standards, and to a lesser extent,
existing comments.

-- Jon


On 5/13/20 10:30 AM, Roger Riggs wrote:
> Hi,
>
> The first sentence is not just  any old sentence.
> It has a very specific role to play in the javadoc both to introduce 
> the class, method, feild, etc.
> AND to stand independently when used in a summary.
> That places a responsibility on the author to craft the sentence for 
> those purposes.
> The author should review their work in the generated javadoc, the 
> summary tables, etc.
> before feeling satisified and moving on.
>
> IMHO the first sentence should be short and to the point and not 
> include markup or
> extra explainatory phrases (such as e.g.).
> I don't think the tools should try to be as understanding as
> the reader or to compensate for the shortcomings of the author.
>
> $.02, Roger
>
>
> On 5/13/20 12:20 PM, Jonathan Gibbons wrote:
>> Pavel,
>>
>> Good write up.   You should link to this from 8232447.
>>
>> -- Jon
>>
>> On 5/13/20 7:44 AM, Pavel Rappo wrote:
>>> The issue:
>>>
>>>      https://bugs.openjdk.java.net/browse/JDK-8232447
>>>
>>> The more I think about this issue, the less I feel like solving it. 
>>> On the one hand, that problem is more complicated than it looks. On 
>>> the other hand, solving that problem doesn’t seem to be that 
>>> important since it’s about making our best-effort to improve 
>>> presentation. I'm leaning towards a solution that is good-enough 
>>> (possibly, the one that we already have) or reconsidering the 
>>> problem altogether.
>>>
>>> Here's what the problem is about. JavaDoc extracts summaries from 
>>> doc comments to place them on documentation pages to assist quick 
>>> scans by humans (think Table of Contents with descriptive headings). 
>>> Since JavaDoc does not understand the meaning of doc comments, to 
>>> extract a summary it relies on a convention [^0] that the first 
>>> sentence of a doc comment is that doc comment's summary. The problem 
>>> is that sometimes JavaDoc gets that first sentence wrong. For 
>>> example, according to JavaDoc, the first sentence of this doc 
>>> comment for `GraphicsEnvironment.preferProportionalFonts` [^1]
>>>
>>>> Indicates a preference for proportional over non-proportional (e.g. 
>>>> dual-spaced CJK fonts) fonts in the mapping of logical fonts to 
>>>> physical fonts. If the default mapping contains fonts for which 
>>>> proportional and non-proportional variants exist, then calling this 
>>>> method indicates the mapping should use a proportional variant.
>>> is
>>>
>>>> Indicates a preference for proportional over non-proportional (e.g.
>>> Now, why does this happen? Unless a more sophisticated mechanism is 
>>> requested or the locale's language is not English, JavaDoc uses a 
>>> simple "dot-space" algorithm to detect a sentence boundary. That 
>>> algorithm scans input from left to right looking for the dot 
>>> character followed by a whitespace. While it looks reasonable, in 
>>> the above case it is clearly inadequate.
>>>
>>> At this point, the reader might say: "Pfft. I know how to fix this." 
>>> Please bear with me and I'll show you that the problem is actually 
>>> multilayered. Not only does it include a sentence segmentation 
>>> algorithm [^2], but input that the algorithm is fed with, as well as 
>>> structure and quality of doc comments the input is created from.
>>>
>>> Instead of jumping head-first into augmenting the "dot-space" 
>>> algorithm with more heuristics, let's try one more thing. If 
>>> instructed to do so or the locale's language is not English, JavaDoc 
>>> uses `BreakIterator` [^3]. That `java.text` mechanism is 
>>> specifically designed to find various boundaries in text. When 
>>> `BreakIterator` is turned on (and after additional tweaking), 
>>> JavaDoc gets that first sentence about "proportional fonts" right, 
>>> however, other issues show up. Consider the following comment for 
>>> `FocusTraversalPolicy.getComponentAfter` [^4]:
>>>
>>>> Returns the Component that should receive the focus after 
>>>> aComponent. aContainer must be a focus cycle root of aComponent or 
>>>> a focus traversal policy provider.
>>> Here `BreakIterator` thinks that the whole paragraph is a single 
>>> sentence. This is because in English sentences begin with capital 
>>> letters. I should pause here. This is an important moment. While 
>>> some doc comments may indeed have typos, irregularities, or quality 
>>> issues, that doc comment about "aComponent" has none of those. It's 
>>> genuine and consists of easily recognizable by humans a couple of 
>>> sentences that do not, however, strictly abide by the rules of 
>>> English Grammar. To me, this (and other experiments with 
>>> `BreakIterator` I've done) shows that doc comments are not your 
>>> regular prose. Unsurprisingly, even a specialized text tool doesn't 
>>> grok it. (Which makes me wonder if that was one of the reasons why 
>>> `BreakIterator` is turned off by default.) Add indentation and 
>>> markup on top of that and you'll see why the ultimate form that 
>>> JavaDoc has to work with is not a string but something like this:
>>>
>>>      list size = 10
>>>       0 = {DCTree$DCStartElement} "<code>"
>>>       1 = {DCTree$DCText} "DOMLocator"
>>>       2 = {DCTree$DCEndElement} "</code>"
>>>       3 = {DCTree$DCText} " is an interface that describes a 
>>> location (e.g.\n where an error occurred).\n "
>>>       4 = {DCTree$DCStartElement} "<p>"
>>>       5 = {DCTree$DCText} "See also the "
>>>       6 = {DCTree$DCStartElement} "<a 
>>> href='http://www.w3.org/TR/2004/REC-DOM-Level-3-Core-20040407'>"
>>>       7 = {DCTree$DCText} "Document Object Model (DOM) Level 3 Core 
>>> Specification"
>>>       8 = {DCTree$DCEndElement} "</a>"
>>>       9 = {DCTree$DCText} "."
>>>
>>> Continuous text we see on a documentation page [^5] in a browser 
>>> comes from a representation such as the above, where the text can be 
>>> scattered across various AST nodes. This has interesting 
>>> implications. Consider the following doc comment (note the 
>>> whitespace after `comment.`):
>>>
>>>      /** This is the first sentence of this <i>comment. </i> This is 
>>> the second sentence. */
>>>
>>> Both simple "dot-space" algorithm and `BreakIterator` fail to 
>>> extract the first sentence here, producing the exact same result 
>>> consisting of both sentences. When `.` is moved immediately after 
>>> the closing `</i>`, they both extract the first sentence correctly. 
>>> However, the HTML output breaks (note the absence of closing `</i>`):
>>>
>>>      <div class="block">This is the first sentence of this 
>>> <i>comment.</div>
>>>
>>> This is partly because JavaDoc does not interpret HTML. Instead, it 
>>> uses a hybrid approach that applies a sentence segmentation 
>>> algorithm as an auxiliary step to individual text nodes (not 
>>> necessarily the whole text) while maintaining awareness of the 
>>> surrounding nodes. The fact that nodes preserve indentation and 
>>> formatting of the original doc comment makes things worse, as 
>>> whitespace is significant in sentence segmentation. No wonder 
>>> JavaDoc hardly sees the forest for the syntax trees! Perhaps, a more 
>>> careful way of doing that would be as follows:
>>>
>>>    1. Interpret markup as text.
>>>    2. Apply sentence segmentation to that text to find the first 
>>> sentence.
>>>    3. Map that first sentence back to markup to accurately extract 
>>> the corresponding portion.
>>>
>>> But even that won't magically solve all the issues as it's not 
>>> possible to decompose an arbitrary markup into independent 
>>> components. Consider the following doc comment:
>>>
>>>      /**
>>>       * <table class="comment">
>>>       *     <tr>
>>>       *        <td><i>Is this the first sentence?</i></td>
>>>       *        <td>Is this the second sentence?</td>
>>>       *     </tr>
>>>       *     <tr>...</tr>
>>>       *  </table>
>>>       ...
>>>
>>> Even if we find that "first sentence", can we safely extract it from 
>>> its table-context? And all this is just the structure layer of the 
>>> problem.
>>>
>>> Next layer is ambiguities. Unless extreme measures are taken those 
>>> are only resolvable by a human, sometimes by an expert in the area 
>>> the documentation relates to. Using abbreviations such as "etc.", 
>>> "e.g.", "i.e.", and "vs." is part of the issue. Early guides [^6] on 
>>> JavaDoc advised against using abbreviations. While I can see now one 
>>> of the reasons for this advice, people use them anyway. Some might 
>>> say that abbreviations can be more succinct and practical. For 
>>> instance, "etc." is shorter than "and so on", "and so forth", or 
>>> "and so on and so forth", and even pronounced literally as "et 
>>> cetera" in speech. Non-standard grammar in abbreviations aggravates 
>>> the issue. For instance, is "ie" a misspelt "i.e.", an initialism of 
>>> Internet Explorer, or a top-level domain name of The Republic of 
>>> Ireland? Or is "etc" is a misspelt "etc." or rather that `/etc` 
>>> directory from the UNIX Filesystem Hierarchy Standard? (When 
>>> scanning OpenJDK repo for occurrences of "etc." in comments, I found 
>>> that it can be written with the number of dots anywhere from 0 to 4. 
>>> The latter could be explained as ellipsis `...` followed by a dot 
>>> `.`, faulty keyboard, or perhaps a muscle twitch.)
>>>
>>> The final layer is typos and low-quality comments. What proportion 
>>> of doc comment follow that convention about the first sentence? What 
>>> proportion of comments respect grammar or have a meaningful 
>>> structure? While we shouldn't aim for a solution that rights the 
>>> wrongs of bad comments (i.e. Garbage In, Garbage Out), this is 
>>> something to keep in mind:
>>>
>>>      /**
>>>       * this function draws the border around each tab
>>>       * note that this function does now draw the background of the 
>>> tab.
>>>       * that is done elsewhere
>>>       ...
>>>       */
>>>       protected void paintTabBorder(Graphics g, int tabPlacement, ...
>>>
>>> There are things we can do to remediate that problem on the doc 
>>> comments side of the equation. Reasonable conventions that are 
>>> adhered to, better structure of doc comments, or hints. For example, 
>>> placing a newline or more than a single whitespace after the first 
>>> sentence. Or indicating the summary part of a doc comment with a 
>>> relatively new `{@summary}` tag. That said, all of those might have 
>>> problems of their own. They are intrusive and require to re-document 
>>> the existing code, which is not always possible. In addition to 
>>> that, `{@summary}` cannot contain nested markup, which is quite 
>>> often used in the summary part. For example
>>>
>>>      /**
>>>       * Returns the runtime class of this {@code Object}. The returned
>>>       * {@code Class} object is the object that is locked by {@code
>>>       * static synchronized} methods of the represented class.
>>>       ...
>>>       */
>>>       public final native Class<?> getClass();
>>>       or
>>>
>>>      /**
>>>       * An ordered collection (also known as a <i>sequence</i>).
>>>       ...
>>>       */
>>>      public interface List<E> extends Collection<E> { ...
>>>      Whatever a solution we choose, there's a risk of playing a 
>>> whac-a-mole game. Maybe we should aim for a solution that is 
>>> good-enough (possibly, the one that we already have) or reconsider 
>>> the problem altogether. For instance, do not extract the first 
>>> sentence (unless it can be done reliably). Instead, get the first N 
>>> characters and indicate continuation (e.g. using ellipsis `...`), or 
>>> use the complete doc-comment, whichever is shorter.
>>>
>>>
>>>
>>>
>>> To sum up, extracting sentences from a text written in a natural 
>>> language is anything but trivial and might require human judgement. 
>>> When done programmatically, occasional mistakes are inevitable. Doc 
>>> comments are barely text. While they have some structure, they also 
>>> use formatting, code, and markup. Hence, without pre-processing text 
>>> tools might not be applicable. Though JavaDoc could improve its 
>>> algorithms and doc comments could be more friendly, what we have 
>>> today works surprisingly well on the OpenJDK codebase. If this is 
>>> not enough, we could find another way of extracting a summary or 
>>> eliminate the need for it completely. That is, change the 
>>> presentation in such a way that it won't require summaries.
>>>
>>> -Pavel
>>>
>>> [^0]: 
>>> https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html#format
>>> [^1]: 
>>> https://docs.oracle.com/en/java/javase/14/docs/api/java.desktop/java/awt/GraphicsEnvironment.html#preferProportionalFonts()
>>> [^2]: https://en.wikipedia.org/wiki/Sentence_boundary_disambiguation
>>> [^3]: 
>>> https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/text/BreakIterator.html
>>> [^4]: 
>>> https://docs.oracle.com/en/java/javase/14/docs/api/java.desktop/java/awt/FocusTraversalPolicy.html#getComponentAfter(java.awt.Container,java.awt.Component)
>>> [^5]: 
>>> https://docs.oracle.com/en/java/javase/14/docs/api/java.xml/org/w3c/dom/DOMLocator.html
>>> [^6]: 
>>> https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html#styleguide
>>>
>>
>