StandardDoclet requires private DocletEnvironment (due to a cast in WorkArounds)

[Jonathan Gibbons]

Dawid,

Thank you for the write-up. It's always interesting to hear stories of
how people are trying to use the tools and APIs. If nothing else, it
helps inform us how to improve our documentation.

In this case, it would (probably) be easier for you to just use a
standard StandardFileManager (no subtyping required) and use
the setLocation call to set the DOCUMENTATION_OUTPUT location
to your zip file system ... avoiding the need to pass the -d option
into the tool.

At least part of the problem we're dealing with here is that JavaFileManager
was not designed for all of javadoc's needs. (It was designed with javac
in mind, and designed before the NIO file system API was available.)
Rather than extend JavaFileManager to become another full-featured
virtual file system, the evolution has been to increase the 
inter-operability
between the StandardFileManager and the NIO file system, and to rely
on the features there. That applies both within the tool and for API 
clients.

On supporting a ZIP file system, ... "possibly".  Previously, we've mostly
considered (and rejected) this in terms of javac, where iterative 
development
is more common, and less desirable to use a zip file system.  In the context
of javadoc, maybe there's enough of a reasonable use case to write the
output into an initially empty zip file.

-- Jon

On 7/10/20 6:03 AM, Dawid Weiss wrote:
> I thought I'd revive this old thread -- didn't have time to check your
> suggestion until now, Jon. I tried the DocumentationTool approach.
> You're right - there's a lot of magic going on behind the scenes in
> javadoc... :) Please read if you have a spare moment (nothing urgent).
>
> So I set up a basic snippet that generates javadoc from code and it
> works... until you wrap it with an (empty) ForwardingJavaFileManager.
> So something like this works:
>
> JavaFileManager fileManager =
>      jdoc.getStandardFileManager(diagnosticListener, Locale.ROOT,
> StandardCharsets.UTF_8);
>
> // fileManager = new ForwardingJavaFileManager<>(fileManager) {};
>
> List<String> options =
>      List.of(
>          "-d", "output", "-subpackages", "com.carrotsearch",
>          "-sourcepath", "src/main/java");
>
> DocumentationTool.DocumentationTask task =
>      jdoc.getTask(extraOutput, fileManager, diagnosticListener, null,
> options, compilationUnits);
>
> But as soon as you uncomment forwarding file manager line the output becomes:
>
>> NOTE: javadoc: Loading source files for package com.carrotsearch...
>> ERROR: compiler message file broken: key=javadoc.err.msg arguments=javadoc, No source files for package com.carrotsearch, {2}, {3}, {4}, {5}, {6}, {7}
>> NOTE: javadoc: 1 error
> The difference has to do with SOURCE_PATH location lookup (when
> javadoc is calling fm.list(...)). The returned file manager is
> JavacFileManager but this check in Start.java only conditionally
> passes certain options:
>
> if (fileManager instanceof BaseFileManager) {
>      ((BaseFileManager) fileManager).handleOptions(fileManagerOpts);
> }
>
> I worked this around by passing the required option explicitly
> (created a full delegate
> to StandardJavaFileManager to get access to setLocation):
>
> fileManager = new ForwardingStandardJavaFileManager(fileManager);
> fileManager.setLocation(StandardLocation.SOURCE_PATH, List.of(new
> File("src/main/java")));
>
> then overrode getFileForOutput, wrapped everything in a zip file
> system and - behold - it worked! :)
>
> It's all rather hacky but I ran it successfully on Lucene (core). I
> didn't see much in terms of speed improvements -- one average one
> second faster than full disk-writing version:
>
> no output: ~10.5s   -
> Zip:  ~11s   size on disk: 5.3 MB
> Disk (SSD): ~11.5s, size on disk: 27.6 MB
> Disk (HDD): ~12s,  size on disk: 27.6 MB
>
> I still think such a "zip" destination would be handy. It'd make
> cleanup or dependency checks in build systems like gradle faster
> (fewer files to check and/or remove).
>
> Anyway, enjoyed the experiment - thank you for the hint!
>
> Dawid
>
>
>
> On Thu, May 7, 2020 at 4:59 PM Jonathan Gibbons
> <jonathan.gibbons at oracle.com> wrote:
>> Dawid,
>>
>> I agree that the situation is less than ideal.  The possibility/desire
>> to subtype the standard doclet and to provide a modified environment
>> was not anticipated when we wrote the current version of the doclet ...
>> and even the need for that internal 'WorkArounds' class is less than
>> ideal, as you might guess from the name.
>>
>> But, keep asking these questions; it helps us to know how folk are
>> trying to use tools like javadoc.
>>
>> -- Jon
>>
>> On 5/7/20 7:40 AM, Dawid Weiss wrote:
>>> Thanks Jon. It is a pity there is so little flexibility around the
>>> standard doclet (or its HTML producer) because rewriting everything
>>> from scratch is out of question. I'll take a look at getting it to
>>> work via tool provider - thank you for the hint!
>>>
>>> D.
>>>
>>> On Thu, May 7, 2020 at 4:33 PM Jonathan Gibbons
>>> <jonathan.gibbons at oracle.com> wrote:
>>>> Dawid,
>>>>
>>>> The issue about being difficult to subtype the StandardDoclet is a
>>>> known issue.
>>>>
>>>> You should be able to provide your own file manager if you invoke
>>>> javadoc via the API available in javax.tools.
>>>>
>>>> https://docs.oracle.com/en/java/javase/11/docs/api/java.compiler/javax/tools/ToolProvider.html#getSystemDocumentationTool()
>>>>
>>>> -- Jon
>>>>
>>>>
>>>> On 5/7/20 1:29 AM, Dawid Weiss wrote:
>>>>> Hello everyone,
>>>>>
>>>>> I had a somewhat wild idea of trying to modify the behavior of the
>>>>> StandardDoclet to write directly to a ZIP file (instead of the file
>>>>> system). All the pieces fall into place relatively easily:
>>>>>
>>>>> 1) Create a custom Doclet that extends (public) StandardDoclet and
>>>>> overrides the run method with:
>>>>>
>>>>> public boolean run(DocletEnvironment docEnv) {
>>>>>        return super.run(new CustomEnvironment(docEnv));
>>>>> }
>>>>>
>>>>> 2) CustomEnvironment delegates everything to docEnv except it returns
>>>>> a ForwardingJavaFileManager over the docEnv.getJavaFileManager with a
>>>>> bunch of "get*Input" methods substituted.
>>>>>
>>>>>
>>>>> The only problem is - it doesn't work. :) Internally WorkArounds has a
>>>>> cast to an internal implementation type:
>>>>>
>>>>> public WorkArounds(BaseConfiguration configuration) {
>>>>>        [...]
>>>>>        this.toolEnv = ((DocEnvImpl)this.configuration.docEnv).toolEnv;
>>>>> }
>>>>>
>>>>> And it's a showstopper (maybe there are others but I didn't have a
>>>>> chance to try).
>>>>>
>>>>> Is there any way forward out of this (or an alternative take I can try)?
>>>>>
>>>>> Dawid