RFR: 8293116: Incremental JDK build could be sped up [v2]

[Erik Joelsson]

On Thu, 1 Sep 2022 12:43:12 GMT, Jan Lahoda <jlahoda at openjdk.org> wrote:

>> make/common/JavaCompilation.gmk line 456:
>> 
>>> 454: 		$$(call LogWarn, Compiling up to $$(words $$($1_SRCS)) files for $1)
>>> 455: 		$$(eval $$(call ListPathsSafely, $1_SRCS, $$($1_FILELIST)))
>>> 456: 		$$(call WriteFile, "-XDmodifiedInputs=$$? ", $$@.modfiles)
>> 
>> I'm not sure WriteFile is safe to use here. `$?` could potentially contain all files in `$1_SRCS`, which is why we use ListPathsSafely in the first place. This is only a problem if using make <4.0, but as long as that's not prohibited, we need to keep it working without the file function.
>
> Is there some recommendation on how to generate what is needed? Which is:
> 
> "-XDmodifiedFiles=<list-of-files> "
> 
> 
> I was looking at the `WriteFile` rule, and it is not clear to be what is the problem there, but I'm not very knowledgeable of the build system.
> 
> I was trying to use `ListPathsSafely`, but wasn't able to find a way that would be working.
> 
> Thanks!

Before make 4.0, there was no `$(file ...)` function in make. That meant that if you needed to write something to a file directly from the makefile, you had to do something like `$(shell echo "foo" > /my/file)`. This works ok as long as the amount of text you need to write is small enough, but when writing long lists of filenames, you will eventually hit OS limits on command line length (which are different on different OSes, most notably much shorter on Windows). Our solution to this was ListPathsSafely, which breaks up long lists of path names into manageable chunks (and also compresses parts of the path using sed).

In make 4.0, the $(file) function was introduced, which makes this so much easier. Since it was also more performant, I implemented a 4.0 version of ListPathsSafely. However, we haven't changed the build to require make >4.0, so we have to keep both implementations around.

The WriteFile macro was only intended writing smaller files, so doesn't have any of the command splitting functionality of ListPathsSafely. 

So to answer your question, no there isn't a good way to achieve what you are asking for today, not without modifying or extending the macros. One could imagine adding a flag to ListPathsSafely that made it use space instead of newline as separator. Another solution could be to have Depend.java accept something like 

"-XDmodifiedFilesFile=/path/to/file"

where the file is a list as ListPathsSafely would create it.

We could also consider requiring make 4.0, but that will need some socializing first.

-------------

PR: https://git.openjdk.org/jdk/pull/10104