RFR: 8348190: Framework for tracing makefile inclusion and parsing

[Magnus Ihse Bursie]

On Fri, 31 Jan 2025 17:45:44 GMT, Magnus Ihse Bursie <ihse at openjdk.org> wrote:

> A major problem when debugging the build system is figuring out what makefiles are evaluated, and in which order. 
> 
> Our makefiles comes in three flavors, which I have termed "files", "includes" and "snippets", for the purpose of being able to reason about them. The difference is this:
> * "files" (that is, "top-level files") are run directly by calling `make -f FooFile.gmk`
> * "includes" are included by other makesfiles using `include FooInclude.gmk`
> * "snippets" are automatically included by the framework, based on special naming rules
> 
> This patch adds a systematic structure to files, includes and snippets, which allows for a logging framework to be attached.
> 
> The log will look something like this:
> 
> 
> :Enter Main.gmk
> :    Enter common/MakeBase.gmk [include]
> :         Enter common/Utils.gmk [include]
> :         Leave common/Utils.gmk
> :         Enter common/FileUtils.gmk [include]
> :         Leave common/FileUtils.gmk
> :    Leave common/MakeBase.gmk
> :    Enter MainSupport.gmk [include]
> :    Leave MainSupport.gmk
> :    Enter common/FindTests.gmk [include]
> :    Leave common/FindTests.gmk
> :    Enter common/Modules.gmk [include]
> :    Leave common/Modules.gmk
> :Leave Main.gmk [now executing rules]
> 
> 
> Furthermore, this patch achieves a secondary benefit, which is arguably just as important, and that is simplifying the boilerplate in makefiles. I am not saying that it offers strictly *less* boilerplate -- in some cases, it is actually more than what we have right now. But it does offer *systematic* boilerplate. You only have to include the proper start/end makefile and then you're good to go. These start/end files implements the best practices that have evolved over the year (like include guards for includes), and standards we employ in the JDK build system (like always starting with a `default: all` rule). It became painfully obvious when implementing this PR that our adherence to these best practices are sketchy, at best. From now on, it will not be possible to miss declaring best practice boilerplate, since it is all concentrated in a single set of files.
> 
> While updating all makefiles to use these new systematic start/end includes, I took the opportunity to clean up the style used and make it more consistent. So now all makefiles follows this pattern (as far as applicable):
> 
> include Make[File/Include]Start.gmk
> 
> ################################################################################
> # High-level description of this file (if present).
> #############################...

This is obviously a huge patch, but I feel confident in the testing I have performed:
* Running tier1-3 on the Oracle CI
* Building all special platform combinations we support on Oracle CI
* Running the COMPARE_BUILD to check that the same output is generated before and after the patch. There are some caveats at this; the check passed with flying colors some days ago on all standard Oracle CI platforms, but after the Init/PreInit split, and the related changed I made in this branch after rebasing to that, the COMPARE_BUILD setup does not work with this particular patch. (Just to be clear: *after* applying this patch, COMPARE_BUILD works again, it is just not made for a patch that modifies Init.gmk under it's feet). I have made a manual run on macos-aarch64 and linux-x64, and these still compare fine without any differences, and that is enough to convince me this patch does not affect the output bits.
* I have run a series of manual adhoc testing to check stuff like COMPARE_BUILD, run-test-prebuild and the new LOG=flow mechanism.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/23399#issuecomment-2628014271