debuginfo/diz files in built images

[David Holmes]

Dan,

Sorry, not meaning to ruffle your feathers.

The reason I say the diz-in-image story is unclear is because there are 
no explicit rules that indicates that diz files should end up in a JDK 
or JRE image. I can accept that everything should go in a full JDK 
image. But a JRE is a JDK with a bunch of things left out and some of 
those things are libraries. But because diz files are not mentioned 
explicitly in the rules for making images we actually get the diz files 
for the libraries that were removed! They are simply copied from the 
build artifacts location to the jre image location specifically 
libattach.diz and libsaproc.diz. Maybe that was a simple oversight.

Additionally it is unclear to me why we have diz files for some 
libraries and not others?

Anyway as I said this will need to be modified for the profile work, in 
consultation with all the interested parties.

Thanks,
David

On 4/09/2012 11:51 PM, Daniel D. Daugherty wrote:
> On 9/2/12 9:37 PM, David Holmes wrote:
>> Hi Dan,
>>
>> On 3/09/2012 12:47 PM, Daniel D. Daugherty wrote:
>>> On 9/2/12 7:26 PM, David Holmes wrote:
>>>> The build will create debuginfo/diz files as requested - that's fine.
>>>>
>>>> What I'm unclear about is where those files should actually appear in
>>>> our build artifacts, specifically the jdk/jre images that are created.
>>>>
>>>> Internally when we create binary images RE strips all the
>>>> debuginfo/diz files out.
>>>
>>> Not quite. The RE makefiles will put debug info files into a
>>> separate debuginfo.zip file. This includes .debuginfo, .diz,
>>> .map and .pdb files. For the regular bundles, the RE makefiles
>>> will not include debug info files. Not quite the same as
>>> stripping them out.
>>
>> Not trying to split hairs but if the image contains them and the RE
>> bundle of the image does not, then the RE process is stripping them out.
>
> The above is implying that there is a single RE bundle and that hasn't
> been true for the tar-style universe for quite a long time; the demo
> bundle was added by around JDK8-B20 or so. For the package-style
> universe, I don't think that has ever been true.
>
> RE's makefiles take the image generated by the build and package up the
> various files into two or more bundles. I believe each platform has two
> styles of bundles. For Solaris, SVR4 style packages and tar/zip bundles.
> For Windows, an EXE installer and tar/zip bundles. For Linux RPM packages
> and tar/ZIP bundles.
>
> So I'll stick with my assertion that the RE process is placing debug info
> files into a separate bundle and is not stripping them out.
>
>
>>
>>>> But the build itself seems to treat them in an ad-hoc manner:
>>>>
>>>> - The new build deliberately excludes debuginfo/diz files associated
>>>> with binaries, but will include any related to libraries (via generic
>>>> copying routine). (It's obvious from the comments related to this that
>>>> there is some puzzlement as to this reasoning.)
>>>
>>> Sorry I haven't looked at the new build system.
>>
>> It tries to emulate the old build.
>>
>>>> - The old build also tries to exclude the files associated with
>>>> binaries, but only handles .debuginfo not .diz :(
>>>
>>> During one round in the FDS project, I included .debuginfo/.diz files
>>> right next to some of the binaries. Apparently that's not allowed
>>> without
>>> explicit permission. The contents of the "bin" directory are controlled
>>> and there are tests to verify those contents.
>>>
>>> I changed the Makefiles for the few binaries that support FDS to not
>>> install the debug info files with the binaries, but they are left in
>>> the normal build artifacts location if someone has the need to use
>>> them. Of course, I've had queries for debug info files for the binaries
>>> to be included in the debuginfo.zip bundles. I'm not planning to fight
>>> that battle.
>>
>> Okay that explains the special handling for binaries. So how do these
>> then get into the debuginfo.zip? Or don't they?
>
> They don't. They are only available via the build artifacts that
> are archived.
>
>
>>
>>>> Those associated with libraries just seem to get copied if they happen
>>>> to be there
>>>
>>> All of the logic that copies debug info files to the image should do
>>> so if they happen to be there. Not all component support FDS now or
>>> in the future so the logic needs to adapt to what is built.
>>>
>>>
>>>> As I said this all seems very ad-hoc to me. I would expect to see no
>>>> debuginfo/diz files in a created image by default, and have a separate
>>>> target that would produce a tar file of all the debuginfo/diz files
>>>> ready to overlay on an existing image.
>>>
>>> Close. The image has to have the debug info or .diz files in order for
>>> the RE makefiles to generate the debuginfo.zip files. It is the RE
>>> makefiles that handle the packaging. Just like the demos are always
>>> built and it is the RE makefiles that put them in a separate bundle.
>>
>> So therein lies the problem. RE lies outside of the OpenJDK build
>> system. If the RE Makefiles can copy an image they can copy the
>> debuginfo files too. I'd much rather see no debuginfo files in the
>> built images (unless requested) and a distinct debuginfo related
>> target (that RE could use if they choose). That would seem to be
>> simpler all round.
>
> My understanding is that "the image" is supposed to be the complete
> image if every bundle is extracted. I believe that is how the packages
> are sanity checked to be complete. RE's Makefiles are meant to process
> "the image" into bundles. They aren't meant to add files of their own.
>
> If you are planning to change the meaning of "the image" or if you
> are planning to put files into bundles that are not in "the image",
> then you'll need to coordinate with RE and whoever does the package
> inventory testing.
>
> Also, if you change the way this works, keep in mind that some of the
> Makefiles generate their objects into a temporary directory and then
> copy things from that temporary directory to the image. Other Makefiles
> generate their objects directly into the image. For the latter, you'll
> have to move the debug info file(s) out of the image into some other
> place where it can be picked up by whatever bundling process you create
> to make the debuginfo.zip file.
>
>
>>
>>>> Thoughts/comments?
>>>>
>>>> I have to deal with this for the SE Profile work, where we will not
>>>> want these files present in any of the images.
>>>
>>> It should be fine if the Embedded profile does not generate debug info
>>> files or if the Embedded profile generates debug info, but does not
>>> copy the debug info files into the image area. The RE makefiles will
>>> handle the presence or absence of debug info files.
>>
>> These are not embedded profiles, these are just profiles - it is all
>> to be part of SE in Java 8. The build is supposed to create an image
>> corresponding to each profile. I can certainly specialise the handling
>> of the debuginfo files for the profiles, but it seemed to me that the
>> overall debuginfo-in-image story was rather unclear.
>
> I'm sorry you think that the debuginfo-in-image story is unclear.
> I thought I did a pretty good job integrating FDS into the whole
> OpenJDK build process somewhat seamlessly. I guess not.
>
> The key things to remember in all this is that debug info files
> (like the demos) have to end up in a separate bundle or bundles.
> So if you change the way this is all implemented, you still have
> to follow the separate bundle rule.
>
>
>>
>>> However, the non-Embedded profile needs to continue to generate and
>>> copy debug info files for those components that already support FDS.
>>> If you change the non-Embedded logic to not copy the debug info files
>>> into the image area, the RE debuginfo.zip bundle will not be created
>>> and that will break FDS.
>>
>> RE processes will have to change to handle profiles in any case. I
>> think they'd rather get a debuginfo.zip out of the build than having
>> to create it themselves.
>
> Interesting and not something that I have ever heard. I got the
> impression that RE needed to have control over what was bundled
> and how it was bundled. It's kind of hard to certify/vouch for a
> process when you don't do it yourself, but that's not really my
> call.
>
> Good luck with the tweaking of the build and packaging processes.
> Just remember: There be dragons here!
>
> Dan
>
>
>
>>
>> Thanks,
>> David
>>
>>>
>>> Dan
>>>
>>>
>>>>
>>>> David