debuginfo/diz files in built images

[David Holmes]

On 6/09/2012 12:00 AM, Daniel D. Daugherty wrote:
> On 9/5/12 4:46 AM, David Holmes wrote:
>> Dan,
>>
>> Sorry, not meaning to ruffle your feathers.
>
> Not ruffled. Just very tired of FDS.

Understood. Thanks for your patience and forbearance. :)

> If the libraries are removed from the image, then that same logic should
> also remove debug info files or .diz files. Maybe I'm misunderstanding
> something here.

The logic to do so is flawed. Here's how libattach gets excluded:

# Attach library is JDK only
NOTJRE_SHARED_LIBS += $(LIB_PREFIX)attach.$(LIBRARY_SUFFIX)

but the debuginfo.diz file is not also added when it should be.

>> Additionally it is unclear to me why we have diz files for some
>> libraries and not others?
>
> I implemented complete FDS for Linux and Solaris for hotspot.
> I implemented FDS for Linux and Solaris for Runtime and Serviceability
> related libraries in the JDK. I did not implement FDS for any other
> team's libraries.

Thanks for clarifying. I don't recognize all the libraries by name so 
couldn't tell what the logical grouping was.

Cheers,
David
-----

> Of course, Win* has had FDS since JDK1.4.1 so complete FDS is
> implemented everywhere on Windows.
>
>
>> Anyway as I said this will need to be modified for the profile work,
>> in consultation with all the interested parties.
>
> Please keep me in the loop.
>
> Dan
>
>
>>
>> Thanks,
>> David
>>
>> On 4/09/2012 11:51 PM, Daniel D. Daugherty wrote:
>>> On 9/2/12 9:37 PM, David Holmes wrote:
>>>> Hi Dan,
>>>>
>>>> On 3/09/2012 12:47 PM, Daniel D. Daugherty wrote:
>>>>> On 9/2/12 7:26 PM, David Holmes wrote:
>>>>>> The build will create debuginfo/diz files as requested - that's fine.
>>>>>>
>>>>>> What I'm unclear about is where those files should actually appear in
>>>>>> our build artifacts, specifically the jdk/jre images that are
>>>>>> created.
>>>>>>
>>>>>> Internally when we create binary images RE strips all the
>>>>>> debuginfo/diz files out.
>>>>>
>>>>> Not quite. The RE makefiles will put debug info files into a
>>>>> separate debuginfo.zip file. This includes .debuginfo, .diz,
>>>>> .map and .pdb files. For the regular bundles, the RE makefiles
>>>>> will not include debug info files. Not quite the same as
>>>>> stripping them out.
>>>>
>>>> Not trying to split hairs but if the image contains them and the RE
>>>> bundle of the image does not, then the RE process is stripping them
>>>> out.
>>>
>>> The above is implying that there is a single RE bundle and that hasn't
>>> been true for the tar-style universe for quite a long time; the demo
>>> bundle was added by around JDK8-B20 or so. For the package-style
>>> universe, I don't think that has ever been true.
>>>
>>> RE's makefiles take the image generated by the build and package up the
>>> various files into two or more bundles. I believe each platform has two
>>> styles of bundles. For Solaris, SVR4 style packages and tar/zip bundles.
>>> For Windows, an EXE installer and tar/zip bundles. For Linux RPM
>>> packages
>>> and tar/ZIP bundles.
>>>
>>> So I'll stick with my assertion that the RE process is placing debug
>>> info
>>> files into a separate bundle and is not stripping them out.
>>>
>>>
>>>>
>>>>>> But the build itself seems to treat them in an ad-hoc manner:
>>>>>>
>>>>>> - The new build deliberately excludes debuginfo/diz files associated
>>>>>> with binaries, but will include any related to libraries (via generic
>>>>>> copying routine). (It's obvious from the comments related to this
>>>>>> that
>>>>>> there is some puzzlement as to this reasoning.)
>>>>>
>>>>> Sorry I haven't looked at the new build system.
>>>>
>>>> It tries to emulate the old build.
>>>>
>>>>>> - The old build also tries to exclude the files associated with
>>>>>> binaries, but only handles .debuginfo not .diz :(
>>>>>
>>>>> During one round in the FDS project, I included .debuginfo/.diz files
>>>>> right next to some of the binaries. Apparently that's not allowed
>>>>> without
>>>>> explicit permission. The contents of the "bin" directory are
>>>>> controlled
>>>>> and there are tests to verify those contents.
>>>>>
>>>>> I changed the Makefiles for the few binaries that support FDS to not
>>>>> install the debug info files with the binaries, but they are left in
>>>>> the normal build artifacts location if someone has the need to use
>>>>> them. Of course, I've had queries for debug info files for the
>>>>> binaries
>>>>> to be included in the debuginfo.zip bundles. I'm not planning to fight
>>>>> that battle.
>>>>
>>>> Okay that explains the special handling for binaries. So how do these
>>>> then get into the debuginfo.zip? Or don't they?
>>>
>>> They don't. They are only available via the build artifacts that
>>> are archived.
>>>
>>>
>>>>
>>>>>> Those associated with libraries just seem to get copied if they
>>>>>> happen
>>>>>> to be there
>>>>>
>>>>> All of the logic that copies debug info files to the image should do
>>>>> so if they happen to be there. Not all component support FDS now or
>>>>> in the future so the logic needs to adapt to what is built.
>>>>>
>>>>>
>>>>>> As I said this all seems very ad-hoc to me. I would expect to see no
>>>>>> debuginfo/diz files in a created image by default, and have a
>>>>>> separate
>>>>>> target that would produce a tar file of all the debuginfo/diz files
>>>>>> ready to overlay on an existing image.
>>>>>
>>>>> Close. The image has to have the debug info or .diz files in order for
>>>>> the RE makefiles to generate the debuginfo.zip files. It is the RE
>>>>> makefiles that handle the packaging. Just like the demos are always
>>>>> built and it is the RE makefiles that put them in a separate bundle.
>>>>
>>>> So therein lies the problem. RE lies outside of the OpenJDK build
>>>> system. If the RE Makefiles can copy an image they can copy the
>>>> debuginfo files too. I'd much rather see no debuginfo files in the
>>>> built images (unless requested) and a distinct debuginfo related
>>>> target (that RE could use if they choose). That would seem to be
>>>> simpler all round.
>>>
>>> My understanding is that "the image" is supposed to be the complete
>>> image if every bundle is extracted. I believe that is how the packages
>>> are sanity checked to be complete. RE's Makefiles are meant to process
>>> "the image" into bundles. They aren't meant to add files of their own.
>>>
>>> If you are planning to change the meaning of "the image" or if you
>>> are planning to put files into bundles that are not in "the image",
>>> then you'll need to coordinate with RE and whoever does the package
>>> inventory testing.
>>>
>>> Also, if you change the way this works, keep in mind that some of the
>>> Makefiles generate their objects into a temporary directory and then
>>> copy things from that temporary directory to the image. Other Makefiles
>>> generate their objects directly into the image. For the latter, you'll
>>> have to move the debug info file(s) out of the image into some other
>>> place where it can be picked up by whatever bundling process you create
>>> to make the debuginfo.zip file.
>>>
>>>
>>>>
>>>>>> Thoughts/comments?
>>>>>>
>>>>>> I have to deal with this for the SE Profile work, where we will not
>>>>>> want these files present in any of the images.
>>>>>
>>>>> It should be fine if the Embedded profile does not generate debug info
>>>>> files or if the Embedded profile generates debug info, but does not
>>>>> copy the debug info files into the image area. The RE makefiles will
>>>>> handle the presence or absence of debug info files.
>>>>
>>>> These are not embedded profiles, these are just profiles - it is all
>>>> to be part of SE in Java 8. The build is supposed to create an image
>>>> corresponding to each profile. I can certainly specialise the handling
>>>> of the debuginfo files for the profiles, but it seemed to me that the
>>>> overall debuginfo-in-image story was rather unclear.
>>>
>>> I'm sorry you think that the debuginfo-in-image story is unclear.
>>> I thought I did a pretty good job integrating FDS into the whole
>>> OpenJDK build process somewhat seamlessly. I guess not.
>>>
>>> The key things to remember in all this is that debug info files
>>> (like the demos) have to end up in a separate bundle or bundles.
>>> So if you change the way this is all implemented, you still have
>>> to follow the separate bundle rule.
>>>
>>>
>>>>
>>>>> However, the non-Embedded profile needs to continue to generate and
>>>>> copy debug info files for those components that already support FDS.
>>>>> If you change the non-Embedded logic to not copy the debug info files
>>>>> into the image area, the RE debuginfo.zip bundle will not be created
>>>>> and that will break FDS.
>>>>
>>>> RE processes will have to change to handle profiles in any case. I
>>>> think they'd rather get a debuginfo.zip out of the build than having
>>>> to create it themselves.
>>>
>>> Interesting and not something that I have ever heard. I got the
>>> impression that RE needed to have control over what was bundled
>>> and how it was bundled. It's kind of hard to certify/vouch for a
>>> process when you don't do it yourself, but that's not really my
>>> call.
>>>
>>> Good luck with the tweaking of the build and packaging processes.
>>> Just remember: There be dragons here!
>>>
>>> Dan
>>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>>>
>>>>> Dan
>>>>>
>>>>>
>>>>>>
>>>>>> David