debuginfo/diz files in built images

[David Holmes]

On 7/09/2012 11:15 PM, Andrew Hughes wrote:
>> On 6/09/2012 1:00 AM, Andrew Hughes wrote:
>>>>
>>>> Should .diz files be removed when packaging JDK/JRE for Linux ?
>>>>
>>>
>>> Specifically talking about packaging, distro build systems already
>>> handle
>>> stripping, etc. themselves so we've been setting
>>> STRIP_POLICY=no_strip.
>>> However, this still creates the diz files, even though the
>>> debuginfo is
>>> still in the binaries.
>>>
>>> I have a couple of changesets to fix this, one for HotSpot&   one
>>> for JDK,
>>> basically making diz production conditional on STRIP_POLICY not
>>> being set
>>> to no_strip.  I'll try and upstream them, if that sounds the
>>> correct thing to do.
>>
>> We already have ENABLE_FULL_DEBUG_SYMBOLS to control whether we
>> should try to produce the debuginfo files.
>
> No.  That implies (at least by the naming) whether or not the binaries/libraries
> are built with debugging symbols, not whether or not they are stripped into separate
> files.  That is governed by STRIP_POLICY as far as I can tell.

That is not my understanding. STRIP_POLICY is applied to the built 
libraries/binaries as a final step in creating the images. That is quite 
distinct to the creation of the debuginfo files when the 
libraries/binaries are actually built. This is certainly the case for 
hotspot and I believe also those parts of the JDK for which FDS was 
enabled/applied.

David
-----

> I have thought of turning off ENABLE_FULL_DEBUG_SYMBOLS based on the existing
> Makefile logic, but this sounds like I'm turning off debug symbols which I very much
> DON'T want to do.  To me, the right solution seems to be to implement STRIP_POLICY=no_strip
> correctly.
>
> We still want debugging symbols as we had before.  We don't want superfluous copies
> in diz files.
>
>> And ZIP_DEBUGINFO_FILES to
>> determine
>> whether we then zip it to produce the .diz file.
>>
>> My query is whether those files should then end up in the jre and jdk
>> images that the build produces. I had thought no because the Oracle
>> JDKs
>> don't contain them and our RE processes have to exclude them when
>> creating our bundles. Further, there is a bug in the build logic that
>> causes a JRE to get diz files for libraries that aren't there. I
>> thought
>> a separate debuginfo.zip target might be better and leave them out of
>> the images.
>>
>> Do other distros "package" things based on the jre and jdk images
>> produced by the build or do they use the raw build artefacts
>> directly?
>>
>
> The ones I'm aware of use the images, but we don't want these diz files.
> Distros have their own ways of stripping debug info and packaging it
> separately that has been in place for many years before this was added
> to OpenJDK.
>
> However, if someone was doing a local build, I presume they'd want them
> wherever they'd be picked up automatically by the debugger, not hidden
> away somewhere.
>
> To be honest, these changes have been nothing but a pain for us, changing
> the type of debug info produced to STABS on some architectures and removing
> needed debug info from files (until we learnt about STRIP_POLICY).  The build
> prior to FDS was preferable and these major build changes seem to have been
> rushed through too quickly, especially to 7u.
>
>> Thanks,
>> David
>>
>>
>>>> 2012/9/5 David Holmes<david.holmes at oracle.com>:
>>>>> Dan,
>>>>>
>>>>> Sorry, not meaning to ruffle your feathers.
>>>>>
>>>>> The reason I say the diz-in-image story is unclear is because
>>>>> there
>>>>> are no
>>>>> explicit rules that indicates that diz files should end up in a
>>>>> JDK
>>>>> or JRE
>>>>> image. I can accept that everything should go in a full JDK
>>>>> image.
>>>>> But a JRE
>>>>> is a JDK with a bunch of things left out and some of those things
>>>>> are
>>>>> libraries. But because diz files are not mentioned explicitly in
>>>>> the rules
>>>>> for making images we actually get the diz files for the libraries
>>>>> that were
>>>>> removed! They are simply copied from the build artifacts location
>>>>> to the jre
>>>>> image location specifically libattach.diz and libsaproc.diz.
>>>>> Maybe
>>>>> that was
>>>>> a simple oversight.
>>>>>
>>>>> Additionally it is unclear to me why we have diz files for some
>>>>> libraries
>>>>> and not others?
>>>>>
>>>>> Anyway as I said this will need to be modified for the profile
>>>>> work, in
>>>>> consultation with all the interested parties.
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>> On 4/09/2012 11:51 PM, Daniel D. Daugherty wrote:
>>>>>>
>>>>>> On 9/2/12 9:37 PM, David Holmes wrote:
>>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> On 3/09/2012 12:47 PM, Daniel D. Daugherty wrote:
>>>>>>>>
>>>>>>>> On 9/2/12 7:26 PM, David Holmes wrote:
>>>>>>>>>
>>>>>>>>> The build will create debuginfo/diz files as requested -
>>>>>>>>> that's
>>>>>>>>> fine.
>>>>>>>>>
>>>>>>>>> What I'm unclear about is where those files should actually
>>>>>>>>> appear in
>>>>>>>>> our build artifacts, specifically the jdk/jre images that are
>>>>>>>>> created.
>>>>>>>>>
>>>>>>>>> Internally when we create binary images RE strips all the
>>>>>>>>> debuginfo/diz files out.
>>>>>>>>
>>>>>>>>
>>>>>>>> Not quite. The RE makefiles will put debug info files into a
>>>>>>>> separate debuginfo.zip file. This includes .debuginfo, .diz,
>>>>>>>> .map and .pdb files. For the regular bundles, the RE makefiles
>>>>>>>> will not include debug info files. Not quite the same as
>>>>>>>> stripping them out.
>>>>>>>
>>>>>>>
>>>>>>> Not trying to split hairs but if the image contains them and
>>>>>>> the
>>>>>>> RE
>>>>>>> bundle of the image does not, then the RE process is stripping
>>>>>>> them out.
>>>>>>
>>>>>>
>>>>>> The above is implying that there is a single RE bundle and that
>>>>>> hasn't
>>>>>> been true for the tar-style universe for quite a long time; the
>>>>>> demo
>>>>>> bundle was added by around JDK8-B20 or so. For the package-style
>>>>>> universe, I don't think that has ever been true.
>>>>>>
>>>>>> RE's makefiles take the image generated by the build and package
>>>>>> up the
>>>>>> various files into two or more bundles. I believe each platform
>>>>>> has two
>>>>>> styles of bundles. For Solaris, SVR4 style packages and tar/zip
>>>>>> bundles.
>>>>>> For Windows, an EXE installer and tar/zip bundles. For Linux RPM
>>>>>> packages
>>>>>> and tar/ZIP bundles.
>>>>>>
>>>>>> So I'll stick with my assertion that the RE process is placing
>>>>>> debug info
>>>>>> files into a separate bundle and is not stripping them out.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>> But the build itself seems to treat them in an ad-hoc manner:
>>>>>>>>>
>>>>>>>>> - The new build deliberately excludes debuginfo/diz files
>>>>>>>>> associated
>>>>>>>>> with binaries, but will include any related to libraries (via
>>>>>>>>> generic
>>>>>>>>> copying routine). (It's obvious from the comments related to
>>>>>>>>> this that
>>>>>>>>> there is some puzzlement as to this reasoning.)
>>>>>>>>
>>>>>>>>
>>>>>>>> Sorry I haven't looked at the new build system.
>>>>>>>
>>>>>>>
>>>>>>> It tries to emulate the old build.
>>>>>>>
>>>>>>>>> - The old build also tries to exclude the files associated
>>>>>>>>> with
>>>>>>>>> binaries, but only handles .debuginfo not .diz :(
>>>>>>>>
>>>>>>>>
>>>>>>>> During one round in the FDS project, I included
>>>>>>>> .debuginfo/.diz
>>>>>>>> files
>>>>>>>> right next to some of the binaries. Apparently that's not
>>>>>>>> allowed
>>>>>>>> without
>>>>>>>> explicit permission. The contents of the "bin" directory are
>>>>>>>> controlled
>>>>>>>> and there are tests to verify those contents.
>>>>>>>>
>>>>>>>> I changed the Makefiles for the few binaries that support FDS
>>>>>>>> to
>>>>>>>> not
>>>>>>>> install the debug info files with the binaries, but they are
>>>>>>>> left in
>>>>>>>> the normal build artifacts location if someone has the need to
>>>>>>>> use
>>>>>>>> them. Of course, I've had queries for debug info files for the
>>>>>>>> binaries
>>>>>>>> to be included in the debuginfo.zip bundles. I'm not planning
>>>>>>>> to
>>>>>>>> fight
>>>>>>>> that battle.
>>>>>>>
>>>>>>>
>>>>>>> Okay that explains the special handling for binaries. So how do
>>>>>>> these
>>>>>>> then get into the debuginfo.zip? Or don't they?
>>>>>>
>>>>>>
>>>>>> They don't. They are only available via the build artifacts that
>>>>>> are archived.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>> Those associated with libraries just seem to get copied if
>>>>>>>>> they
>>>>>>>>> happen
>>>>>>>>> to be there
>>>>>>>>
>>>>>>>>
>>>>>>>> All of the logic that copies debug info files to the image
>>>>>>>> should do
>>>>>>>> so if they happen to be there. Not all component support FDS
>>>>>>>> now
>>>>>>>> or
>>>>>>>> in the future so the logic needs to adapt to what is built.
>>>>>>>>
>>>>>>>>
>>>>>>>>> As I said this all seems very ad-hoc to me. I would expect to
>>>>>>>>> see no
>>>>>>>>> debuginfo/diz files in a created image by default, and have a
>>>>>>>>> separate
>>>>>>>>> target that would produce a tar file of all the debuginfo/diz
>>>>>>>>> files
>>>>>>>>> ready to overlay on an existing image.
>>>>>>>>
>>>>>>>>
>>>>>>>> Close. The image has to have the debug info or .diz files in
>>>>>>>> order for
>>>>>>>> the RE makefiles to generate the debuginfo.zip files. It is
>>>>>>>> the
>>>>>>>> RE
>>>>>>>> makefiles that handle the packaging. Just like the demos are
>>>>>>>> always
>>>>>>>> built and it is the RE makefiles that put them in a separate
>>>>>>>> bundle.
>>>>>>>
>>>>>>>
>>>>>>> So therein lies the problem. RE lies outside of the OpenJDK
>>>>>>> build
>>>>>>> system. If the RE Makefiles can copy an image they can copy the
>>>>>>> debuginfo files too. I'd much rather see no debuginfo files in
>>>>>>> the
>>>>>>> built images (unless requested) and a distinct debuginfo
>>>>>>> related
>>>>>>> target (that RE could use if they choose). That would seem to
>>>>>>> be
>>>>>>> simpler all round.
>>>>>>
>>>>>>
>>>>>> My understanding is that "the image" is supposed to be the
>>>>>> complete
>>>>>> image if every bundle is extracted. I believe that is how the
>>>>>> packages
>>>>>> are sanity checked to be complete. RE's Makefiles are meant to
>>>>>> process
>>>>>> "the image" into bundles. They aren't meant to add files of
>>>>>> their
>>>>>> own.
>>>>>>
>>>>>> If you are planning to change the meaning of "the image" or if
>>>>>> you
>>>>>> are planning to put files into bundles that are not in "the
>>>>>> image",
>>>>>> then you'll need to coordinate with RE and whoever does the
>>>>>> package
>>>>>> inventory testing.
>>>>>>
>>>>>> Also, if you change the way this works, keep in mind that some
>>>>>> of
>>>>>> the
>>>>>> Makefiles generate their objects into a temporary directory and
>>>>>> then
>>>>>> copy things from that temporary directory to the image. Other
>>>>>> Makefiles
>>>>>> generate their objects directly into the image. For the latter,
>>>>>> you'll
>>>>>> have to move the debug info file(s) out of the image into some
>>>>>> other
>>>>>> place where it can be picked up by whatever bundling process you
>>>>>> create
>>>>>> to make the debuginfo.zip file.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>>> Thoughts/comments?
>>>>>>>>>
>>>>>>>>> I have to deal with this for the SE Profile work, where we
>>>>>>>>> will
>>>>>>>>> not
>>>>>>>>> want these files present in any of the images.
>>>>>>>>
>>>>>>>>
>>>>>>>> It should be fine if the Embedded profile does not generate
>>>>>>>> debug info
>>>>>>>> files or if the Embedded profile generates debug info, but
>>>>>>>> does
>>>>>>>> not
>>>>>>>> copy the debug info files into the image area. The RE
>>>>>>>> makefiles
>>>>>>>> will
>>>>>>>> handle the presence or absence of debug info files.
>>>>>>>
>>>>>>>
>>>>>>> These are not embedded profiles, these are just profiles - it
>>>>>>> is
>>>>>>> all
>>>>>>> to be part of SE in Java 8. The build is supposed to create an
>>>>>>> image
>>>>>>> corresponding to each profile. I can certainly specialise the
>>>>>>> handling
>>>>>>> of the debuginfo files for the profiles, but it seemed to me
>>>>>>> that
>>>>>>> the
>>>>>>> overall debuginfo-in-image story was rather unclear.
>>>>>>
>>>>>>
>>>>>> I'm sorry you think that the debuginfo-in-image story is
>>>>>> unclear.
>>>>>> I thought I did a pretty good job integrating FDS into the whole
>>>>>> OpenJDK build process somewhat seamlessly. I guess not.
>>>>>>
>>>>>> The key things to remember in all this is that debug info files
>>>>>> (like the demos) have to end up in a separate bundle or bundles.
>>>>>> So if you change the way this is all implemented, you still have
>>>>>> to follow the separate bundle rule.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> However, the non-Embedded profile needs to continue to
>>>>>>>> generate
>>>>>>>> and
>>>>>>>> copy debug info files for those components that already
>>>>>>>> support
>>>>>>>> FDS.
>>>>>>>> If you change the non-Embedded logic to not copy the debug
>>>>>>>> info
>>>>>>>> files
>>>>>>>> into the image area, the RE debuginfo.zip bundle will not be
>>>>>>>> created
>>>>>>>> and that will break FDS.
>>>>>>>
>>>>>>>
>>>>>>> RE processes will have to change to handle profiles in any
>>>>>>> case.
>>>>>>> I
>>>>>>> think they'd rather get a debuginfo.zip out of the build than
>>>>>>> having
>>>>>>> to create it themselves.
>>>>>>
>>>>>>
>>>>>> Interesting and not something that I have ever heard. I got the
>>>>>> impression that RE needed to have control over what was bundled
>>>>>> and how it was bundled. It's kind of hard to certify/vouch for a
>>>>>> process when you don't do it yourself, but that's not really my
>>>>>> call.
>>>>>>
>>>>>> Good luck with the tweaking of the build and packaging
>>>>>> processes.
>>>>>> Just remember: There be dragons here!
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> David
>>>>>>>
>>>>>>>>
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> David
>>>>
>>>
>>
>