debuginfo/diz files in built images

[David Holmes]

Andrew,

On 6/09/2012 1:00 AM, Andrew Hughes wrote:
> ----- Original Message -----
>> Interesting point.
>>
>> Should .diz files be removed when packaging JDK/JRE for Linux ?
>>
>
> Specifically talking about packaging, distro build systems already handle
> stripping, etc. themselves so we've been setting STRIP_POLICY=no_strip.
> However, this still creates the diz files, even though the debuginfo is
> still in the binaries.
>
> I have a couple of changesets to fix this, one for HotSpot&  one for JDK,
> basically making diz production conditional on STRIP_POLICY not being set
> to no_strip.  I'll try and upstream them, if that sounds the correct thing
> to do.

We already have ENABLE_FULL_DEBUG_SYMBOLS to control whether we should 
try to produce the debuginfo files. And ZIP_DEBUGINFO_FILES to determine 
whether we then zip it to produce the .diz file.

My query is whether those files should then end up in the jre and jdk 
images that the build produces. I had thought no because the Oracle JDKs 
don't contain them and our RE processes have to exclude them when 
creating our bundles. Further, there is a bug in the build logic that 
causes a JRE to get diz files for libraries that aren't there. I thought 
a separate debuginfo.zip target might be better and leave them out of 
the images.

Do other distros "package" things based on the jre and jdk images 
produced by the build or do they use the raw build artefacts directly?

Thanks,
David


>> 2012/9/5 David Holmes<david.holmes at oracle.com>:
>>> Dan,
>>>
>>> Sorry, not meaning to ruffle your feathers.
>>>
>>> The reason I say the diz-in-image story is unclear is because there
>>> are no
>>> explicit rules that indicates that diz files should end up in a JDK
>>> or JRE
>>> image. I can accept that everything should go in a full JDK image.
>>> But a JRE
>>> is a JDK with a bunch of things left out and some of those things
>>> are
>>> libraries. But because diz files are not mentioned explicitly in
>>> the rules
>>> for making images we actually get the diz files for the libraries
>>> that were
>>> removed! They are simply copied from the build artifacts location
>>> to the jre
>>> image location specifically libattach.diz and libsaproc.diz. Maybe
>>> that was
>>> a simple oversight.
>>>
>>> Additionally it is unclear to me why we have diz files for some
>>> libraries
>>> and not others?
>>>
>>> Anyway as I said this will need to be modified for the profile
>>> work, in
>>> consultation with all the interested parties.
>>>
>>> Thanks,
>>> David
>>>
>>> On 4/09/2012 11:51 PM, Daniel D. Daugherty wrote:
>>>>
>>>> On 9/2/12 9:37 PM, David Holmes wrote:
>>>>>
>>>>> Hi Dan,
>>>>>
>>>>> On 3/09/2012 12:47 PM, Daniel D. Daugherty wrote:
>>>>>>
>>>>>> On 9/2/12 7:26 PM, David Holmes wrote:
>>>>>>>
>>>>>>> The build will create debuginfo/diz files as requested - that's
>>>>>>> fine.
>>>>>>>
>>>>>>> What I'm unclear about is where those files should actually
>>>>>>> appear in
>>>>>>> our build artifacts, specifically the jdk/jre images that are
>>>>>>> created.
>>>>>>>
>>>>>>> Internally when we create binary images RE strips all the
>>>>>>> debuginfo/diz files out.
>>>>>>
>>>>>>
>>>>>> Not quite. The RE makefiles will put debug info files into a
>>>>>> separate debuginfo.zip file. This includes .debuginfo, .diz,
>>>>>> .map and .pdb files. For the regular bundles, the RE makefiles
>>>>>> will not include debug info files. Not quite the same as
>>>>>> stripping them out.
>>>>>
>>>>>
>>>>> Not trying to split hairs but if the image contains them and the
>>>>> RE
>>>>> bundle of the image does not, then the RE process is stripping
>>>>> them out.
>>>>
>>>>
>>>> The above is implying that there is a single RE bundle and that
>>>> hasn't
>>>> been true for the tar-style universe for quite a long time; the
>>>> demo
>>>> bundle was added by around JDK8-B20 or so. For the package-style
>>>> universe, I don't think that has ever been true.
>>>>
>>>> RE's makefiles take the image generated by the build and package
>>>> up the
>>>> various files into two or more bundles. I believe each platform
>>>> has two
>>>> styles of bundles. For Solaris, SVR4 style packages and tar/zip
>>>> bundles.
>>>> For Windows, an EXE installer and tar/zip bundles. For Linux RPM
>>>> packages
>>>> and tar/ZIP bundles.
>>>>
>>>> So I'll stick with my assertion that the RE process is placing
>>>> debug info
>>>> files into a separate bundle and is not stripping them out.
>>>>
>>>>
>>>>
>>>>>
>>>>>>> But the build itself seems to treat them in an ad-hoc manner:
>>>>>>>
>>>>>>> - The new build deliberately excludes debuginfo/diz files
>>>>>>> associated
>>>>>>> with binaries, but will include any related to libraries (via
>>>>>>> generic
>>>>>>> copying routine). (It's obvious from the comments related to
>>>>>>> this that
>>>>>>> there is some puzzlement as to this reasoning.)
>>>>>>
>>>>>>
>>>>>> Sorry I haven't looked at the new build system.
>>>>>
>>>>>
>>>>> It tries to emulate the old build.
>>>>>
>>>>>>> - The old build also tries to exclude the files associated with
>>>>>>> binaries, but only handles .debuginfo not .diz :(
>>>>>>
>>>>>>
>>>>>> During one round in the FDS project, I included .debuginfo/.diz
>>>>>> files
>>>>>> right next to some of the binaries. Apparently that's not
>>>>>> allowed
>>>>>> without
>>>>>> explicit permission. The contents of the "bin" directory are
>>>>>> controlled
>>>>>> and there are tests to verify those contents.
>>>>>>
>>>>>> I changed the Makefiles for the few binaries that support FDS to
>>>>>> not
>>>>>> install the debug info files with the binaries, but they are
>>>>>> left in
>>>>>> the normal build artifacts location if someone has the need to
>>>>>> use
>>>>>> them. Of course, I've had queries for debug info files for the
>>>>>> binaries
>>>>>> to be included in the debuginfo.zip bundles. I'm not planning to
>>>>>> fight
>>>>>> that battle.
>>>>>
>>>>>
>>>>> Okay that explains the special handling for binaries. So how do
>>>>> these
>>>>> then get into the debuginfo.zip? Or don't they?
>>>>
>>>>
>>>> They don't. They are only available via the build artifacts that
>>>> are archived.
>>>>
>>>>
>>>>
>>>>>
>>>>>>> Those associated with libraries just seem to get copied if they
>>>>>>> happen
>>>>>>> to be there
>>>>>>
>>>>>>
>>>>>> All of the logic that copies debug info files to the image
>>>>>> should do
>>>>>> so if they happen to be there. Not all component support FDS now
>>>>>> or
>>>>>> in the future so the logic needs to adapt to what is built.
>>>>>>
>>>>>>
>>>>>>> As I said this all seems very ad-hoc to me. I would expect to
>>>>>>> see no
>>>>>>> debuginfo/diz files in a created image by default, and have a
>>>>>>> separate
>>>>>>> target that would produce a tar file of all the debuginfo/diz
>>>>>>> files
>>>>>>> ready to overlay on an existing image.
>>>>>>
>>>>>>
>>>>>> Close. The image has to have the debug info or .diz files in
>>>>>> order for
>>>>>> the RE makefiles to generate the debuginfo.zip files. It is the
>>>>>> RE
>>>>>> makefiles that handle the packaging. Just like the demos are
>>>>>> always
>>>>>> built and it is the RE makefiles that put them in a separate
>>>>>> bundle.
>>>>>
>>>>>
>>>>> So therein lies the problem. RE lies outside of the OpenJDK build
>>>>> system. If the RE Makefiles can copy an image they can copy the
>>>>> debuginfo files too. I'd much rather see no debuginfo files in
>>>>> the
>>>>> built images (unless requested) and a distinct debuginfo related
>>>>> target (that RE could use if they choose). That would seem to be
>>>>> simpler all round.
>>>>
>>>>
>>>> My understanding is that "the image" is supposed to be the
>>>> complete
>>>> image if every bundle is extracted. I believe that is how the
>>>> packages
>>>> are sanity checked to be complete. RE's Makefiles are meant to
>>>> process
>>>> "the image" into bundles. They aren't meant to add files of their
>>>> own.
>>>>
>>>> If you are planning to change the meaning of "the image" or if you
>>>> are planning to put files into bundles that are not in "the
>>>> image",
>>>> then you'll need to coordinate with RE and whoever does the
>>>> package
>>>> inventory testing.
>>>>
>>>> Also, if you change the way this works, keep in mind that some of
>>>> the
>>>> Makefiles generate their objects into a temporary directory and
>>>> then
>>>> copy things from that temporary directory to the image. Other
>>>> Makefiles
>>>> generate their objects directly into the image. For the latter,
>>>> you'll
>>>> have to move the debug info file(s) out of the image into some
>>>> other
>>>> place where it can be picked up by whatever bundling process you
>>>> create
>>>> to make the debuginfo.zip file.
>>>>
>>>>
>>>>
>>>>>
>>>>>>> Thoughts/comments?
>>>>>>>
>>>>>>> I have to deal with this for the SE Profile work, where we will
>>>>>>> not
>>>>>>> want these files present in any of the images.
>>>>>>
>>>>>>
>>>>>> It should be fine if the Embedded profile does not generate
>>>>>> debug info
>>>>>> files or if the Embedded profile generates debug info, but does
>>>>>> not
>>>>>> copy the debug info files into the image area. The RE makefiles
>>>>>> will
>>>>>> handle the presence or absence of debug info files.
>>>>>
>>>>>
>>>>> These are not embedded profiles, these are just profiles - it is
>>>>> all
>>>>> to be part of SE in Java 8. The build is supposed to create an
>>>>> image
>>>>> corresponding to each profile. I can certainly specialise the
>>>>> handling
>>>>> of the debuginfo files for the profiles, but it seemed to me that
>>>>> the
>>>>> overall debuginfo-in-image story was rather unclear.
>>>>
>>>>
>>>> I'm sorry you think that the debuginfo-in-image story is unclear.
>>>> I thought I did a pretty good job integrating FDS into the whole
>>>> OpenJDK build process somewhat seamlessly. I guess not.
>>>>
>>>> The key things to remember in all this is that debug info files
>>>> (like the demos) have to end up in a separate bundle or bundles.
>>>> So if you change the way this is all implemented, you still have
>>>> to follow the separate bundle rule.
>>>>
>>>>
>>>>
>>>>>
>>>>>> However, the non-Embedded profile needs to continue to generate
>>>>>> and
>>>>>> copy debug info files for those components that already support
>>>>>> FDS.
>>>>>> If you change the non-Embedded logic to not copy the debug info
>>>>>> files
>>>>>> into the image area, the RE debuginfo.zip bundle will not be
>>>>>> created
>>>>>> and that will break FDS.
>>>>>
>>>>>
>>>>> RE processes will have to change to handle profiles in any case.
>>>>> I
>>>>> think they'd rather get a debuginfo.zip out of the build than
>>>>> having
>>>>> to create it themselves.
>>>>
>>>>
>>>> Interesting and not something that I have ever heard. I got the
>>>> impression that RE needed to have control over what was bundled
>>>> and how it was bundled. It's kind of hard to certify/vouch for a
>>>> process when you don't do it yourself, but that's not really my
>>>> call.
>>>>
>>>> Good luck with the tweaking of the build and packaging processes.
>>>> Just remember: There be dragons here!
>>>>
>>>> Dan
>>>>
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> David
>>
>