debuginfo/diz files in built images

[Andrew Hughes]

----- Original Message -----
> Interesting point.
> 
> Should .diz files be removed when packaging JDK/JRE for Linux ?
> 

Specifically talking about packaging, distro build systems already handle
stripping, etc. themselves so we've been setting STRIP_POLICY=no_strip.
However, this still creates the diz files, even though the debuginfo is
still in the binaries.

I have a couple of changesets to fix this, one for HotSpot & one for JDK,
basically making diz production conditional on STRIP_POLICY not being set
to no_strip.  I'll try and upstream them, if that sounds the correct thing
to do.

> 2012/9/5 David Holmes <david.holmes at oracle.com>:
> > Dan,
> >
> > Sorry, not meaning to ruffle your feathers.
> >
> > The reason I say the diz-in-image story is unclear is because there
> > are no
> > explicit rules that indicates that diz files should end up in a JDK
> > or JRE
> > image. I can accept that everything should go in a full JDK image.
> > But a JRE
> > is a JDK with a bunch of things left out and some of those things
> > are
> > libraries. But because diz files are not mentioned explicitly in
> > the rules
> > for making images we actually get the diz files for the libraries
> > that were
> > removed! They are simply copied from the build artifacts location
> > to the jre
> > image location specifically libattach.diz and libsaproc.diz. Maybe
> > that was
> > a simple oversight.
> >
> > Additionally it is unclear to me why we have diz files for some
> > libraries
> > and not others?
> >
> > Anyway as I said this will need to be modified for the profile
> > work, in
> > consultation with all the interested parties.
> >
> > Thanks,
> > David
> >
> > On 4/09/2012 11:51 PM, Daniel D. Daugherty wrote:
> >>
> >> On 9/2/12 9:37 PM, David Holmes wrote:
> >>>
> >>> Hi Dan,
> >>>
> >>> On 3/09/2012 12:47 PM, Daniel D. Daugherty wrote:
> >>>>
> >>>> On 9/2/12 7:26 PM, David Holmes wrote:
> >>>>>
> >>>>> The build will create debuginfo/diz files as requested - that's
> >>>>> fine.
> >>>>>
> >>>>> What I'm unclear about is where those files should actually
> >>>>> appear in
> >>>>> our build artifacts, specifically the jdk/jre images that are
> >>>>> created.
> >>>>>
> >>>>> Internally when we create binary images RE strips all the
> >>>>> debuginfo/diz files out.
> >>>>
> >>>>
> >>>> Not quite. The RE makefiles will put debug info files into a
> >>>> separate debuginfo.zip file. This includes .debuginfo, .diz,
> >>>> .map and .pdb files. For the regular bundles, the RE makefiles
> >>>> will not include debug info files. Not quite the same as
> >>>> stripping them out.
> >>>
> >>>
> >>> Not trying to split hairs but if the image contains them and the
> >>> RE
> >>> bundle of the image does not, then the RE process is stripping
> >>> them out.
> >>
> >>
> >> The above is implying that there is a single RE bundle and that
> >> hasn't
> >> been true for the tar-style universe for quite a long time; the
> >> demo
> >> bundle was added by around JDK8-B20 or so. For the package-style
> >> universe, I don't think that has ever been true.
> >>
> >> RE's makefiles take the image generated by the build and package
> >> up the
> >> various files into two or more bundles. I believe each platform
> >> has two
> >> styles of bundles. For Solaris, SVR4 style packages and tar/zip
> >> bundles.
> >> For Windows, an EXE installer and tar/zip bundles. For Linux RPM
> >> packages
> >> and tar/ZIP bundles.
> >>
> >> So I'll stick with my assertion that the RE process is placing
> >> debug info
> >> files into a separate bundle and is not stripping them out.
> >>
> >>
> >>
> >>>
> >>>>> But the build itself seems to treat them in an ad-hoc manner:
> >>>>>
> >>>>> - The new build deliberately excludes debuginfo/diz files
> >>>>> associated
> >>>>> with binaries, but will include any related to libraries (via
> >>>>> generic
> >>>>> copying routine). (It's obvious from the comments related to
> >>>>> this that
> >>>>> there is some puzzlement as to this reasoning.)
> >>>>
> >>>>
> >>>> Sorry I haven't looked at the new build system.
> >>>
> >>>
> >>> It tries to emulate the old build.
> >>>
> >>>>> - The old build also tries to exclude the files associated with
> >>>>> binaries, but only handles .debuginfo not .diz :(
> >>>>
> >>>>
> >>>> During one round in the FDS project, I included .debuginfo/.diz
> >>>> files
> >>>> right next to some of the binaries. Apparently that's not
> >>>> allowed
> >>>> without
> >>>> explicit permission. The contents of the "bin" directory are
> >>>> controlled
> >>>> and there are tests to verify those contents.
> >>>>
> >>>> I changed the Makefiles for the few binaries that support FDS to
> >>>> not
> >>>> install the debug info files with the binaries, but they are
> >>>> left in
> >>>> the normal build artifacts location if someone has the need to
> >>>> use
> >>>> them. Of course, I've had queries for debug info files for the
> >>>> binaries
> >>>> to be included in the debuginfo.zip bundles. I'm not planning to
> >>>> fight
> >>>> that battle.
> >>>
> >>>
> >>> Okay that explains the special handling for binaries. So how do
> >>> these
> >>> then get into the debuginfo.zip? Or don't they?
> >>
> >>
> >> They don't. They are only available via the build artifacts that
> >> are archived.
> >>
> >>
> >>
> >>>
> >>>>> Those associated with libraries just seem to get copied if they
> >>>>> happen
> >>>>> to be there
> >>>>
> >>>>
> >>>> All of the logic that copies debug info files to the image
> >>>> should do
> >>>> so if they happen to be there. Not all component support FDS now
> >>>> or
> >>>> in the future so the logic needs to adapt to what is built.
> >>>>
> >>>>
> >>>>> As I said this all seems very ad-hoc to me. I would expect to
> >>>>> see no
> >>>>> debuginfo/diz files in a created image by default, and have a
> >>>>> separate
> >>>>> target that would produce a tar file of all the debuginfo/diz
> >>>>> files
> >>>>> ready to overlay on an existing image.
> >>>>
> >>>>
> >>>> Close. The image has to have the debug info or .diz files in
> >>>> order for
> >>>> the RE makefiles to generate the debuginfo.zip files. It is the
> >>>> RE
> >>>> makefiles that handle the packaging. Just like the demos are
> >>>> always
> >>>> built and it is the RE makefiles that put them in a separate
> >>>> bundle.
> >>>
> >>>
> >>> So therein lies the problem. RE lies outside of the OpenJDK build
> >>> system. If the RE Makefiles can copy an image they can copy the
> >>> debuginfo files too. I'd much rather see no debuginfo files in
> >>> the
> >>> built images (unless requested) and a distinct debuginfo related
> >>> target (that RE could use if they choose). That would seem to be
> >>> simpler all round.
> >>
> >>
> >> My understanding is that "the image" is supposed to be the
> >> complete
> >> image if every bundle is extracted. I believe that is how the
> >> packages
> >> are sanity checked to be complete. RE's Makefiles are meant to
> >> process
> >> "the image" into bundles. They aren't meant to add files of their
> >> own.
> >>
> >> If you are planning to change the meaning of "the image" or if you
> >> are planning to put files into bundles that are not in "the
> >> image",
> >> then you'll need to coordinate with RE and whoever does the
> >> package
> >> inventory testing.
> >>
> >> Also, if you change the way this works, keep in mind that some of
> >> the
> >> Makefiles generate their objects into a temporary directory and
> >> then
> >> copy things from that temporary directory to the image. Other
> >> Makefiles
> >> generate their objects directly into the image. For the latter,
> >> you'll
> >> have to move the debug info file(s) out of the image into some
> >> other
> >> place where it can be picked up by whatever bundling process you
> >> create
> >> to make the debuginfo.zip file.
> >>
> >>
> >>
> >>>
> >>>>> Thoughts/comments?
> >>>>>
> >>>>> I have to deal with this for the SE Profile work, where we will
> >>>>> not
> >>>>> want these files present in any of the images.
> >>>>
> >>>>
> >>>> It should be fine if the Embedded profile does not generate
> >>>> debug info
> >>>> files or if the Embedded profile generates debug info, but does
> >>>> not
> >>>> copy the debug info files into the image area. The RE makefiles
> >>>> will
> >>>> handle the presence or absence of debug info files.
> >>>
> >>>
> >>> These are not embedded profiles, these are just profiles - it is
> >>> all
> >>> to be part of SE in Java 8. The build is supposed to create an
> >>> image
> >>> corresponding to each profile. I can certainly specialise the
> >>> handling
> >>> of the debuginfo files for the profiles, but it seemed to me that
> >>> the
> >>> overall debuginfo-in-image story was rather unclear.
> >>
> >>
> >> I'm sorry you think that the debuginfo-in-image story is unclear.
> >> I thought I did a pretty good job integrating FDS into the whole
> >> OpenJDK build process somewhat seamlessly. I guess not.
> >>
> >> The key things to remember in all this is that debug info files
> >> (like the demos) have to end up in a separate bundle or bundles.
> >> So if you change the way this is all implemented, you still have
> >> to follow the separate bundle rule.
> >>
> >>
> >>
> >>>
> >>>> However, the non-Embedded profile needs to continue to generate
> >>>> and
> >>>> copy debug info files for those components that already support
> >>>> FDS.
> >>>> If you change the non-Embedded logic to not copy the debug info
> >>>> files
> >>>> into the image area, the RE debuginfo.zip bundle will not be
> >>>> created
> >>>> and that will break FDS.
> >>>
> >>>
> >>> RE processes will have to change to handle profiles in any case.
> >>> I
> >>> think they'd rather get a debuginfo.zip out of the build than
> >>> having
> >>> to create it themselves.
> >>
> >>
> >> Interesting and not something that I have ever heard. I got the
> >> impression that RE needed to have control over what was bundled
> >> and how it was bundled. It's kind of hard to certify/vouch for a
> >> process when you don't do it yourself, but that's not really my
> >> call.
> >>
> >> Good luck with the tweaking of the build and packaging processes.
> >> Just remember: There be dragons here!
> >>
> >> Dan
> >>
> >>
> >>
> >>>
> >>> Thanks,
> >>> David
> >>>
> >>>>
> >>>> Dan
> >>>>
> >>>>
> >>>>>
> >>>>> David
> 

-- 
Andrew :)

Free Java Software Engineer
Red Hat, Inc. (http://www.redhat.com)

PGP Key: 248BDC07 (https://keys.indymedia.org/)
Fingerprint = EC5A 1F5E C0AD 1D15 8F1F  8F91 3B96 A578 248B DC07