RFR: 8237192: Generate stripped/public pdbs on Windows for jdk images

Wed Feb 12 13:26:00 UTC 2020

Hi Magnus, Erik,

I started off with Matthias’ patch and tried to address your concerns. Especially I explored adding the stripped pdbs to the jmods as well. Here is what I came up with:
http://cr.openjdk.java.net/~clanger/webrevs/8237192.0/

It’s a bit hacky in that it’ll copy support/modules_libs to support/modules_libs_stripped and fix the pdbs to ship in there. The same goes for modules_cmds. The problem with that approach is that probably not all dependencies are placed correctly and also there’s a bit more of duplication of binaries in the build directories. I think, it could be repaired eventually by refactoring, e.g. have a support/modules_dbgsymbols folder where the real debug symbol files get placed and used from there.

There’s also two additional caveats, one is that jimage.pdb and jpackage.pdb exist twice. Once for the libs and once for the launchers with the same name. This will cause failures when jlinking. I decided to keep the pdbs for the libs. And I also had to take care of the classlist generation, to have the resulting classlist placed in support/modules_libs_stripped as well.

I furthermore updated the naming of options and variables, hopefully to your like. And I made the debug output logInfo.

I tested (on Windows), both, with --enable-public-debug-symbols and without. Without the option, everything seems as before. With the option enabled, the stripped debug symbols will be installed in the bundles and also in the jmods. 😊

Please let me know what you think.

Thanks & Best regards
Christoph

From: Magnus Ihse Bursie <magnus.ihse.bursie at oracle.com>
Sent: Freitag, 7. Februar 2020 14:09
To: Baesken, Matthias <matthias.baesken at sap.com>; Erik Joelsson <erik.joelsson at oracle.com>; Langer, Christoph <christoph.langer at sap.com>; David Holmes <david.holmes at oracle.com>; 'build-dev at openjdk.java.net' <build-dev at openjdk.java.net>; 'hotspot-dev at openjdk.java.net' <hotspot-dev at openjdk.java.net>
Subject: Re: RFR: 8237192: Generate stripped/public pdbs on Windows for jdk images

On 2020-02-07 09:50, Baesken, Matthias wrote:

Hello, here is  a slightly changed  new  webrev :

http://cr.openjdk.java.net/~mbaesken/webrevs/8237192.4/
In Bundles.gmk, this line:

$(ECHO) found stripped pdb file $$$${f}, we rename it to: $$$${f%stripped.pdb}pdb; \
It looks almost like left-over debug output. If you want to keep it, please rephrase to something more terse, that fits better with the existing style of build messages. Also, it should probably be on the LOG=info level, so add a $(LOG_INFO).

In NativeCompilation.gmk:
Why not just a simple,
          ifeq ($(ENABLE_STRIPPED_PDBS), true)
            $1_EXTRA_LDFLAGS += "-pdbstripped:$$($1_SYMBOLS_DIR)/$$($1_NOSUFFIX).stripped.pdb"
          endif
?
I see no reason to duplicate code.

In jdk-options.m4:

I'm not 100% sure about the name of the new option. --enable-stripped-pdb does not fully convey the fact that we do not strip the *existing* pdb:s, but instead also add a new type. Maybe --enable-bundle-stripped-pdb?

/Magnus

(adjusted $(JRE_STRIPPED_PDB_FILES)  in Bundles.gmk, that was in the wrong place )  .

I think it needs to be a separate option as it's all orthogonal to the

main debug symbols file creation.

Yes it is a separate option I agree that’s better .  One has to set  --enable-stripped-pdbs=yes  .

Best regards , Matthias

On 2020-02-06 04:48, Magnus Ihse Bursie wrote:

On 2020-02-06 12:36, Langer, Christoph wrote:

Hi,

let me chime in to this discussion at this point. So, first of all, Matthias,

thanks for your work so far.

Erik, I fully understand your points and I agree that it's probably a good

idea to refactor this whole process of creating these different types of

bundles.

Our idea is to provide functionality in the build system to add "public" or

stripped debug files to the delivery image of the JDK. This would provide

better information in case of crashes and would hence allow for better

supportability. That's something we'd probably want to enable in

SapMachine binary distributions.

I very much support the idea of using these stripped pdb files. It has

been a long standing issue in hotspot on Windows to not have access to

stacktraces.

So, can we get to add a configure option like the one proposed by

Matthias into the current code base?

The option should be turned off by default. If it is switched on, the

images/jdk folder in the build directory will have both, the *stripped.pdb

files and the standard *.pdb files. However, having *stripped.pdb files

around should not matter in terms of functionality (for developers and

testing) as they'd not be used in the presence of the "real" pdb files anyway.

The actual JDK bundle for delivery would then contain the *stripped.pdb

files, renamed to *.pdb. And the debug symbols bundle would have the full

*.pdb files only. Both could then be overlaid as needed.

I think you raised two concerns.

One is that you'd rather like to refactor bundling for several reasons. But I

guess, should you eventually get to your refactoring, it shouldn't be a

problem to take the functionality of this new option along.

The other was regarding JMODs. I believe it's correct, that JMODs have

never carried external debug information. For platforms other than MS

Windows that's actually not a big problem because debug information can be

internalized. And jlink has gotten several additions to set flags for stripping

that data to the right level. I assume if jlinked images on Windows should

ever be enabled to carry debug information, inclusion of external debug files

would have to be added to JMODs and jlink. But that's definitely out of scope

here.

The argument "jmods have never carried external debug information" just

doesn't apply here. Neither has the distribution bundles, for the exact

same reason. You really should not compare these new stripped pdb files

to the existing debug symbol files, they are different files with

different purposes. One is meant to be shipped to customers, the other

is not. You say you want to ship these new stripped pdb files with your

distribution so that customers have them present when they use your

distribution. If you then omit these new files from the jmods, any

customer created jlinked image will not have these new stripped pdb

files, which IMO is a very weird and unexpected behavior from a customer

point of view. Jlinking new images is an integral and promoted way of

using a JDK, so any mismatch between the original JDK distribution and

what you are able to jlink is a serious discrepancy.

So, what do you think? What speaks against adding this option (that is off

by default)?

My main objective is that you introduce further discrepancies between

the original distribution JDK image and what's possible to generate

using jlink from that distribution JDK image. My second objective is

that the already convoluted bundles creation logic becomes even more

convoluted. I believe there is a better possible solution in the build

but it will require a lot more work to figure out.

All that said, if you still wish to continue, I will stop standing in

the way.

While Erik will need to comment on this himself, from my POV this

looks okay. The conditions are:

* This is controlled by a separate option (or perhaps even better as a

new argument to --with-native-debug-symbols), so without this, nothing

will change.

I think it needs to be a separate option as it's all orthogonal to the

main debug symbols file creation.

* The code need to make sure to separate stripped.pdb and normal pdb

files, when enabled.

* And there must not be a heavy penalty in added code complexity.

/Erik

/Magnus

Thanks

Christoph

-----Original Message-----

From: build-dev<build-dev-bounces at openjdk.java.net><mailto:build-dev-bounces at openjdk.java.net>  On Behalf Of

Erik

Joelsson

Sent: Donnerstag, 23. Januar 2020 18:49

To: Baesken, Matthias<matthias.baesken at sap.com><mailto:matthias.baesken at sap.com>; David Holmes

<david.holmes at oracle.com><mailto:david.holmes at oracle.com>; 'build-dev at openjdk.java.net<mailto:build-dev at openjdk.java.net>' <build-

dev at openjdk.java.net<mailto:dev at openjdk.java.net>>; 'hotspot-dev at openjdk.java.net<mailto:hotspot-dev at openjdk.java.net>' <hotspot-

dev at openjdk.java.net<mailto:dev at openjdk.java.net>>

Subject: Re: RFR: 8237192: Generate stripped/public pdbs on Windows

for

jdk images

On 2020-01-23 00:03, Baesken, Matthias wrote:

Hi Erik,  yes true sorry for answering  your comments a bit late .

If a user runs jlink and includes all the jmods we ship with the JDK, the

result

should be essentially equivalent to the original JDK image. The way

the

stripped pdb files are included in the bundles sort of at the last

second of the build here breaks this property.

I think we should address this in a separate bug/CR .

Maybe. I realize that my proposal below is quite a big task. But on the

other hand, I don't think breaking the relationship between the jmods

and the distribution bundles is on the table really.

Looking for example  into a Linux  build, I see  a lot of debuginfo files  in

the

jdk image (more or less for every shared lib)  .

But when looking into the jmods  of that jdk image ,  no debuginfo files

are

in there ( I checked the java.base jmod).

So  putting  the  files with debug information into the jmods  seems to

be

something that was not done so far  cross platform  (or is there some

build

switch  for it that I did not check?) .

Maybe to keep the jmods as small as possible .

No, we do not put the debuginfo files in the jmods nor the bundles

because those are not intended to be shipped to customers. We are

currently overlaying them into images/jdk in the build output to make

them available for local debugging. (This is convoluted and I would very

much like to get away from this practice at some point so that there is

a 1-1 mapping between images/jdk and bundles/jdk*-bin.tar.gz.) The

stripped pdb files you are proposing are on the contrary intended for

shipping to customers (as I understand your proposal) so comparing

them

with the debuginfo files is not relevant.

Now if MS had been kind enough to define a separate file type for the

stripped pdbs, so that they could live alongside the full pdbs, we

wouldn't have this issue. The heart of the problem is that only one set

of files (either stripped or full) can be present and usable in

images/jdk at a time. We have 2 main uses for images/jdk.

1. Developer running and debugging locally

2. Serve as the source for generating the distribution bundles

We currently have one image serving both of these purposes, which is

already creating complicated and convoluted build steps. To properly

solve this I would argue for splitting these two apart into two

different images for the two different purposes. The build procedure

would then be, first build the images for distribution, only containing

what should go into each bundle. Then create the developer jdk image

by

copying files from the distribution images. On Windows, the first image

would contain the stripped pdbs and when building the second, those

would get overwritten with the full pdbs.

Now that I figured out a working model that would solve a bunch of

other

problems as well, I would love to implement it, but I doubt I will have

time in the near future.

/Erik

To properly implement this, care will need to be taken to juggle the

two

sets of pdb files around, making sure each build and test use case has

the correct one in place where and when it's needed. Quite possibly,

we

cannot cover all use cases with one build configuration. Developers

needing the full debug symbols when debugging locally would likely

need

to disable the stripped symbols so they get the full symbols

everywhere.

Possibly this would need to be the default for debug builds and

configurable for release builds.

  From my limited experience , the developers  do not work with the

bundles (that  would contain now after my patch the stripped pds)  but

with

a "normal" jdk image that  is created my make all.

Best regards, Matthias

This still does not address anything in my objection.

/Erik

On 2020-01-22 07:46, Baesken, Matthias wrote:

Hello,  here is an updated version  :

http://cr.openjdk.java.net/~mbaesken/webrevs/8237192.3/

this one supports a configure switch  "--enable-stripped-pdbs"    to

enable

the feature .

Best regards, Matthias

-----Original Message-----

From: Baesken, Matthias

Sent: Dienstag, 21. Januar 2020 11:03

To: 'David Holmes'<david.holmes at oracle.com><mailto:david.holmes at oracle.com>; 'build-

dev at openjdk.java.net<mailto:dev at openjdk.java.net>'<build-dev at openjdk.java.net><mailto:build-dev at openjdk.java.net>; 'hotspot-

dev at openjdk.java.net<mailto:dev at openjdk.java.net>'<hotspot-dev at openjdk.java.net><mailto:hotspot-dev at openjdk.java.net>

Subject: RE: RFR: 8237192: Generate stripped/public pdbs on

Windows

for

jdk images

Hi David ,  yes I think it makes sense to have a configure option for

this .

Not everyone would like to have a larger JDK (even it is only a bit

larger).