Potential sensitive information leak through JVM crash logs

[Kevin Walls]

Hi,

8266967: debug.cpp utility find() should print Java Object fields.

This was providing tooling which brought product builds in line with debug builds for object printing.  So what we now see in hs_err was previously limited to debug builds.  (Your customers probably weren't running debug builds of course (and that's fine!), so the object details were previously hidden.)

Escaping Strings in the printer seems worth considering anyway, as a Java String could contain control characters, and yes that will help those who want to process the file.  We can think further if anything else needs doing.

Thanks
Kevin


-----Original Message-----
From: Alexandr Miloslavskiy <alexandr.miloslavskiy at syntevo.com> 
Sent: 24 December 2022 13:15
To: hotspot-dev at openjdk.java.net
Cc: kevinw at openjdk.org; Marc Strapetz <marc.strapetz at syntevo.com>
Subject: Potential sensitive information leak through JVM crash logs

Hello,

For a quick understanding, consider the following potential hs_err_pid*.log file:

----------------------------------------
...
stack at sp + 2 slots: 0x00000000c0401098 is an oop: org.x.y.z {0x00000000c0401098} - klass: 'org/x/y/z'
  - ---- fields (total size 141 words):
  ...
  - 'x' 'Lorg/a/b/c;' @184  NULL (0)
  - 'myPasswordString' 'Ljava/lang/String;' @188  "My password:
correct-horse-battery-staple"{0x00000000c0b01ef0} (c0b01ef0)
  - 'y' 'Ljava/lang/String;' @192  "foobar"
...
stack at sp + 3 slots: 0x00007fb06effd7c0 is pointing into the stack for
thread: 0x00007fb0680240f0
...
----------------------------------------

Here, crash log accidentally exposes a sensitive bit of user's information (the password).

In our product, when JVM crashes, we ask user to send a crash report.
But users shouldn't be worried that said report may contain something really sensitive.

For this reason, since we upgraded to JDK that has this feature, we decided to strip these object dumps from the crash log before sending it.

So far we decided to strip all lines starting from `- ---- fields` until one of
1) `stack at sp + 3 slots` where `+ 3` is exactly 1 more than
    previously seen `stack at sp + 2 slots`
2) a line that starts with `R??=` where ?? is any two chars,
    to handle registers
3) next section in crash log such as
    'Registers:' or
    '---------------  P R O C E S S  ---------------'.

It can be seen how it's not convenient and not very reliable to strip due to parsing issues. Especially when a String contains embedded \n, which prevents the easy solution to strip all lines beginning with ` - ` (see example crash log above).

We propose the following possible solutions:
1) Add some unique marker to begin and end of each object dump, so that everything between the markers can be stripped.
2) Otherwise, ensure that Strings don't span to next line when there's an embedded \n.

For your reference, a typical crash log can be seen in [1], where object dumps are seen in both sections 'Register to memory mapping:'
'Stack slot to memory mapping:'

According to my research, the feature of dumping objects in crash logs was (accidentally?) introduced in commit:
   5e557d86 by Kevin Walls, 2021-06-08 01:26:13
   8266967: debug.cpp utility find() should print Java Object fields.

[1] https://bugs.openjdk.org/browse/JDK-8284988