RFR: 8366154: Validate thread type requirements in debug commands

[David Holmes]

On Mon, 1 Sep 2025 11:36:21 GMT, Kerem Kat <krk at openjdk.org> wrote:

> Prevents segmentation faults during `gdb` sessions. The crashes were caused by the `ResourceMark` constructor being called on a native thread, which is not supported. This happened when invoking debug commands that require a `Thread` or `JavaThread` context from an incorrect thread type.
> 
> ### Solution
> 
> This change introduces `onThread()` and `onJavaThread()` helper methods to the `Command` class. These methods validate the thread context and ensure `ResourceMark` is only created when on a valid VM thread. All thread-dependent debug commands now use these guards to validate the context, printing a clear error and exiting gracefully upon failure.
> 
> ### Testing
> 
> Manually verified using `gdb` by calling the modified commands (`ps`, `universe`, `pns`, etc.) from different thread contexts (native, Java, and non-java threads) to ensure they fail gracefully with an error message instead of crashing the debug session.

We typically assume the user knows what they are doing at this level, but I suppose making it a little more robust doesn't hurt. I don't think we quite need everything though.

Thanks

src/hotspot/share/utilities/debug.cpp line 326:

> 324:     }
> 325: 
> 326:     if (!_has_rm) {

Is it even possible for this not to be false with correct usage?

src/hotspot/share/utilities/debug.cpp line 327:

> 325: 
> 326:     if (!_has_rm) {
> 327:       ::new (&_rm) ResourceMark();

There should be `#include <new>` to use global placement-new.

src/hotspot/share/utilities/debug.cpp line 341:

> 339:     }
> 340:     return true;
> 341:   }

I don't think we need this. The commands that require a JavaThread should be checking that directly themselves. Typically we assume/expect the person debugging to know what they are dealing with and use the appropriate commands.

-------------

Changes requested by dholmes (Reviewer).

PR Review: https://git.openjdk.org/jdk/pull/27033#pullrequestreview-3174512977
PR Review Comment: https://git.openjdk.org/jdk/pull/27033#discussion_r2314601972
PR Review Comment: https://git.openjdk.org/jdk/pull/27033#discussion_r2314595903
PR Review Comment: https://git.openjdk.org/jdk/pull/27033#discussion_r2314599669