RFR: 8267666: Add option to jcmd GC.heap_dump to use existing file [v2]

[Ralf Schmelter]

On Fri, 28 May 2021 10:49:32 GMT, Anton Kozlov <akozlov at openjdk.org> wrote:

>> Please review a small change that adds an option to GC.heap_dump to use an existing file. 
>> 
>> The option is necessary if the target file is a predefined file-like object, like a named pipe. This opens up a lot of possibilities to process a heap dump without storing it to the file system first.
>> 
>> Reviews of the CSR linked to the bug would be appreciated as well.
>
> Anton Kozlov has updated the pull request incrementally with one additional commit since the last revision:
> 
>   Extend HeapDumpTest

I would rename the option to `-overwrite`, since it has a clear meaning (to replace the old content with new one). 

And to really implement the overwrite semantics, the file should be opened with `O_TRUNC`. Currently `-rewrite` just replaces the start of an already existing file with a the heap dump. If the original file was larger than the heap dump, we have trailing bytes, which would lead to errors when tools try to parse the dump. 

And on Unix it might be a good idea to use O_NOCTTY, so we don't accidentally assign a controlling tty when dumping to a console ('ve never seen actual problems omitting this, but it seems safe to add it).

Even with this changes you can still write to a fifo/tty on Unix or a named pipe on Windows, since O_TRUNC is ignored for these types of files.

And since you already created a CSR, I would propose to close [JDK-8263066](https://bugs.openjdk.java.net/browse/JDK-8263066) and instead try to get this into the mainline.

-------------

PR: https://git.openjdk.java.net/jdk/pull/4183