RFR(L) 8237354: Add option to jcmd to write a gzipped heap dump

[serguei.spitsyn at oracle.com]

I also like this proposal.

Thanks,
Serguei


On 2/19/20 5:45 PM, Ioi Lam wrote:
> I like this proposal. It's simple, easily extensible and should work 
> on all platforms.
>
> Thanks
> - Ioi
>
> On 2/19/20 3:59 PM, Yasumasa Suenaga wrote:
>> Hi,
>>
>> Generally I agree with Ioi, but I think it is not a problem only for 
>> gzipped heap dump.
>>
>> For example, Compiler.codelist and Compiler.CodeHeap_Analytics might 
>> be large text.
>> In addition, some users want to redirect the result from jcmd to 
>> other command or log collector.
>>
>> So I think it would be better if jcmd provides stdout redurect option 
>> to all subocmmands. E.g.
>>
>>   $ jcmd <PID> GC.heap_dump -stdout | gzip -c - > heapdump.hprof.gz
>>
>>
>> Thanks,
>>
>> Yasumasa
>>
>>
>> On 2020/02/20 2:40, Ioi Lam wrote:
>>>
>>>
>>> On 2/19/20 7:24 AM, Schmelter, Ralf wrote:
>>>> Hi Ioi,
>>>>
>>>>> This seems to be an edge case (where your environment has more
>>>>> RAM than disk)
>>>> I would not say it's an edge case. Especially in a cloud 
>>>> environment, your container does not need much free diskspace, 
>>>> since the data is stored in a database and logging goes to stdout.
>>>>
>>>>> I think it would be better to handle this outside of the JVM
>>>>> (using a named pipe and and external program such as the parallel 
>>>>> gzip
>>>>> "pigz") to limit the maintenance overhead of the JVM.
>>>> But then you would have to implement writing the heap dump to a 
>>>> named pipe (and not only on Unix, but on Windows too). And you 
>>>> would still want to do the writing in background threads, so most 
>>>> of the code would stay. You need something like netcat on Windows. 
>>>> And it doesn't cover writing a heap dump on OOM via the VM flag.
>>>>
>>>> And you should to compress the hprof file in a specific way, since 
>>>> it will make it much faster to random access the gzipped hprof file 
>>>> directly.
>>>>
>>>> Note that I think it is a good idea to be able to write the dump to 
>>>> non-file destination. But removing the compression will not save 
>>>> much code and will make the handling messier.
>>>
>>> I was thinking of doing something like this:
>>>
>>> $ mkfifo /tmp/pipe
>>> $ cat /tmp/pipe | gzip -c - > /tmp/zipped &
>>> $ jcmd $PID GC.heap_dump filename=/tmp/pipe
>>>
>>> You can replace the "> /tmp/zipped" part with a program that reads 
>>> from stdin and send it over the network.
>>>
>>> I tried the above with a recent JDK build (with your changes in 
>>> JDK-8234510: Remove file seeking requirement for writing a heap 
>>> dump), but it doesn't seem to work, probably because we need to 
>>> change this code a little bit
>>>
>>> http://hg.openjdk.java.net/jdk/jdk/file/7ef41e83066b/src/hotspot/share/services/heapDumper.cpp#l465 
>>>
>>>
>>> DumpWriter::DumpWriter(const char* path) : _fd(-1), 
>>> _bytes_written(0), _pos(0),
>>> _in_dump_segment(false), _error(NULL) {
>>>      ...
>>>      _fd = os::create_binary_file(path, false);    // don't replace 
>>> existing file <<<
>>>
>>> I also saw a post saying that the JVM can write to named pipes on 
>>> Windows:
>>>
>>> https://urldefense.com/v3/__https://stackoverflow.com/questions/634564/how-to-open-a-windows-named-pipe-from-java__;!!GqivPVa7Brio!NzwD3eTX5oDe2WDGidQjXgiDXpQ7SdnRdyo4D9qxHI46dPcXb5PVzrxZ4UNiUw$ 
>>>
>>> There's no built-in mkfifo command on Windows, but the above link 
>>> points to a .NET example that creates a named pipe and uses that to 
>>> communicate with the JVM.
>>>
>>> I don't know whether this will be a better solution than your 
>>> proposed changes, but I think it should be explored as a possible 
>>> alternative. It does seem to require a little work to get your whole 
>>> data collection system working, but it also seems more flexible and 
>>> extensible.
>>>
>>> Thanks
>>> - Ioi
>>>
>>>
>>>>
>>>> Best regards,
>>>> Ralf
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Ioi Lam <ioi.lam at oracle.com>
>>>> Sent: Mittwoch, 19. Februar 2020 01:16
>>>> To: serguei.spitsyn at oracle.com; Schmelter, Ralf 
>>>> <ralf.schmelter at sap.com>; hotspot-runtime-dev at openjdk.java.net 
>>>> runtime <hotspot-runtime-dev at openjdk.java.net>
>>>> Cc: Laurence Cable <larry.cable at oracle.com>; 
>>>> serviceability-dev at openjdk.java.net
>>>> Subject: Re: RFR(L) 8237354: Add option to jcmd to write a gzipped 
>>>> heap dump
>>>>
>>>> Hi Ralf,
>>>>
>>>> We are usually pretty picky about adding new features into the JVM. 
>>>> This
>>>> seems to be an edge case (where your environment has more RAM than
>>>> disk). I think it would be better to handle this outside of the JVM
>>>> (using a named pipe and and external program such as the parallel gzip
>>>> "pigz") to limit the maintenance overhead of the JVM.
>>>>
>>>> This would also have the benefit that you can do it with almost no 
>>>> local
>>>> storage -- you can read from the named pipe, optionally compress the
>>>> data, and send that over the network.
>>>>
>>>> Thanks
>>>> - Ioi
>>>
>