Regarding options of error and dump file paths

[Kevin Walls]

Hi Koichi,

Yes, just wanted to (a little late) acknowledge that a few others were thinking about this kind of thing. 8-)

I was thinking from a container point of view, had not heard the demand for this from support teams, but I can see the point somewhat.

Running in a container, where you have some volume/location available for logs to persist, we would ideally have one VM option to set a base/root location for various output files that may currently default to the current directory, or somewhere else.

We really want to take applications as they are, without changing their startup scripts etc, but adding one VM option seems reasonable.

Recently I logged as a placeholder for exactly this kind of option...
 
8270552: Container convenience option.
https://bugs.openjdk.java.net/browse/JDK-8270552

...although I didn't progress it much so far, and have not suggested an option name.

There are some complications I'm sure.  e.g. Would the new option provide a root, and other settings e.g. ErrorFile or HeapDumpPath ALWAYS have the new root prepended?  
Or do we let absolute paths "escape" from the new root? (which might be more work for the users, as you may have several VM options to change, to make use of the new option).  I think the new option is always the new root, for the affected paths.


Also, in a container, we want to explore if this new location can be used for the attach api.  There is currently much scanning of many /proc dirs on Linux.  That is  more involved, but could make use of the same option (the goal is to use fewer options).  But this does not necessarily have to be implemented at the same time (as long as the new option is named appropriately).

More to discuss...

Thanks!
Kevin


-----Original Message-----
From: hotspot-dev <hotspot-dev-retn at openjdk.java.net> On Behalf Of Koichi Sakata
Sent: 14 September 2021 07:45
To: hotspot-dev at openjdk.java.net
Subject: Re: Regarding options of error and dump file paths

Hi all,

I believe that the option helps us, especially people who belong to support team.　Because it enables us easily to get required files to troubleshoot. It's also useful in container environment. We save those files when we set a path of the option to persistent volume, even if container are deleted.

So I'm thinking about how the option works. First of all, it should deal with following files.
- GC (heap dumps)
- JIT (replay files)
- hs_err files
- JFR (a number of files)

Whereas it should exclude files as follows.
- jcmd/dcmd dumps
- Unified logging

Let's see concrete usage examples of the option. Suppose we name the option ReportDir.

Case 1: Set no options
JVM outputs files in each default directory when we set no options.
- GC: ./java_pid%p.hprof
- JIT: ./replay_pid%p.log
- hs_err files: ./hs_err_pid%p.log
- JFR: ./hs_err_pid%p.jfr, ./hs_oom_pid%p.jfr, ./hs_soe_pid%p.jfr

Case 2: Set the option only
We just run `java -XX:ReportDir=/foo/bar/ ...`, then those files are putted in the /foo/bar/ directory.
- GC: /foo/bar/java_pid%p.hprof
- JIT: /foo/bar/replay_pid%p.log
- hs_err files: /foo/bar/hs_err_pid%p.log
- JFR: /foo/bar/hs_err_pid%p.jfr, /foo/bar/hs_oom_pid%p.jfr, /foo/bar/hs_soe_pid%p.jfr

Case 3: Set the option with a relative path Suppose the working directory is /home/duke, run `java -XX:ReportDir=./foo/bar/ ...`. JVM finds the output directory from the working directory and the relative path.
- GC: /home/duke/foo/bar/java_pid%p.hprof
- JIT: /home/duke/foo/bar/replay_pid%p.log
- hs_err files: /home/duke/foo/bar/hs_err_pid%p.log
- JFR: /home/duke/foo/bar/hs_err_pid%p.jfr,
/home/duke/foo/bar/hs_oom_pid%p.jfr, /home/duke/foo/bar/hs_soe_pid%p.jfr

Case 4: Set the option with the existing path option Run `java -XX:ReportDir=/foo/bar/ -XX:ErrorFile=/home/duke/hs_err_pid%p.log ...`. The path of ErrorFile overrides the value of ReportDir.
- GC: /foo/bar/java_pid%p.hprof
- JIT: /foo/bar/replay_pid%p.log
- hs_err files: /home/duke/hs_err_pid%p.log <- It differs from the others
- JFR: /foo/bar/hs_err_pid%p.jfr, /foo/bar/hs_oom_pid%p.jfr, /foo/bar/hs_soe_pid%p.jfr

Case 5: Set the option with the existing path option which has a relative path Suppose the working directory is /home/duke, run `java -XX:ReportDir=./foo/bar/ -XX:HeapDumpPath=./baz/ -XX:+HeapDumpOnOutOfMemoryError ...`.
- GC: /home/duke/foo/bar/baz/java_pid%p.hprof <- It differs from the others
- JIT: /home/duke/foo/bar/replay_pid%p.log
- hs_err files: /home/duke/foo/bar/hs_err_pid%p.log
- JFR: /home/duke/foo/bar/hs_err_pid%p.jfr,
/home/duke/foo/bar/hs_oom_pid%p.jfr, /home/duke/foo/bar/hs_soe_pid%p.jfr

The above example finds the heap dump path by the combination of the working directory, the relative path of ReportDir and the relative path of HeapDumpPath.
As an alternative idea, we can ignore the relative path of ReportDir when HeapDumpPath has a relative path. In that case, the heap dump path is as follows.
- GC: /home/duke/baz/java_pid%p.hprof

In either case, I recognize that using relative paths will be slightly complicated...

Last but not least, I should be pleased if we could go ahead with this topic.

Regards,
Koichi


On 03-09-2021 05:41 PM, Koichi Sakata wrote:
> Hi David,
> 
> I’m sorry for the late reply. Thank you for your great advice.
> 
>> Having an explicit option override the default directory option is a 
>> good idea, but I'm not sure it is that clear cut. If you can specify 
>> a relative directory and file name for a given dump file, might you 
>> not want that to be relative to the specified default path, rather 
>> than relative to the pwd?
> 
> I occasionally want to use a relative path from the specified default 
> path. This usage might confuse the path where files are outputted and 
> complicate to fix, so we probably should prohibit relative paths when 
> we use the default path. We can choose the specification after we find 
> detailed expectations.
> 
>> And we actually have quite a lot of potential output files from:
>>    - GC (heap dumps)
>>    - JIT (replay files)
>>    - hs_err files
>>    - JFR (a number of files)
>>    - jcmd/dcmd dumps?
>>    - Unified logging?
>> 
>> I think figuring out the exact details of how this should work, and 
>> interact with all the different files involved may be more involved 
>> than just prepending a path component.
> 
> I completely agree with you. To enable the new option needs a lot of 
> our work, but that will improve convenience for users, I believe.
> Enabling easily to gathering error related files in one place helps us 
> to troubleshoot. Not so many users set all these path options. If they 
> use the new option, all they have to do will be sending files in the 
> directory to their support personnel. In addition, they will get 
> easier to keep files even on container environments.
> 
>> I also think I would need to hear much greater demand, with detailed 
>> usage expectations, before supporting this.
> 
> I think so, too. I'd like to hear various people's point of view.
> 
> Regards,
> Koichi
> 
> 
> On 2021/08/26 15:23, David Holmes wrote:
>> Hi Koichi,
>> 
>> On 23/08/2021 1:29 pm, Koichi Sakata wrote:
>>> Hi all,
>>> 
>>> I'm writing to get feedback on my idea about options for error and 
>>> dump file paths.
>>> 
>>> First of all, we can specify several options related to error and 
>>> dump files. For example, the HeapDumpPath option sets the heap dump 
>>> file and the ErrorFile option sets the hs_error file.
>>> 
>>> I've felt inconvenience about that because we need to write all path 
>>> options to put those files in a specific directory. I also recognize 
>>> that they are outputted in the working directory when I run an 
>>> application with no options. But I'd like to keep them in any 
>>> directory. So the new option that sets the directory where those 
>>> files are outputted would be useful, I think.
>>> 
>>> The new option helps us especially to run applications on containers 
>>> like Docker, Kubernetes etc. If we run them without those existing 
>>> options on containers, files will be put in the local directory of 
>>> each container. We lose files after we operate the container such as 
>>> deleting it. The option enables us to keep certainly all error and 
>>> dump files if we just specify the path of the persistent volume for 
>>> the new option.
>>> 
>>> As a concrete example, when we specify 
>>> -XX:ErrorAndDumpPath=/foo/bar/ (This option name is tentative), 
>>> -XX:+HeapDumpOnOutOfMemoryError and -XX:StartFlightRecording etc., 
>>> files are generated in the /foo/bar directory. From my point of 
>>> view, the option will deal with the following files:
>>> - heap dump file (java_pid%p.hprof)
>>> - error log file (hs_err_pid%p.log)
>>> - JFR emergency dumps (hs_err_pid%p.jfr, hs_oom_pid%p.jfr,
>>> hs_soe_pid%p.jfr)
>>> - replay file (replay_pid%p.log)
>>> 
>>> The existing path options should override the new option. If I set 
>>> -XX:ErrorAndDumpPath=/foo/bar/ and -XX:HeapDumpPath=/foo/baz/, a 
>>> heap dump file will be in the /foo/baz directory and other files 
>>> will be created in the /foo/bar.
>>> 
>>> I would like to hear your point of view. If some people agree to 
>>> this idea, I will write a patch.
>> 
>> My initial reaction was that this seemed something better handled in 
>> a launch script because I figured if you had complex needs in 
>> relation to where these files were being placed, then you'd use a 
>> launch script to help manage that anyway.
>> 
>> But I can see there would be some convenience to controlling the 
>> output directory without also having to restate the default file 
>> names.
>> 
>> Having an explicit option override the default directory option is a 
>> good idea, but I'm not sure it is that clear cut. If you can specify 
>> a relative directory and file name for a given dump file, might you 
>> not want that to be relative to the specified default path, rather 
>> than relative to the pwd?
>> 
>> And we actually have quite a lot of potential output files from:
>>   - GC (heap dumps)
>>   - JIT (replay files)
>>   - hs_err files
>>   - JFR (a number of files)
>>   - jcmd/dcmd dumps?
>>   - Unified logging?
>> 
>> I think figuring out the exact details of how this should work, and 
>> interact with all the different files involved may be more involved 
>> than just prepending a path component.
>> 
>> I also think I would need to hear much greater demand, with detailed 
>> usage expectations, before supporting this.
>> 
>> Just my 2c.
>> 
>> Cheers,
>> David
>> -----
>> 
>>> Regards,
>>> Koichi