RFR: 7903580: Allow for re-attempting agent creation when an attempt fails [v4]

[Jaikiran Pai]

On Wed, 15 Nov 2023 01:43:24 GMT, Jaikiran Pai <jpai at openjdk.org> wrote:

>> Can I please get a review of this change which proposes to implement the enhancement request noted in https://bugs.openjdk.org/browse/CODETOOLS-7903580?
>> 
>> Several times in our CI instances we have noticed that a test (action) fails because it is unable to create an `Agent` instance due to the socket connection not being established between the `AgentServer` and the `ServerSocket` running within the jtreg process. This causes the test itself to fail with `Error. Cannot get VM for test: java.net.SocketTimeoutException: Accept timed out`. It has also been noticed that most of the times this is intermittent and subsequent attempt for a different test (action) passes. 
>> 
>> The proposed change here introduces a new configuration parameter `--agent-attempts` under "Agent Pool Options", which allows for configuring the maximum number of attempts that are allowed for getting an agent for an action. This is an optional parameter and by default it has been set to a value of `2`, which by default then allows the agent creation to be retried once if the previous attempt fails. This then means that existing installations/usages of jtreg need not set a value for `--agent-attempts` to enroll for this feature. A value of `1` for this parameter implies that the agent creation attempt will be done only once, which is what currently happens in the absence of this proposed feature.
>> 
>> Inability to obtain an agent, for whatever reason, after attempting `--agent-attempts` times continues to result in test (action) failure, like it does today.
>> 
>> Some additional minor logging related changes have been done too, to help debugging some of the current observed failures.
>> 
>> This change has been tested in the following manners:
>> 
>> 1. This modified build of jtreg has been used to run tier1, tier2 and tier3 of JDK mainline and it has been verified that nothing regresses with this change. That however doesn't mean that agent creation re-attempt logic was tested in these runs (since the first attempt never failed).
>> 2. Locally this build was jtreg was additionally modified to selectively not connect to the `ServerSocket`'s port thus simulating a "Accept timed out" exception. That change then verified that the re-attempt logic does indeed correctly kick in and the test action passes after picking up the newly created agent.
>> 3. `--agent-attempts 1` was passed to the above modified build (from step 2) to verify that the re-attempt isn't attempted when `--agent-attempts 1` an...
>
> Jaikiran Pai has updated the pull request incrementally with one additional commit since the last revision:
> 
>   add a FAQ entry

Hello Mark,

> For my own edification, I'm curious as to why the Process::pid method needs to be called reflectively when pb.start()

I see Christian already answered this part - for jtreg to be usable in Java 8, we cannot statically use APIs that have been introduced in later versions. `Process::pid()` was introduced in Java 9.

> If the accept is the source of the IOException and the initiation of a retry, what happens to the instantiated process, which could still be potentially alive. Do you check that it is alive and (forcibly) terminate it if necessary ?

When an `Agent` is constructed, the `ProcessBuilder` launches the `AgentServer` class which uses `new Socket(....)` to connect to the `ServerSocket` which is hosted by the jtreg process. If the `accept()` times out or fails due to an IOException, then the `ServerSocket` is closed. Once this happens, the jtreg process itself doesn't do any checks on the `AgentServer` process since the `Agent` instance itself couldn't be constructed. However, since the `AgentServer` process is in the `new Socket(...)` code waiting for the connection to be accepted, that call would then fail (there's no server port listening for connections, keeping aside cases like some other process now listening on this port) and that would then trigger the AgentServer process to exit.

Having said that, there's a case to be made that if the construction of the `Agent` class fails, then the created resources (the AgentServer `process` in this case) should also be destroyed. I'll take a look to see if doing that will cause any issues.

-------------

PR Comment: https://git.openjdk.org/jtreg/pull/173#issuecomment-1831904702