RFC: jtreg IntelliJ plugin: run individual test methods

[Jorn Vernee]

I've found an additional issue with the current query string format: it 
does not support @Nested JUnit test classes (which we're actually using 
in practice at the moment in the project I'm working on), because the 
JUnit runner currently always passed the top-level class name to JUnit, 
rather than the class name of the @Nested class.

So, if we want to support selecting methods in nested test classes as 
well, we need to incorporate the class name in the query string. In that 
case, I suggest the following format:

<className>::<methodName>[(<parameterTypeName>[...,<parameterTypeName>])]

Where, by default, an absent/empty query string will just select the 
class specified by the @run tag. (We could also make the `<className>::` 
prefix optional, I think)

I've pushed an update which implements this, if you want to check it out.

Jorn

On 7-1-2025 18:33, Jorn Vernee wrote:
> Unfortunately without support for parameter type names, JUnit test 
> methods that accept parameters will not work at all. i.e. it is not 
> just about overload selection. If the parameter types are not 
> specified, JUnit will look for a method taking no parameters.
>
> Note, though, that I'm not just using a string-based heuristic to 
> derive the binary name. Intellij actually has facilities to resolve a 
> source name + resolution scope into a 'PsiClass'. The issue is just 
> that there is no straightforward way to get a binary name from a 
> PsiClass. So, for nested classes I am reconstructing the binary name 
> manually from the fully qualified name, and replacing dots with 
> dollars in the name based on the actual enclosing classes of the type.
>
> Jorn
>
> On 7-1-2025 18:24, Maurizio Cimadamore wrote:
>> Hi Jorn,
>> your extension to the plugin seems a fine idea - it would be nice to 
>> be able to run specific methods through the plugin.
>>
>> IMHO, adding parameter types might be a step too far - the general 
>> problem of "predicting" which binary name a source class name might 
>> have is rather hard, and it amounts at heuristics (e.g. there's more 
>> to it than just replacing dots with dollars, as you also take into 
>> account as to whether other class declared with a dollar in their 
>> name (!!) are present).
>>
>> I wonder if a sensible first step would be to just support the same 
>> logic that jtreg runners also support -- e.g. just a method name. 
>> After all, it is extremely unlikely to have overloaded test methods. 
>> E.g. it seems to me that we're in a case where we can get almost all 
>> the benefit at a fraction of the complexity.
>>
>> What do you think?
>>
>> Maurizio
>>
>> On 07/01/2025 17:05, Jorn Vernee wrote:
>>> Hello,
>>>
>>> Jtreg has for a while now had the ability to attach a query string 
>>> to the name of a test, which then gets used by the JUnit or TestNG 
>>> runner to select a particular test method to run.
>>>
>>> The intellij plugin currently doesn't have support for specifying a 
>>> query string, or running individual test methods, so I've put 
>>> together a patch that adds that support [1]. This support has two 
>>> parts: add a 'Query String' field to the jtreg test configuration 
>>> window in intellij, and automatically populate the query string when 
>>> creating a test configuration from a test method in a source file.
>>>
>>> Most of this is pretty straightforward. I did run into a small 
>>> issue: when a method takes parameters, junit requires the parameter 
>>> types the be specified as binary names. It takes some work to derive 
>>> the binary name of the parameter types from the source AST 
>>> representation exposed by intellij to the plugin.
>>>
>>> Additionally, I had to amend the format of the query strings 
>>> accepted by the TestNG and JUnit runners to accept a trailing list 
>>> of parameter type names. The new format of the query string is:
>>>
>>> <methodName>[-<parameterTypeName>[...,<parameterTypeName>]]
>>>
>>> Does this seem like a good approach?
>>>
>>> Future enhancements could include:
>>>
>>> 1.) displaying individual test methods as results in the intellij 
>>> UI. The problem with doing this is currently that jtreg does not 
>>> forward the results from JUnit/TestNG to the observer the plugin 
>>> attaches to jtreg. Adding support for this seems straightforward 
>>> enough. Potentially it would also be useful to 'itemize' other steps 
>>> that a jtreg test takes, such as compiling, or individual @run tags.
>>> 2.) for parameterized tests, the TestNG runner that comes with 
>>> intellij shows each case (i.e. combination of paramters) 
>>> individually as a result in the UI, and allows re-running just a 
>>> single case through a unique ID. I think JUnit has similar support 
>>> for this. We could add a similar feature to the jtreg plugin.
>>>
>>> However, I thought what I have currently is useful enough in 
>>> isolation to start the discussion on this.
>>>
>>> Jorn
>>>
>>> [1]: 
>>> https://github.com/openjdk/jtreg/compare/master...JornVernee:jtreg:RunMethods
>>>