Did the tests fail due to TestNG dataprovider parameter type check?

Daniel Fuchs daniel.fuchs at oracle.com
Wed Nov 23 13:59:20 UTC 2016


Hi Frank,

I ran only the tests under jaxp/tests/javax - and one test was failing
(DurationTest) with a similar error (something about a mismatch between
Integer & Long in data provider).

This looked to be completely unrelated to the permission changes, so I
thought it would be better to investigate it as a separate issue.
The failure in the Duration test looked simpler - so it might be a good
place to look at to try and figure out what is going on.

best regards,

-- daniel


On 23/11/16 08:31, Frank Yuan wrote:
> Hi Jon
>
> I run the whole jdk regression tests with jtreg-4.2-b03 [1], and then found lots of tests in different test groups failed with the error message like following:
> test test.java.time.format.TestNumberPrinter.test_pad_ALWAYS(): failure
> org.testng.internal.reflect.MethodMatcherException:
> Data provider mismatch
> Method: test_pad_ALWAYS([Parameter{index=0, type=int, declaredAnnotations=[]}, Parameter{index=1, type=int, declaredAnnotations=[]}, Parameter{index=2, type=long, declaredAnnotations=[]}, Parameter{index=3, type=java.lang.String, declaredAnnotations=[]}])
> Arguments: [(java.lang.Integer)1,(java.lang.Integer)1,(java.lang.Integer)-10,null]
> 	at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:52)
> 	at org.testng.internal.Invoker.injectParameters(Invoker.java:1228)
> 	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1125)
> 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
> 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
> 	at org.testng.TestRunner.privateRun(TestRunner.java:778)
> 	at org.testng.TestRunner.run(TestRunner.java:632)
> 	at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
> 	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
> 	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
> 	at org.testng.SuiteRunner.run(SuiteRunner.java:268)
> 	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
> 	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
> 	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1225)
> 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1150)
> 	at org.testng.TestNG.runSuites(TestNG.java:1075)
> 	at org.testng.TestNG.run(TestNG.java:1047)
> 	at com.sun.javatest.regtest.TestNGAction$TestNGRunner.main(TestNGAction.java:220)
> 	at com.sun.javatest.regtest.TestNGAction$TestNGRunner.main(TestNGAction.java:184)
> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.base/java.lang.reflect.Method.invoke(Method.java:537)
> 	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110)
> 	at java.base/java.lang.Thread.run(Thread.java:844)
>
>
> This test methd signature is:
>  public void test_pad_ALWAYS(int minPad, int maxPad, long value, String result)
>
> The provider return the following data
> Object[][] provider_pad() {
>         return new Object[][] {
>             {1, 1, -10, null},
>             {1, 1, -9, "9"},
>             {1, 1, -1, "1"},
>             ...
>
> However, I got message "Cannot find class java/lang/reflect/JTRegModuleHelper.class to init patch directory" when I run jtreg, although I didn't find any code related to testng dataprovider in jtreg source, I would double confirm with you if this is a definite issue or anything I made incorrectly?
>
> To Christoph
>
> Except above issue, I didn't find any other issue in jdk and langtools repo so far.
>
>
> [1] https://adopt-openjdk.ci.cloudbees.com/job/jtreg/lastSuccessfulBuild/artifact/jtreg-4.2-b03.tar.gz
>
>
> Thanks
> Frank
>
>> -----Original Message-----
>> From: Langer, Christoph [mailto:christoph.langer at sap.com]
>> Sent: Wednesday, November 23, 2016 3:41 PM
>> To: Frank Yuan
>> Cc: code-tools-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-use at openjdk.java.net; 'Volker Simonis'; 'Daniel Fuchs'
>> Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission" "accessDeclaredMembers")
>>
>> Thanks, Frank.
>>
>> we run scheduled jtreg tests for jdk every night so we should have encountered issues if there were some. But langtools could be interesting, I
>> don't think those run automatically for OpenJDK in our environment.
>>
>> Best regards
>> Christoph
>>
>>> -----Original Message-----
>>> From: Frank Yuan [mailto:frank.yuan at oracle.com]
>>> Sent: Mittwoch, 23. November 2016 06:26
>>> To: Langer, Christoph <christoph.langer at sap.com>; 'Volker Simonis'
>>> <volker.simonis at gmail.com>; 'Daniel Fuchs' <daniel.fuchs at oracle.com>
>>> Cc: code-tools-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-
>>> use at openjdk.java.net
>>> Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission"
>>> "accessDeclaredMembers")
>>>
>>> Hi Christoph and Volker
>>>
>>> I have been launching jdk and langtools tests with the new jtreg, will update to
>>> you once I get the result.
>>> Hope jaxp test is special because most of tests should control the Security
>>> Manager setting inside the test methods.
>>>
>>> Thanks
>>> Frank
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: core-libs-dev [mailto:core-libs-dev-bounces at openjdk.java.net] On
>>> Behalf Of Langer, Christoph
>>>> Sent: Wednesday, November 23, 2016 3:51 AM
>>>> Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission"
>>> "accessDeclaredMembers")
>>>>
>>>> Thanks a lot Volker and Daniel for the big support to analyze and fix this.
>>>>
>>>> It seems to me that the proposed fix
>>>> (http://cr.openjdk.java.net/~dfuchs/webrev_8170192/webrev.00/ ) looks like
>>>> the best that can be done at the moment. I agree that it would be nicer if
>>>> jtreg would leave the jtreg lib path as java property to be able to elevate
>>>> all of its contents. But the current proposal with a set of TEST_JARS of
>>>> jtreg, javatest and testng is probably sufficient for jaxp testing.
>>>>
>>>> The best thing to find out about other issues with the new version of testng
>>>> would certainly be if Oracle's internal version of jtreg be updated to use
>>>> the latest testng :-)
>>>>
>>>> Best regards
>>>> Christoph
>>>>
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Volker Simonis [mailto:volker.simonis at gmail.com]
>>>>> Sent: Dienstag, 22. November 2016 20:25
>>>>> To: Daniel Fuchs <daniel.fuchs at oracle.com>
>>>>> Cc: Langer, Christoph <christoph.langer at sap.com>; code-tools-
>>>>> dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-
>>>>> use at openjdk.java.net
>>>>> Subject: Re: Issues running JAXP jtreg tests
>>>>> ("java.lang.RuntimePermission"
>>>>> "accessDeclaredMembers")
>>>>>
>>>>> Hi Daniel,
>>>>>
>>>>> thanks for your patch!
>>>>>
>>>>> I've meanwhile tried to better understand the root cause of the problem.
>>>>>
>>>>> I don't think that the invocation order of the data provider the
>>>>> listener have changed. If you look at the the two version 6.9.5 and
>>>>> 6.9.13 of testng, the org.testng.TestRunner.run() methods look exactly
>>>>> the same in both 6.9.5 [1] and 6.9.13 [2]:
>>>>>
>>>>>   public void run() {
>>>>>     beforeRun();
>>>>>
>>>>>     try {
>>>>>       XmlTest test= getTest();
>>>>>       if(test.isJUnit()) {
>>>>>         privateRunJUnit(test);
>>>>>       }
>>>>>       else {
>>>>>         privateRun(test);
>>>>>       }
>>>>>     }
>>>>>     finally {
>>>>>       afterRun();
>>>>>     }
>>>>>
>>>>> I think the problem is in
>>>>> org.testng.internal.ClassHelper.getAvailableMethods() where we testng
>>>>> only collected the methods until (i.e. excluding) java.lang.Object in
>>>>> 6.9.5 [3] but including java.lang.Object in 6.9.13 [4]:
>>>>>
>>>>> 6.9.5
>>>>> =====
>>>>>   public static Set<Method> getAvailableMethods(Class<?> clazz) {
>>>>>     Set<Method> methods = Sets.newHashSet();
>>>>>     methods.addAll(Arrays.asList(clazz.getDeclaredMethods()));
>>>>>
>>>>>     Class<?> parent = clazz.getSuperclass();
>>>>>     while (Object.class != parent) {
>>>>>       methods.addAll(extractMethods(clazz, parent, methods));
>>>>>       parent = parent.getSuperclass();
>>>>>     }
>>>>>
>>>>> 6.9.13
>>>>> =====
>>>>>   public static Set<Method> getAvailableMethods(Class<?> clazz) {
>>>>>     Set<Method> methods = Sets.newHashSet();
>>>>>     methods.addAll(Arrays.asList(clazz.getDeclaredMethods()));
>>>>>
>>>>>     Class<?> parent = clazz.getSuperclass();
>>>>>     while (null != parent) {
>>>>>       methods.addAll(extractMethods(clazz, parent, methods));
>>>>>       parent = parent.getSuperclass();
>>>>>     }
>>>>>
>>>>> But java.lang.Object has a different class loader (null) compared to
>>>>> the test class (which is loaded by the application class loader),
>>>>> which leads to the AccessControlException with 6.9.13.
>>>>>
>>>>> As far as I can see, this was changed in testng 6.9.10 [5] to fix
>>>>> https://github.com/cbeust/testng/issues/876
>>>>>
>>>>> This behavior may potentially also affect other tests which are
>>>>> running with a security manger so I'm not sure you fix will help for
>>>>> all of them. And I also wonder why this hasn't been detected by other
>>>>> who run testng with a security manager (but maybe nobody is doing that
>>>>> :)
>>>>>
>>>>> Regards,
>>>>> Volker
>>>>>
>>>>> [1] https://github.com/cbeust/testng/blob/testng-
>>>>> 6.9.5/src/main/java/org/testng/TestRunner.java
>>>>> [2]
>>>>>
>>> https://github.com/cbeust/testng/blob/6.9.13/src/main/java/org/testng/TestRu
>>>>> nner.java
>>>>> [3] https://github.com/cbeust/testng/blob/testng-
>>>>> 6.9.5/src/main/java/org/testng/internal/ClassHelper.java
>>>>> [4]
>>>>>
>>> https://github.com/cbeust/testng/blob/6.9.13/src/main/java/org/testng/interna
>>>>> l/ClassHelper.java
>>>>> [5]
>>>>>
>>> https://github.com/cbeust/testng/pull/886/commits/fefedec34706e40ff2bf780b
>>>>> ff7716fca29daaab
>>>>>
>>>>>
>>>>> On Tue, Nov 22, 2016 at 5:24 PM, Daniel Fuchs <daniel.fuchs at oracle.com>
>>>>> wrote:
>>>>>> Hi Christoph,
>>>>>>
>>>>>> I have logged https://bugs.openjdk.java.net/browse/JDK-8170192
>>>>>>
>>>>>> best regards,
>>>>>>
>>>>>> -- daniel
>>>>>>
>>>>>>
>>>>>> On 22/11/16 14:47, Daniel Fuchs wrote:
>>>>>>>
>>>>>>> On 22/11/16 14:43, Langer, Christoph wrote:
>>>>>>>>
>>>>>>>> But, as for fixing with the new testng, will you look into this? Shall
>>>>>>>> I open a bug?
>>>>>>
>>>>>>
>
>



More information about the code-tools-dev mailing list