Did the tests fail due to TestNG dataprovider parameter type check?

Frank Yuan frank.yuan at oracle.com
Wed Nov 23 08:31:50 UTC 2016


Hi Jon

I run the whole jdk regression tests with jtreg-4.2-b03 [1], and then found lots of tests in different test groups failed with the error message like following:
test test.java.time.format.TestNumberPrinter.test_pad_ALWAYS(): failure
org.testng.internal.reflect.MethodMatcherException: 
Data provider mismatch
Method: test_pad_ALWAYS([Parameter{index=0, type=int, declaredAnnotations=[]}, Parameter{index=1, type=int, declaredAnnotations=[]}, Parameter{index=2, type=long, declaredAnnotations=[]}, Parameter{index=3, type=java.lang.String, declaredAnnotations=[]}])
Arguments: [(java.lang.Integer)1,(java.lang.Integer)1,(java.lang.Integer)-10,null]
	at org.testng.internal.reflect.DataProviderMethodMatcher.getConformingArguments(DataProviderMethodMatcher.java:52)
	at org.testng.internal.Invoker.injectParameters(Invoker.java:1228)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1125)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at org.testng.TestRunner.privateRun(TestRunner.java:778)
	at org.testng.TestRunner.run(TestRunner.java:632)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:366)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:361)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:319)
	at org.testng.SuiteRunner.run(SuiteRunner.java:268)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1225)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1150)
	at org.testng.TestNG.runSuites(TestNG.java:1075)
	at org.testng.TestNG.run(TestNG.java:1047)
	at com.sun.javatest.regtest.TestNGAction$TestNGRunner.main(TestNGAction.java:220)
	at com.sun.javatest.regtest.TestNGAction$TestNGRunner.main(TestNGAction.java:184)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:537)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110)
	at java.base/java.lang.Thread.run(Thread.java:844)


This test methd signature is:
 public void test_pad_ALWAYS(int minPad, int maxPad, long value, String result)

The provider return the following data
Object[][] provider_pad() {
        return new Object[][] {
            {1, 1, -10, null},
            {1, 1, -9, "9"},
            {1, 1, -1, "1"},
            ...

However, I got message "Cannot find class java/lang/reflect/JTRegModuleHelper.class to init patch directory" when I run jtreg, although I didn't find any code related to testng dataprovider in jtreg source, I would double confirm with you if this is a definite issue or anything I made incorrectly?

To Christoph

Except above issue, I didn't find any other issue in jdk and langtools repo so far.


[1] https://adopt-openjdk.ci.cloudbees.com/job/jtreg/lastSuccessfulBuild/artifact/jtreg-4.2-b03.tar.gz


Thanks
Frank

> -----Original Message-----
> From: Langer, Christoph [mailto:christoph.langer at sap.com]
> Sent: Wednesday, November 23, 2016 3:41 PM
> To: Frank Yuan
> Cc: code-tools-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-use at openjdk.java.net; 'Volker Simonis'; 'Daniel Fuchs'
> Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission" "accessDeclaredMembers")
> 
> Thanks, Frank.
> 
> we run scheduled jtreg tests for jdk every night so we should have encountered issues if there were some. But langtools could be interesting, I
> don't think those run automatically for OpenJDK in our environment.
> 
> Best regards
> Christoph
> 
> > -----Original Message-----
> > From: Frank Yuan [mailto:frank.yuan at oracle.com]
> > Sent: Mittwoch, 23. November 2016 06:26
> > To: Langer, Christoph <christoph.langer at sap.com>; 'Volker Simonis'
> > <volker.simonis at gmail.com>; 'Daniel Fuchs' <daniel.fuchs at oracle.com>
> > Cc: code-tools-dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-
> > use at openjdk.java.net
> > Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission"
> > "accessDeclaredMembers")
> >
> > Hi Christoph and Volker
> >
> > I have been launching jdk and langtools tests with the new jtreg, will update to
> > you once I get the result.
> > Hope jaxp test is special because most of tests should control the Security
> > Manager setting inside the test methods.
> >
> > Thanks
> > Frank
> >
> >
> >
> > > -----Original Message-----
> > > From: core-libs-dev [mailto:core-libs-dev-bounces at openjdk.java.net] On
> > Behalf Of Langer, Christoph
> > > Sent: Wednesday, November 23, 2016 3:51 AM
> > > Subject: RE: Issues running JAXP jtreg tests ("java.lang.RuntimePermission"
> > "accessDeclaredMembers")
> > >
> > > Thanks a lot Volker and Daniel for the big support to analyze and fix this.
> > >
> > > It seems to me that the proposed fix
> > > (http://cr.openjdk.java.net/~dfuchs/webrev_8170192/webrev.00/ ) looks like
> > > the best that can be done at the moment. I agree that it would be nicer if
> > > jtreg would leave the jtreg lib path as java property to be able to elevate
> > > all of its contents. But the current proposal with a set of TEST_JARS of
> > > jtreg, javatest and testng is probably sufficient for jaxp testing.
> > >
> > > The best thing to find out about other issues with the new version of testng
> > > would certainly be if Oracle's internal version of jtreg be updated to use
> > > the latest testng :-)
> > >
> > > Best regards
> > > Christoph
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Volker Simonis [mailto:volker.simonis at gmail.com]
> > > > Sent: Dienstag, 22. November 2016 20:25
> > > > To: Daniel Fuchs <daniel.fuchs at oracle.com>
> > > > Cc: Langer, Christoph <christoph.langer at sap.com>; code-tools-
> > > > dev at openjdk.java.net; core-libs-dev at openjdk.java.net; jtreg-
> > > > use at openjdk.java.net
> > > > Subject: Re: Issues running JAXP jtreg tests
> > > > ("java.lang.RuntimePermission"
> > > > "accessDeclaredMembers")
> > > >
> > > > Hi Daniel,
> > > >
> > > > thanks for your patch!
> > > >
> > > > I've meanwhile tried to better understand the root cause of the problem.
> > > >
> > > > I don't think that the invocation order of the data provider the
> > > > listener have changed. If you look at the the two version 6.9.5 and
> > > > 6.9.13 of testng, the org.testng.TestRunner.run() methods look exactly
> > > > the same in both 6.9.5 [1] and 6.9.13 [2]:
> > > >
> > > >   public void run() {
> > > >     beforeRun();
> > > >
> > > >     try {
> > > >       XmlTest test= getTest();
> > > >       if(test.isJUnit()) {
> > > >         privateRunJUnit(test);
> > > >       }
> > > >       else {
> > > >         privateRun(test);
> > > >       }
> > > >     }
> > > >     finally {
> > > >       afterRun();
> > > >     }
> > > >
> > > > I think the problem is in
> > > > org.testng.internal.ClassHelper.getAvailableMethods() where we testng
> > > > only collected the methods until (i.e. excluding) java.lang.Object in
> > > > 6.9.5 [3] but including java.lang.Object in 6.9.13 [4]:
> > > >
> > > > 6.9.5
> > > > =====
> > > >   public static Set<Method> getAvailableMethods(Class<?> clazz) {
> > > >     Set<Method> methods = Sets.newHashSet();
> > > >     methods.addAll(Arrays.asList(clazz.getDeclaredMethods()));
> > > >
> > > >     Class<?> parent = clazz.getSuperclass();
> > > >     while (Object.class != parent) {
> > > >       methods.addAll(extractMethods(clazz, parent, methods));
> > > >       parent = parent.getSuperclass();
> > > >     }
> > > >
> > > > 6.9.13
> > > > =====
> > > >   public static Set<Method> getAvailableMethods(Class<?> clazz) {
> > > >     Set<Method> methods = Sets.newHashSet();
> > > >     methods.addAll(Arrays.asList(clazz.getDeclaredMethods()));
> > > >
> > > >     Class<?> parent = clazz.getSuperclass();
> > > >     while (null != parent) {
> > > >       methods.addAll(extractMethods(clazz, parent, methods));
> > > >       parent = parent.getSuperclass();
> > > >     }
> > > >
> > > > But java.lang.Object has a different class loader (null) compared to
> > > > the test class (which is loaded by the application class loader),
> > > > which leads to the AccessControlException with 6.9.13.
> > > >
> > > > As far as I can see, this was changed in testng 6.9.10 [5] to fix
> > > > https://github.com/cbeust/testng/issues/876
> > > >
> > > > This behavior may potentially also affect other tests which are
> > > > running with a security manger so I'm not sure you fix will help for
> > > > all of them. And I also wonder why this hasn't been detected by other
> > > > who run testng with a security manager (but maybe nobody is doing that
> > > > :)
> > > >
> > > > Regards,
> > > > Volker
> > > >
> > > > [1] https://github.com/cbeust/testng/blob/testng-
> > > > 6.9.5/src/main/java/org/testng/TestRunner.java
> > > > [2]
> > > >
> > https://github.com/cbeust/testng/blob/6.9.13/src/main/java/org/testng/TestRu
> > > > nner.java
> > > > [3] https://github.com/cbeust/testng/blob/testng-
> > > > 6.9.5/src/main/java/org/testng/internal/ClassHelper.java
> > > > [4]
> > > >
> > https://github.com/cbeust/testng/blob/6.9.13/src/main/java/org/testng/interna
> > > > l/ClassHelper.java
> > > > [5]
> > > >
> > https://github.com/cbeust/testng/pull/886/commits/fefedec34706e40ff2bf780b
> > > > ff7716fca29daaab
> > > >
> > > >
> > > > On Tue, Nov 22, 2016 at 5:24 PM, Daniel Fuchs <daniel.fuchs at oracle.com>
> > > > wrote:
> > > > > Hi Christoph,
> > > > >
> > > > > I have logged https://bugs.openjdk.java.net/browse/JDK-8170192
> > > > >
> > > > > best regards,
> > > > >
> > > > > -- daniel
> > > > >
> > > > >
> > > > > On 22/11/16 14:47, Daniel Fuchs wrote:
> > > > >>
> > > > >> On 22/11/16 14:43, Langer, Christoph wrote:
> > > > >>>
> > > > >>> But, as for fixing with the new testng, will you look into this? Shall
> > > > >>> I open a bug?
> > > > >
> > > > >




More information about the code-tools-dev mailing list