From sitnikov.vladimir at gmail.com Fri Aug 8 13:25:34 2025 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Fri, 8 Aug 2025 16:25:34 +0300 Subject: Option for adding and extra TestResultCollector when executing tests Message-ID: Hi, WDYT of adding an option so the end-users could add TestResultCollector via command-line options? I'm integrating jcstress with junit-jupiter engine, and I would like to get notifications from jcstress for the test success/failures. It looks like the current JCStress hard-codes the listeners, and duplicating the JCStress class itself in a custom runner sounds scary. Vladimir -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvanek at redhat.com Mon Aug 11 10:17:52 2025 From: jvanek at redhat.com (Jiri Vanek) Date: Mon, 11 Aug 2025 12:17:52 +0200 Subject: Option for adding and extra TestResultCollector when executing tests In-Reply-To: References: Message-ID: hi! https://github.com/openjdk/jcstress/pull/158 This PR is adding junit compatible results output. Would it not be enough for you? But otherwise I think Aleksey would like your idea, that my junit collector would stay out of main codebase:) However, my mine purpose is, that we can be able to exchange structured jcstress results. That means one shared junit-like output. Otherwise each of of us will have theirs own junit-based collector with output:( Thanx!! J. On 8/8/25 15:25, Vladimir Sitnikov wrote: > Hi, > > WDYT of adding an option so the end-users could add?TestResultCollector via command-line options? > I'm integrating jcstress with junit-jupiter engine, and I would like to get notifications from jcstress for the test success/failures. > > It looks like the current?JCStress hard-codes the listeners, and duplicating the JCStress class itself in a custom runner sounds scary. > > Vladimir -- Jiri Vanek Principal JDK QA engineer IBM Middleware Czechia s.r.o. +420 775 39 01 09 From sitnikov.vladimir at gmail.com Mon Aug 11 10:54:27 2025 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Mon, 11 Aug 2025 13:54:27 +0300 Subject: Option for adding and extra TestResultCollector when executing tests In-Reply-To: References: Message-ID: >This PR is adding junit compatible results output. >Would it not be enough for you? It won't be enough :-/ I would like to execute jcstress tests with junit-jupiter engine. In other words, I would like to execute jcstress-based tests just like I execute JUnit-based tests in my Gradle project: ./gradlew test # execute all tests (both JUnit and JCStress) ./gradlew test --tests "org.example.concurrency.ConcurrencyTest" # execute a selected test (it could be either JCSTress or JUnit) At the same time, I want jcstress to report test execution status to Gradle as it executes tests, so a mere xml does not help much. Vladimir -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvanek at redhat.com Mon Aug 11 11:10:33 2025 From: jvanek at redhat.com (Jiri Vanek) Date: Mon, 11 Aug 2025 13:10:33 +0200 Subject: Option for adding and extra TestResultCollector when executing tests In-Reply-To: References: Message-ID: On 8/11/25 12:54, Vladimir Sitnikov wrote: > >This PR is adding junit compatible results output. > >Would it not be enough for you? > > It won't be enough :-/ But custom collector is not enough to help with this: > I would like to execute jcstress tests with junit-jupiter engine. Yup, you already wrote about that. Jcstress now can be already started for one tes only, so I do not miss this feature. To do this properly is a lot of work. However it should be pretty straightforward to generate a junit testfiles, each for test. And you will be able to run it as you described. But then there is question, if this granularity is enough for you, or if you would like to consider each test variant as separate test. When I was writing https://github.com/openjdk/jcstress/pull/158, this difference was crucail for me. End I ended to generate both. Because the test by test is not granualr enoug, and the test variants differs to much between HW and jdks. In case of generated testwrapper, the https://github.com/openjdk/jcstress/pull/158 would be the only missing part. Note that I;m not sugesting any process spawning (that do jcstress on its own later), but direct executions of individual classes from generated stubs. > In other words, I would like to execute jcstress-based tests just like I execute JUnit-based tests in my Gradle project: > ./gradlew test # execute all tests (both JUnit and JCStress) > ./gradlew test --tests "org.example.concurrency.ConcurrencyTest" # execute a selected test (it could be either JCSTress or JUnit) Jcstress can be wrapped like this. And it may be much more secure then repalcing the internal runner, which is doing its job pretty well. Especially if you do not care about the dynamic nature of test variants. Such test files can be easily provided artifact. > > At the same time, I want jcstress to report test execution status to Gradle as it executes tests, so a mere xml does not help much. > > Vladimir > -- Jiri Vanek Principal JDK QA engineer IBM Middleware Czechia s.r.o. +420 775 39 01 09 From sitnikov.vladimir at gmail.com Mon Aug 11 11:16:38 2025 From: sitnikov.vladimir at gmail.com (Vladimir Sitnikov) Date: Mon, 11 Aug 2025 14:16:38 +0300 Subject: Option for adding and extra TestResultCollector when executing tests In-Reply-To: References: Message-ID: >Jcstress can be wrapped like this. And it may be much more secure then repalcing the internal runner, which is doing its job pretty well. Especially if you do not care about the dynamic nature of test variants. Such test files can be easily provided artifact. Could you please clarify how one can "wrap jcstress like this"? Just to recap: 1) I would like ./gradlew test to work the same for both regular junit tests and jcstress tests 2) I would like to execute JCStress-based tests from IDE (e.g. IntelliJ IDEA) with regular "run test" action. I don't quite get the way you suggest to wrap it. Vladimir -------------- next part -------------- An HTML attachment was scrubbed... URL: From jvanek at redhat.com Mon Aug 11 11:25:27 2025 From: jvanek at redhat.com (Jiri Vanek) Date: Mon, 11 Aug 2025 13:25:27 +0200 Subject: Option for adding and extra TestResultCollector when executing tests In-Reply-To: References: Message-ID: On 8/11/25 13:16, Vladimir Sitnikov wrote: > >Jcstress can be wrapped like this. And it may be much more secure then repalcing the internal runner, which is doing its job pretty well. Especially if you do not care? about the dynamic nature of test variants. Such test files can be > easily provided artifact. > > Could you please clarify how one can "wrap jcstress like this"? > > Just to recap: > 1) I would like ./gradlew test to work the same for both regular junit tests and jcstress tests > 2) I would like to execute JCStress-based tests from IDE (e.g. IntelliJ IDEA) with regular "run test" action. > > I don't quite get the way you suggest to wrap it. That was refereing to possibility to generate junit test files. I can imagine the build of jcstress generates full set of classes: org.openjdk.jcstress.junit.whatever where each @test method, will call `org.openjdk.jcstress.Main("org.openjdk.jcstress.tests.whatever");` That maybe to dummy for your usecase, but I'm still unsure why you need it at all (and is much safer then replacing the current runner). j. From jvanek at openjdk.org Thu Aug 14 11:05:43 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:43 GMT Subject: RFR: 7903754: jcstress: Implement fail-on-error run option [v7] In-Reply-To: References: Message-ID: On Mon, 23 Jun 2025 08:21:56 GMT, Ji?? Van?k wrote: >> Initial PoC >> >> It currently show how to set up argument, and how the framework will be terminated. Feedback welcomed. Should be finished soon > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The incremental webrev excludes the unrelated changes brought in by the merge/rebase. The pull request contains 11 additional commits since the last revision: > > - Removed counting of groups. > > It do not fill its purpose in real life > - Simplified to simpe -foe > > Both existing swithces with parameter changed to simngle -foe swithc, > whcih do not have any parameter, and if used , jcstress ends as soon as > possible after first error. > > Dropped the %% relative metric. Was good, but was never deployed afaik. > Simplifed test counting, now only tests are counted (no longer group) > Kept possibility to forbid exiting in the middle of group via property > > Kept posibility to set limit both for absolute and relative treshold, > so user can still wait for more errors to appear before it is canceled. > - replaced incorrect System.out by output > - Experimental counting of softErrors through property > - Fix in logic > - Repalced list.size by counting collector > - Implemented main terminating logic > - Moved the dying message to correct place > - Refactored ConsoleReport and Failfast to share test counting logic > - typo > - ... and 1 more: https://git.openjdk.org/jcstress/compare/59185922...389dd675 Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/157#issuecomment-3188041766 From jvanek at openjdk.org Thu Aug 14 11:05:38 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:38 GMT Subject: RFR: 7903889: jcstress: document used properites (and env variables) [v6] In-Reply-To: References: <_id_iOGQYGai8cFr-oaXer0R8UivNlj7-ZavmtRzetU=.1c07c8af-058d-4a9e-b83b-3a23bfb68776@github.com> Message-ID: <5lTkk8pW2HLHHXdcT2Al45EmiBV6aKN9tdtkzZZAj4E=.e0e38eac-2a89-4b3b-96fb-dad792d498c7@github.com> On Mon, 23 Jun 2025 08:49:27 GMT, Ji?? Van?k wrote: >> The purpose of this PR is to move properties and properties based initializations to single place, and to document them, and to enable printing the listing with the documentation via help. >> >> No Env. vars were found. >> >> The verbose help can be removed, the individual properties with defaults are wrapped to its class. The OptionSpec and OptionFormatter can be used if needed > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains four commits: > > - Moved declaration of individual proeprties to ceparate classes by family > - Extracted properties encapsualted in objects > - Removed autoformatted whitespaces > - 7903889: jcstress: document used properites (and env variables) > > Known properties unifed in one class and printed in verbose help: > The verbose help can be removed, the individual properties with defaults > can be wrapped to its class. Even the OptionSpec and OptionFormatter can > be used if needed Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/156#issuecomment-3188041881 From jvanek at openjdk.org Thu Aug 14 11:05:43 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:43 GMT Subject: RFR: 7903819: jcstress do not print structuralised results [v2] In-Reply-To: <6FMz9T4k7cDvZyBPXZrnTG1YCh758jJXoxGAJpGYIro=.657b12f6-3cad-4920-a7c7-2723c3e2203e@github.com> References: <6FMz9T4k7cDvZyBPXZrnTG1YCh758jJXoxGAJpGYIro=.657b12f6-3cad-4920-a7c7-2723c3e2203e@github.com> Message-ID: On Mon, 23 Jun 2025 08:50:02 GMT, Ji?? Van?k wrote: >> Jcstress are generating very good html report, which is however not exactly great for automated comapriosns over various systyems and architectures. This pr is adding junit-like, configurable report. > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 25 commits: > > - Dropped the dom-based generation > > Without stream the serilializer was unnecessarily big > - Started migration to proepr DOM > - Revert "FIXME Added fast fake results generator REMOVE IT" > > This reverts commit 0d758a79012dcb11f2a98b74a326b58809a97599. > The junit result looks good. Now testing on real data > - Fixed few noted comments/variables > - made default output xsd valid again > > and keeping option to enable the widely used, but not standart parts for > higher readability > - changed comments proeprty to comment out non-standart elements > > soem of the wide spread features like nested testsuites and properties > are nmot part of xsd... those will be commented out b default, and moved > to fail/err if possible > - Unifide testcase print > - Support for time attribute and removed some dnagling code > - now prints correct summary > > stil not perfect... > mia: interesting? time? > and move to dom.... > - Not printing the seed argument > - ... and 15 more: https://git.openjdk.org/jcstress/compare/1d143cbd...870f1605 Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/158#issuecomment-3188041661 From jvanek at openjdk.org Thu Aug 14 11:05:44 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:44 GMT Subject: RFR: 7903750: TimeBudget (-tb) does not fulfill its promisses [v8] In-Reply-To: References: Message-ID: On Mon, 23 Jun 2025 08:22:32 GMT, Ji?? Van?k wrote: >> Added warning, when time budget is to small, and final run will run at least two times longer. > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 11 commits: > > - Removed duplicated construcion of timebudget > - directly return new instance without intermediate variable > > Co-authored-by: Galder Zamarre?o > - Removed table like formatting for time budget warnning > - Removed unnecesasary NPE check in getTimeBudget > > The NPE workaround was causing NPE to be only delayed for later > - Removed unnecessary shortening of MS before next conversion > > - ms -= TimeUnit.HOURS.toMillis(hours); > - ms -= TimeUnit.MINUTES.toMillis(minutes); > - Used WARNING: and FATAL: instead of +++ WARNING - and +++ FATAL - > - renamed getNiceMsTimeDate to formatMsToDaysAndTime > - Replaced System.out by preset out > - added warning if timebudget is to small > - print time budfget also for listing > > Required two small refactorings to get nice code reused > - ... and 1 more: https://git.openjdk.org/jcstress/compare/1d143cbd...cbedff70 Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/161#issuecomment-3188041452 From jvanek at openjdk.org Thu Aug 14 11:05:42 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:42 GMT Subject: RFR: 7903671: jcstress: Update buffer tests for JDK-8318966 [v2] In-Reply-To: References: Message-ID: <-rNbK5sALGFxKJC4JZPz-WWkDRNF_nycS4Kxg_7pBa0=.293c3163-c04d-4e17-89d4-81b706e322cd@github.com> On Mon, 23 Jun 2025 09:10:54 GMT, Ji?? Van?k wrote: >> part 1: Detecting jdk of target VM and exuding illegible tests >> >> This is prequel to full fixing of CODETOOLS-7903671. This PR detects target JDK version and then disables illegible tests. >> part 2 will be done in templates of affected tests and logic around generating the classes from them. In meantime this should server. >> >> The jdk detection will be most likely used anyway. WDYT? If we agree on similar approach, I will create a new bug for this PR only. > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains two commits: > > - Excluding jdk23+ illegible tests on jdk23+ > - Added detection on target JDK version Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/159#issuecomment-3188041564 From jvanek at openjdk.org Thu Aug 14 11:05:42 2025 From: jvanek at openjdk.org (=?UTF-8?B?SmnFmcOt?= =?UTF-8?B?IA==?= =?UTF-8?B?VmFuxJtr?=) Date: Thu, 14 Aug 2025 11:05:42 GMT Subject: RFR: 7903774: make all tests combinations printing nicer and/or configurable [v10] In-Reply-To: References: <3wZeI91CX_iFefOZJdSxdmZFSyzjd8-w1PJXYufNxrM=.77abf5c8-c5ca-45b9-9b8b-e5c0a7e9b478@github.com> Message-ID: On Mon, 23 Jun 2025 08:45:52 GMT, Ji?? Van?k wrote: >> This is still WIP, but already on row what [CODETOOLS-7903774](https://bugs.openjdk.org/browse/CODETOOLS-7903774) suggested. >> >> It will be finished once all desired output combinations are included, but already now is doing the usage of listing much more comfortable and code readable. >> >> From: >> >> `TESTS` - as introduced by https://github.com/openjdk/jcstress/pull/149 >> `COMBINATIONS` - as introduced in https://github.com/openjdk/jcstress/pull/149 verbose mode (thus surpassing the `-v` impact >> `ALL_TESTS` - all tests as it was before https://github.com/openjdk/jcstress/pull/149 >> `COMBINATIONS_GROUPED ` - as suggested in https://github.com/openjdk/jcstress/pull/149#discussion_r1668678910 >> `COMBINATIONS_JSON` same as combinations, but a valid json >> `COMBINATIONS_GROUPS` same as COMBINATIONS_GROUPED only instead of enumeration in each group, only count of members will be printed >> >> >> * TESTS are already implemented as ALL_MATCHING >> * COMBINATIONS as ALL_MATCHING_COMBINATIONS >> * ALL_TESTS as ALL >> >> COMBINATIONS_GROUPED, and COMBINATIONS_GROUPS are under development. >> The _JSON variants are tobe decided, but ability to JQL it sounds nice. >> >> Suggestion to better names welcomed. > > Ji?? Van?k has updated the pull request with a new target base due to a merge or a rebase. The pull request now contains 10 commits: > > - Implemented proper json listings > > - id can be used both flatten (so equal to plaintext) or properly > strcutured for JQ (default) > - listing can be redirecd to stderr to use with `jq` directly > - ALL now corectly return all tests > - Added property to control json variants fragemnts > > if true, the key will be written as: > "[actor1, actor2], spinLoopStyle: Thread.onSpinWait(), threads: 2, forkId: 0, maxFootprintMB: 64, compileMode: 7, shClass: (PG 0, CG 0), (PG 0, CG 0), strideSize: 256, strideCount: 40, cpuMap: null, [-XX:+UseBiasedLocking, -XX:+StressLCM, -XX:+StressGCM, -XX:+StressIGVN, -XX:+StressCCP, -XX:StressSeed=yyyyyyyy]" > > otherwise (default), it will be written as > > ... > "actorNames": ["actor1", "actor2"], > "spinLoopStyle": "Thread.onSpinWait()", > "threads": 2, > "forkId": 0, > "maxFootprintMB": 64, > "compileMode": 7, > "shClass": "(PG 0, CG 0), (PG 0, CG 0)", > "strideSize": 256, > "strideCount": 40, > "cpuMap": "null", > "jvmArgs": [-XX:+UseBiasedLocking, -XX:+StressLCM, -XX:+StressGCM, -XX:+StressIGVN, -XX:+StressCCP, -XX:StressSeed=yyyyyyyy]" > ... > - Implemented naive json formatting > > To cut away the header one can use: | sed '0,/^All.*[0-9]\+$/d' > - Implemented totaling of listing only > - Fixed spacing in help > - Removed redundnant hyphen > - implemented MATCHING_(INVERTED)GROUPS(_COUNT) > - Refactored lsiting so it takes proper argument > > instead of misusing verbose switch > and thus returned the ability to print allavailable tetest > - refactored listTests() to its own class Is there hope? ------------- PR Comment: https://git.openjdk.org/jcstress/pull/153#issuecomment-3188041969