RFR: JDK-8221596: test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java failed with FileAlreadyExistsException

[David Holmes]

Hi Jie,

On 29/03/2019 12:30 pm, Jie Fu wrote:
> Hi Misha,
> 
> Thanks for your deep analysis.
> 
> Actually, this bug was originally detected by a script like this
> ----------------------------------------------
> rm JTreport JTwork -rf
> 
> i=0
> while(true); do
> 
>    let i++
>    echo -e "\nRun ${i} ...\n"
> 
>    ${JTREG}/bin/jtreg \
>      -jdk:${TESTJDK} \
>      ${DOCKER_ARGS} \
>      test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
> 
> done
> ----------------------------------------------
> 
> This script was used to reproduce random bugs triggered by jtreg tests 
> for our mips-port.
> Yes, we do rm JT* before the test, but would not like to do that during 
> the loop.
> 
> It would take much more time to reproduce and fix bugs like that if we 
> had to rm JT* every time.
> So I think it's worth fixing it.

You're likely to find other tests that also have issues being re-run 
this way. For example I just found runtime/appcds/DirClasspathTest.

I was somewhat surprised to find that not deleting the work directory 
saved about 33% on the runtime (for the appcds tests: 18m vs. 29m).

So this fix seems okay to me - and it won't hurt anyone who does always 
clear JTwork first.

Thanks,
David

> Thanks a lot.
> 
> Best regards,
> Jie
> 
> 
> On 2019/3/29 上午5:50, Mikhailo Seledtsov wrote:
>> Hi Jie, David,
>>
>> I did a bit of digging:
>>
>>    - @run driver ClassFileInstaller -jar whitebox.jar 
>> sun.hotspot.WhiteBox sun.hotspot.WhiteBox$WhiteBoxPermission
>>      is used in TestCPUSets.java, TestMemoryAwareness.java and 
>> TestMisc.java
>>      This instruction will generate whitebox.jar and place it in 
>> "current working jtreg directory", which is
>>      ./JTwork/scratch/
>>
>>    - in order to be able to use the WhiteBox.jar from within the 
>> Docker container, it is copied
>>      to the "JTReg test classes" directory (defined by 
>> jdk.test.lib.utils.TEST_CLASSES)
>>      This Utils.TEST_CLASSES directory is mounted via --volume at 
>> docker-run-time, so that classes can be
>>      visible/usable/loadable from within the container.
>>
>>    -  This Utils.TEST_CLASSES directory is unique for each test run:
>> ./JTwork/classes/runtime/containers/docker/TestMisc.d/whitebox.jar
>> ./JTwork/classes/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>
>> ./JTwork/classes/runtime/containers/docker/TestCPUSets.d/whitebox.jar
>>       Hence, I do not see a possibility of file access clash among the 
>> different tests.
>>
>>    - if tests are run concurrently (with -conc:X feature), each test 
>> gets their own scratch directory
>>      classes directories are also separated, as in non-concurrent case
>>      ./JTwork/scratch/1/whitebox.jar
>>      ./JTwork/scratch/0/whitebox.jar
>>      ./JTwork/scratch/2/whitebox.jar
>> ./JTwork/classes/3/runtime/containers/docker/TestCPUSets.d/whitebox.jar
>> ./JTwork/classes/4/runtime/containers/docker/TestMisc.d/whitebox.jar
>> ./JTwork/classes/0/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>
>>      No file access clashes among tests either
>>
>>    Is it possible that you did not clean out the directory from the 
>> previous run? For test execution, we recommend
>>    clearning JT* directories from the prior run before a new run (rm 
>> -Rf JT* or similar). This recommendation is not
>>    specific to Docker testing, it is a good practice in general for 
>> executing of jtreg tests.
>>
>>
>>
>> Thank you,
>> Misha
>>
>> On 3/28/19, 4:16 AM, David Holmes wrote:
>>> Hi Jie,
>>>
>>> On 28/03/2019 5:48 pm, Jie Fu wrote:
>>>> Hi David,
>>>>
>>>> Thanks for your review.
>>>>
>>>>>
>>>>> Normally to use WhiteBox a test specifies:
>>>>>
>>>>>  @run driver ClassFileInstaller sun.hotspot.WhiteBox
>>>> Yes, we do have it in 
>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>
>>> Okay I'm not understanding why Common is copying the WhiteBox classes 
>>> or where it is copying them to? If this is something to do with the 
>>> image shouldn't it be completely clean when the test is run - either 
>>> by deleting existing files or by ensuring a unique location to begin 
>>> with?
>>>
>>> Thanks,
>>> David
>>>
>>>> Best regard,
>>>> Jie
>>>>>
>>>>> Is there a reason this test can't use that?
>>>>>
>>>>> Thanks,
>>>>> David
>>>>> -----
>>>>
>