RFR: JDK-8221596: test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java failed with FileAlreadyExistsException

[Jie Fu]

Hi Misha,

Thank you for your review.

Could you please sponsor it?
Thanks a lot.

Best regards,
Jie


On 2019/3/30 上午12:02, mikhailo.seledtsov at oracle.com wrote:
> David, Jie,
>
>   Thank you for providing additional details. This makes sense now, I 
> agree with this fix.
>
>
> Thank you,
>
> Misha
>
>
> On 3/28/19 9:44 PM, Jie Fu wrote:
>> Thanks David.
>>
>> On 2019/3/29 下午12:24, David Holmes wrote:
>>> Hi Jie,
>>>
>>> On 29/03/2019 12:30 pm, Jie Fu wrote:
>>>> Hi Misha,
>>>>
>>>> Thanks for your deep analysis.
>>>>
>>>> Actually, this bug was originally detected by a script like this
>>>> ----------------------------------------------
>>>> rm JTreport JTwork -rf
>>>>
>>>> i=0
>>>> while(true); do
>>>>
>>>>    let i++
>>>>    echo -e "\nRun ${i} ...\n"
>>>>
>>>>    ${JTREG}/bin/jtreg \
>>>>      -jdk:${TESTJDK} \
>>>>      ${DOCKER_ARGS} \
>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>
>>>> done
>>>> ----------------------------------------------
>>>>
>>>> This script was used to reproduce random bugs triggered by jtreg 
>>>> tests for our mips-port.
>>>> Yes, we do rm JT* before the test, but would not like to do that 
>>>> during the loop.
>>>>
>>>> It would take much more time to reproduce and fix bugs like that if 
>>>> we had to rm JT* every time.
>>>> So I think it's worth fixing it.
>>>
>>> You're likely to find other tests that also have issues being re-run 
>>> this way. For example I just found runtime/appcds/DirClasspathTest.
>>>
>>> I was somewhat surprised to find that not deleting the work 
>>> directory saved about 33% on the runtime (for the appcds tests: 18m 
>>> vs. 29m).
>>>
>>> So this fix seems okay to me - and it won't hurt anyone who does 
>>> always clear JTwork first.
>>>
>>> Thanks,
>>> David
>>>
>>>> Thanks a lot.
>>>>
>>>> Best regards,
>>>> Jie
>>>>
>>>>
>>>> On 2019/3/29 上午5:50, Mikhailo Seledtsov wrote:
>>>>> Hi Jie, David,
>>>>>
>>>>> I did a bit of digging:
>>>>>
>>>>>    - @run driver ClassFileInstaller -jar whitebox.jar 
>>>>> sun.hotspot.WhiteBox sun.hotspot.WhiteBox$WhiteBoxPermission
>>>>>      is used in TestCPUSets.java, TestMemoryAwareness.java and 
>>>>> TestMisc.java
>>>>>      This instruction will generate whitebox.jar and place it in 
>>>>> "current working jtreg directory", which is
>>>>>      ./JTwork/scratch/
>>>>>
>>>>>    - in order to be able to use the WhiteBox.jar from within the 
>>>>> Docker container, it is copied
>>>>>      to the "JTReg test classes" directory (defined by 
>>>>> jdk.test.lib.utils.TEST_CLASSES)
>>>>>      This Utils.TEST_CLASSES directory is mounted via --volume at 
>>>>> docker-run-time, so that classes can be
>>>>>      visible/usable/loadable from within the container.
>>>>>
>>>>>    -  This Utils.TEST_CLASSES directory is unique for each test run:
>>>>> ./JTwork/classes/runtime/containers/docker/TestMisc.d/whitebox.jar
>>>>> ./JTwork/classes/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>
>>>>> ./JTwork/classes/runtime/containers/docker/TestCPUSets.d/whitebox.jar
>>>>>       Hence, I do not see a possibility of file access clash among 
>>>>> the different tests.
>>>>>
>>>>>    - if tests are run concurrently (with -conc:X feature), each 
>>>>> test gets their own scratch directory
>>>>>      classes directories are also separated, as in non-concurrent 
>>>>> case
>>>>>      ./JTwork/scratch/1/whitebox.jar
>>>>>      ./JTwork/scratch/0/whitebox.jar
>>>>>      ./JTwork/scratch/2/whitebox.jar
>>>>> ./JTwork/classes/3/runtime/containers/docker/TestCPUSets.d/whitebox.jar 
>>>>>
>>>>> ./JTwork/classes/4/runtime/containers/docker/TestMisc.d/whitebox.jar
>>>>> ./JTwork/classes/0/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>
>>>>>      No file access clashes among tests either
>>>>>
>>>>>    Is it possible that you did not clean out the directory from 
>>>>> the previous run? For test execution, we recommend
>>>>>    clearning JT* directories from the prior run before a new run 
>>>>> (rm -Rf JT* or similar). This recommendation is not
>>>>>    specific to Docker testing, it is a good practice in general 
>>>>> for executing of jtreg tests.
>>>>>
>>>>>
>>>>>
>>>>> Thank you,
>>>>> Misha
>>>>>
>>>>> On 3/28/19, 4:16 AM, David Holmes wrote:
>>>>>> Hi Jie,
>>>>>>
>>>>>> On 28/03/2019 5:48 pm, Jie Fu wrote:
>>>>>>> Hi David,
>>>>>>>
>>>>>>> Thanks for your review.
>>>>>>>
>>>>>>>>
>>>>>>>> Normally to use WhiteBox a test specifies:
>>>>>>>>
>>>>>>>>  @run driver ClassFileInstaller sun.hotspot.WhiteBox
>>>>>>> Yes, we do have it in 
>>>>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>>>
>>>>>> Okay I'm not understanding why Common is copying the WhiteBox 
>>>>>> classes or where it is copying them to? If this is something to 
>>>>>> do with the image shouldn't it be completely clean when the test 
>>>>>> is run - either by deleting existing files or by ensuring a 
>>>>>> unique location to begin with?
>>>>>>
>>>>>> Thanks,
>>>>>> David
>>>>>>
>>>>>>> Best regard,
>>>>>>> Jie
>>>>>>>>
>>>>>>>> Is there a reason this test can't use that?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> David
>>>>>>>> -----
>>>>>>>
>>>>
>>
>