RFR: JDK-8221596: test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java failed with FileAlreadyExistsException

[mikhailo.seledtsov at oracle.com]

On 3/29/19 5:38 PM, Jie Fu wrote:
> Hi Misha,
>
> Thank you for your review.
>
> Could you please sponsor it?
Certainly.

Thank you,
Misha
> Thanks a lot.
>
> Best regards,
> Jie
>
>
> On 2019/3/30 上午12:02, mikhailo.seledtsov at oracle.com wrote:
>> David, Jie,
>>
>>   Thank you for providing additional details. This makes sense now, I 
>> agree with this fix.
>>
>>
>> Thank you,
>>
>> Misha
>>
>>
>> On 3/28/19 9:44 PM, Jie Fu wrote:
>>> Thanks David.
>>>
>>> On 2019/3/29 下午12:24, David Holmes wrote:
>>>> Hi Jie,
>>>>
>>>> On 29/03/2019 12:30 pm, Jie Fu wrote:
>>>>> Hi Misha,
>>>>>
>>>>> Thanks for your deep analysis.
>>>>>
>>>>> Actually, this bug was originally detected by a script like this
>>>>> ----------------------------------------------
>>>>> rm JTreport JTwork -rf
>>>>>
>>>>> i=0
>>>>> while(true); do
>>>>>
>>>>>    let i++
>>>>>    echo -e "\nRun ${i} ...\n"
>>>>>
>>>>>    ${JTREG}/bin/jtreg \
>>>>>      -jdk:${TESTJDK} \
>>>>>      ${DOCKER_ARGS} \
>>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>>
>>>>> done
>>>>> ----------------------------------------------
>>>>>
>>>>> This script was used to reproduce random bugs triggered by jtreg 
>>>>> tests for our mips-port.
>>>>> Yes, we do rm JT* before the test, but would not like to do that 
>>>>> during the loop.
>>>>>
>>>>> It would take much more time to reproduce and fix bugs like that 
>>>>> if we had to rm JT* every time.
>>>>> So I think it's worth fixing it.
>>>>
>>>> You're likely to find other tests that also have issues being 
>>>> re-run this way. For example I just found 
>>>> runtime/appcds/DirClasspathTest.
>>>>
>>>> I was somewhat surprised to find that not deleting the work 
>>>> directory saved about 33% on the runtime (for the appcds tests: 18m 
>>>> vs. 29m).
>>>>
>>>> So this fix seems okay to me - and it won't hurt anyone who does 
>>>> always clear JTwork first.
>>>>
>>>> Thanks,
>>>> David
>>>>
>>>>> Thanks a lot.
>>>>>
>>>>> Best regards,
>>>>> Jie
>>>>>
>>>>>
>>>>> On 2019/3/29 上午5:50, Mikhailo Seledtsov wrote:
>>>>>> Hi Jie, David,
>>>>>>
>>>>>> I did a bit of digging:
>>>>>>
>>>>>>    - @run driver ClassFileInstaller -jar whitebox.jar 
>>>>>> sun.hotspot.WhiteBox sun.hotspot.WhiteBox$WhiteBoxPermission
>>>>>>      is used in TestCPUSets.java, TestMemoryAwareness.java and 
>>>>>> TestMisc.java
>>>>>>      This instruction will generate whitebox.jar and place it in 
>>>>>> "current working jtreg directory", which is
>>>>>>      ./JTwork/scratch/
>>>>>>
>>>>>>    - in order to be able to use the WhiteBox.jar from within the 
>>>>>> Docker container, it is copied
>>>>>>      to the "JTReg test classes" directory (defined by 
>>>>>> jdk.test.lib.utils.TEST_CLASSES)
>>>>>>      This Utils.TEST_CLASSES directory is mounted via --volume at 
>>>>>> docker-run-time, so that classes can be
>>>>>>      visible/usable/loadable from within the container.
>>>>>>
>>>>>>    -  This Utils.TEST_CLASSES directory is unique for each test run:
>>>>>> ./JTwork/classes/runtime/containers/docker/TestMisc.d/whitebox.jar
>>>>>> ./JTwork/classes/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>>
>>>>>> ./JTwork/classes/runtime/containers/docker/TestCPUSets.d/whitebox.jar 
>>>>>>
>>>>>>       Hence, I do not see a possibility of file access clash 
>>>>>> among the different tests.
>>>>>>
>>>>>>    - if tests are run concurrently (with -conc:X feature), each 
>>>>>> test gets their own scratch directory
>>>>>>      classes directories are also separated, as in non-concurrent 
>>>>>> case
>>>>>>      ./JTwork/scratch/1/whitebox.jar
>>>>>>      ./JTwork/scratch/0/whitebox.jar
>>>>>>      ./JTwork/scratch/2/whitebox.jar
>>>>>> ./JTwork/classes/3/runtime/containers/docker/TestCPUSets.d/whitebox.jar 
>>>>>>
>>>>>> ./JTwork/classes/4/runtime/containers/docker/TestMisc.d/whitebox.jar
>>>>>> ./JTwork/classes/0/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>>
>>>>>>      No file access clashes among tests either
>>>>>>
>>>>>>    Is it possible that you did not clean out the directory from 
>>>>>> the previous run? For test execution, we recommend
>>>>>>    clearning JT* directories from the prior run before a new run 
>>>>>> (rm -Rf JT* or similar). This recommendation is not
>>>>>>    specific to Docker testing, it is a good practice in general 
>>>>>> for executing of jtreg tests.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thank you,
>>>>>> Misha
>>>>>>
>>>>>> On 3/28/19, 4:16 AM, David Holmes wrote:
>>>>>>> Hi Jie,
>>>>>>>
>>>>>>> On 28/03/2019 5:48 pm, Jie Fu wrote:
>>>>>>>> Hi David,
>>>>>>>>
>>>>>>>> Thanks for your review.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Normally to use WhiteBox a test specifies:
>>>>>>>>>
>>>>>>>>>  @run driver ClassFileInstaller sun.hotspot.WhiteBox
>>>>>>>> Yes, we do have it in 
>>>>>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>>>>
>>>>>>> Okay I'm not understanding why Common is copying the WhiteBox 
>>>>>>> classes or where it is copying them to? If this is something to 
>>>>>>> do with the image shouldn't it be completely clean when the test 
>>>>>>> is run - either by deleting existing files or by ensuring a 
>>>>>>> unique location to begin with?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> David
>>>>>>>
>>>>>>>> Best regard,
>>>>>>>> Jie
>>>>>>>>>
>>>>>>>>> Is there a reason this test can't use that?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> David
>>>>>>>>> -----
>>>>>>>>
>>>>>
>>>
>>
>