RFR: JDK-8221596: test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java failed with FileAlreadyExistsException

[Jie Fu]

Thank you so much, Misha.

On 2019/3/30 上午8:48, mikhailo.seledtsov at oracle.com wrote:
>
>
> On 3/29/19 5:38 PM, Jie Fu wrote:
>> Hi Misha,
>>
>> Thank you for your review.
>>
>> Could you please sponsor it?
> Certainly.
>
> Thank you,
> Misha
>> Thanks a lot.
>>
>> Best regards,
>> Jie
>>
>>
>> On 2019/3/30 上午12:02, mikhailo.seledtsov at oracle.com wrote:
>>> David, Jie,
>>>
>>>   Thank you for providing additional details. This makes sense now, 
>>> I agree with this fix.
>>>
>>>
>>> Thank you,
>>>
>>> Misha
>>>
>>>
>>> On 3/28/19 9:44 PM, Jie Fu wrote:
>>>> Thanks David.
>>>>
>>>> On 2019/3/29 下午12:24, David Holmes wrote:
>>>>> Hi Jie,
>>>>>
>>>>> On 29/03/2019 12:30 pm, Jie Fu wrote:
>>>>>> Hi Misha,
>>>>>>
>>>>>> Thanks for your deep analysis.
>>>>>>
>>>>>> Actually, this bug was originally detected by a script like this
>>>>>> ----------------------------------------------
>>>>>> rm JTreport JTwork -rf
>>>>>>
>>>>>> i=0
>>>>>> while(true); do
>>>>>>
>>>>>>    let i++
>>>>>>    echo -e "\nRun ${i} ...\n"
>>>>>>
>>>>>>    ${JTREG}/bin/jtreg \
>>>>>>      -jdk:${TESTJDK} \
>>>>>>      ${DOCKER_ARGS} \
>>>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>>>
>>>>>> done
>>>>>> ----------------------------------------------
>>>>>>
>>>>>> This script was used to reproduce random bugs triggered by jtreg 
>>>>>> tests for our mips-port.
>>>>>> Yes, we do rm JT* before the test, but would not like to do that 
>>>>>> during the loop.
>>>>>>
>>>>>> It would take much more time to reproduce and fix bugs like that 
>>>>>> if we had to rm JT* every time.
>>>>>> So I think it's worth fixing it.
>>>>>
>>>>> You're likely to find other tests that also have issues being 
>>>>> re-run this way. For example I just found 
>>>>> runtime/appcds/DirClasspathTest.
>>>>>
>>>>> I was somewhat surprised to find that not deleting the work 
>>>>> directory saved about 33% on the runtime (for the appcds tests: 
>>>>> 18m vs. 29m).
>>>>>
>>>>> So this fix seems okay to me - and it won't hurt anyone who does 
>>>>> always clear JTwork first.
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>> Thanks a lot.
>>>>>>
>>>>>> Best regards,
>>>>>> Jie
>>>>>>
>>>>>>
>>>>>> On 2019/3/29 上午5:50, Mikhailo Seledtsov wrote:
>>>>>>> Hi Jie, David,
>>>>>>>
>>>>>>> I did a bit of digging:
>>>>>>>
>>>>>>>    - @run driver ClassFileInstaller -jar whitebox.jar 
>>>>>>> sun.hotspot.WhiteBox sun.hotspot.WhiteBox$WhiteBoxPermission
>>>>>>>      is used in TestCPUSets.java, TestMemoryAwareness.java and 
>>>>>>> TestMisc.java
>>>>>>>      This instruction will generate whitebox.jar and place it in 
>>>>>>> "current working jtreg directory", which is
>>>>>>>      ./JTwork/scratch/
>>>>>>>
>>>>>>>    - in order to be able to use the WhiteBox.jar from within the 
>>>>>>> Docker container, it is copied
>>>>>>>      to the "JTReg test classes" directory (defined by 
>>>>>>> jdk.test.lib.utils.TEST_CLASSES)
>>>>>>>      This Utils.TEST_CLASSES directory is mounted via --volume 
>>>>>>> at docker-run-time, so that classes can be
>>>>>>>      visible/usable/loadable from within the container.
>>>>>>>
>>>>>>>    -  This Utils.TEST_CLASSES directory is unique for each test 
>>>>>>> run:
>>>>>>> ./JTwork/classes/runtime/containers/docker/TestMisc.d/whitebox.jar
>>>>>>> ./JTwork/classes/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>>>
>>>>>>> ./JTwork/classes/runtime/containers/docker/TestCPUSets.d/whitebox.jar 
>>>>>>>
>>>>>>>       Hence, I do not see a possibility of file access clash 
>>>>>>> among the different tests.
>>>>>>>
>>>>>>>    - if tests are run concurrently (with -conc:X feature), each 
>>>>>>> test gets their own scratch directory
>>>>>>>      classes directories are also separated, as in 
>>>>>>> non-concurrent case
>>>>>>>      ./JTwork/scratch/1/whitebox.jar
>>>>>>>      ./JTwork/scratch/0/whitebox.jar
>>>>>>>      ./JTwork/scratch/2/whitebox.jar
>>>>>>> ./JTwork/classes/3/runtime/containers/docker/TestCPUSets.d/whitebox.jar 
>>>>>>>
>>>>>>> ./JTwork/classes/4/runtime/containers/docker/TestMisc.d/whitebox.jar 
>>>>>>>
>>>>>>> ./JTwork/classes/0/runtime/containers/docker/TestMemoryAwareness.d/whitebox.jar 
>>>>>>>
>>>>>>>      No file access clashes among tests either
>>>>>>>
>>>>>>>    Is it possible that you did not clean out the directory from 
>>>>>>> the previous run? For test execution, we recommend
>>>>>>>    clearning JT* directories from the prior run before a new run 
>>>>>>> (rm -Rf JT* or similar). This recommendation is not
>>>>>>>    specific to Docker testing, it is a good practice in general 
>>>>>>> for executing of jtreg tests.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thank you,
>>>>>>> Misha
>>>>>>>
>>>>>>> On 3/28/19, 4:16 AM, David Holmes wrote:
>>>>>>>> Hi Jie,
>>>>>>>>
>>>>>>>> On 28/03/2019 5:48 pm, Jie Fu wrote:
>>>>>>>>> Hi David,
>>>>>>>>>
>>>>>>>>> Thanks for your review.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Normally to use WhiteBox a test specifies:
>>>>>>>>>>
>>>>>>>>>>  @run driver ClassFileInstaller sun.hotspot.WhiteBox
>>>>>>>>> Yes, we do have it in 
>>>>>>>>> test/hotspot/jtreg/runtime/containers/docker/TestCPUSets.java
>>>>>>>>
>>>>>>>> Okay I'm not understanding why Common is copying the WhiteBox 
>>>>>>>> classes or where it is copying them to? If this is something to 
>>>>>>>> do with the image shouldn't it be completely clean when the 
>>>>>>>> test is run - either by deleting existing files or by ensuring 
>>>>>>>> a unique location to begin with?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> David
>>>>>>>>
>>>>>>>>> Best regard,
>>>>>>>>> Jie
>>>>>>>>>>
>>>>>>>>>> Is there a reason this test can't use that?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> David
>>>>>>>>>> -----
>>>>>>>>>
>>>>>>
>>>>
>>>
>>
>