RFR: 8227642: [TESTBUG] Make docker tests podman compatible

Tue Jul 16 21:11:13 UTC 2019

Severin,

This might be a reasonable update to jtreg, to allow /usr/sbin on the path
on Unix-like systems.  The intent of jtreg is to protect tests from random
crufty stuff on the PATH, and /usr/sbin is not in that category.

I've created
CODETOOLS-7902505: Consider allowing /usr/sbin on $PATH
https://bugs.openjdk.java.net/browse/CODETOOLS-7902505

The short-term workaround is to use the jtreg command-line option

     -e:PATH

which should override the default settign for PATH and pass through
whatever you have set for $PATH.

-- Jon

On 07/16/2019 02:01 PM, Severin Gehwolf wrote:
> Hi Igor,
>
> I understand the concern and I guess I could remove it and locally
> install a wrapper in /bin or /usr/bin for podman which adds /usr/sbin
> to the path. On the other hand...
>
> This seems to be an issue of code being run through jtreg. Looking at
> the jtr files I see this:
>
> ----------rerun:(21/1545)*----------
> cd /disk/openjdk/upstream-sources/openjdk-head/JTwork/scratch && \\
> DISPLAY=:0 \\
> HOME=/home/sgehwolf \\
> LANG=en_US.UTF-8 \\
> PATH=/bin:/usr/bin \\
> XMODIFIERS=@im=ibus \\
> [...]
>
> So jtreg reduces the host's PATH to /bin:/usr/bin, which is
> insufficient for the podman case. The tag-spec docs[1] for jtreg
> mention for "shell" tests that it sets the PATH to the above settings.
> This affects Java tests as well it seems.
>
> ProcessBuilder outside jtreg has a sensible PATH as set up by the
> system, FWIW.
>
> So while my system is properly set up, jtreg interferes and renders
> this necessary. Any suggestions as to how to convince jtreg to use the
> host's PATH setting?
>
> Thanks,
> Severin
>
> [1] http://openjdk.java.net/jtreg/tag-spec.html
>
> On Tue, 2019-07-16 at 13:32 -0700, Igor Ignatyev wrote:
>> Hi Misha,
>>
>> I understand that it doesn't alter the host system. my concern is
>> that we move problem of host-configuration into tests. let's say
>> tomorrow a new container engine will require something from
>> /usr/local/sbin, or /usr/local/Cellar/docker/bin on another OS, or,
>> god forbid, C:\Program Files(x86)\podman\bin. it has nothing to do w/
>> tests, it's a question of configuring a host, as I said, should be
>> handled at another level by "scripts" run (once) prior test
>> execution.
>>
>> -- Igor
>>
>>> On Jul 16, 2019, at 1:23 PM, mikhailo.seledtsov at oracle.com wrote:
>>>
>>> Hi Igor,
>>>
>>>     In both cases the environment variable is set for the
>>> Docker/Podman container process, not the host system. This will not
>>> affect the host system in any way. The docker process has its own
>>> namespace for environment variables. Does this alleviate your
>>> concerns?
>>>
>>>
>>> Thank you,
>>>
>>> Misha
>>>
>>> On 7/16/19 11:49 AM, Igor Ignatyev wrote:
>>>> Hi Severin,
>>>>
>>>> I don't think that tests (or test libraries for that matter)
>>>> should be responsible for setting correct PATH value, it should
>>>> be a part of host configuration procedure (tests can/should check
>>>> that all required bins are available though). in other words, I'd
>>>> prefer if you remove 'env.put("PATH", ...)' lines from both
>>>> DockerTestUtils and TestJFREvents. the rest looks good to me.
>>>>
>>>> Thanks,
>>>> -- Igor
>>>>
>>>>> On Jul 16, 2019, at 5:36 AM, Severin Gehwolf <
>>>>> sgehwolf at redhat.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I believe I still need a *R*eviewer for this. Any takers?
>>>>>
>>>>> Thanks,
>>>>> Severin
>>>>>
>>>>> On Fri, 2019-07-12 at 15:19 -0700,
>>>>> mikhailo.seledtsov at oracle.com wrote:
>>>>>> Hi Severin,
>>>>>>
>>>>>>    The change looks good to me. Thank you for adding support
>>>>>> for Podman
>>>>>> container technology.
>>>>>>
>>>>>> Testing: I ran both HotSpot and JDK container tests with your
>>>>>> patch;
>>>>>> tests executed on Oracle Linux 7.6 using default container
>>>>>> engine (Docker):
>>>>>>
>>>>>>      test/hotspot/jtreg/containers/   AND
>>>>>> test/jdk/jdk/internal/platform/docker/
>>>>>>
>>>>>> All PASS
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Misha
>>>>>>
>>>>>>
>>>>>> On 7/12/19 11:08 AM, Severin Gehwolf wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> There is an alternative container engine which is being
>>>>>>> used by Fedora
>>>>>>> and RHEL 8, called podman[1]. It's mostly compatible with
>>>>>>> docker. It
>>>>>>> looks like OpenJDK docker tests can be made podman
>>>>>>> compatible with a
>>>>>>> few little tweaks. One "interesting" one is to not assert
>>>>>>> "Successfully
>>>>>>> built" in the build output but only rely on the exit code,
>>>>>>> which seems
>>>>>>> to be OK for my testing. Interestingly the test would be
>>>>>>> skipped in
>>>>>>> that case.
>>>>>>>
>>>>>>> Bug: https://bugs.openjdk.java.net/browse/JDK-8227642
>>>>>>> webrev:
>>>>>>> http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8227642/01/webrev/
>>>>>>>
>>>>>>> Adjustments I've done:
>>>>>>>   * Don't assert "Successfully built" in image build
>>>>>>> output[2].
>>>>>>>   * Add /usr/sbin to PATH as the podman binary relies on
>>>>>>> iptables for it
>>>>>>>     to work which is in /usr/sbin on Fedora
>>>>>>>   * Allow for Metrics.getCpuSystemUsage() and
>>>>>>> Metrics.getCpuUserUsage()
>>>>>>>     to be equal to the previous value. I've found those
>>>>>>> counters to be
>>>>>>>     slowly increasing, which made the tests unreliable.
>>>>>>>
>>>>>>> Testing:
>>>>>>>
>>>>>>> Running docker tests with docker as engine. Did the same
>>>>>>> with podman as
>>>>>>> engine via -Djdk.test.docker.command=podman on Linux
>>>>>>> x86_64. Both
>>>>>>> passed (non-trivially).
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Severin
>>>>>>>
>>>>>>> [1] https://podman.io/
>>>>>>> [2] Image builds with podman look
>>>>>>>      like ("COMMIT" over "Successfully built"):
>>>>>>> STEP 1: FROM fedora:29
>>>>>>> STEP 2: RUN dnf install -y java-11-openjdk-devel &&     dnf
>>>>>>> clean all
>>>>>>> --> Using cache
>>>>>>> 96f8b1a0dfe7dba581a64fc67a27002ddf52e032af55f9ddc765182a690
>>>>>>> afd9d
>>>>>>> STEP 3: COPY TestMetrics.class  TestMetrics.java /opt/
>>>>>>> 269042160f7a4e6a06789cd19640ea658a8f941bc53de0fd40a574dc3bd
>>>>>>> b49a8
>>>>>>> STEP 4: CMD /usr/lib/jvm/java-11-openjdk/bin/java -cp /opt
>>>>>>> --add-modules java.base --add-exports
>>>>>>> java.base/jdk.internal.platform=ALL-UNNAMED TestMetrics
>>>>>>> STEP 5: COMMIT fedora-metrics-11
>>>>>>> d749088d6ce4510f212820ad4eca55a9b05e5c5c245f2372b6cfe91926e
>>>>>>> 8cd7e
>>>>>>>