From martinrb at google.com  Mon Jan  9 18:43:44 2017
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 9 Jan 2017 10:43:44 -0800
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
Message-ID: <CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>

Relatedly, I'm writing whitebox jtreg tests and would like to use
VarHandles to access internal data structures.
Because I have:

 * @modules java.base/java.util.concurrent:open

I can use reflection with setAccessible to obtain a usable Field, but if I
try to turn that into a VarHandle I get:

        Field f = ConcurrentLinkedQueue.class.getDeclaredField("head");
        f.setAccessible(true);
        VarHandle v = java.lang.invoke.MethodHandles.lookup()
            .unreflectVarHandle(f);

 java.lang.IllegalAccessException: member is private to package:
java.util.concurrent.ConcurrentLinkedQueue.head/java.util.concurrent.ConcurrentLinkedQueue$Node/getField,
from WhiteBox (unnamed module @7a187f14)

It seems wrong to have this sort of mismatch - setAccessible works, but the
strictly more powerful VarHandles remain unobtainable.

I happen to have a workaround, but I don't see a general solution for
whitebox testing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/12000fb8/attachment.html>

From uschindler at apache.org  Mon Jan  9 18:49:44 2017
From: uschindler at apache.org (Uwe Schindler)
Date: Mon, 09 Jan 2017 18:49:44 +0000
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
	<CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
Message-ID: <CE6B1E4C-565A-4EAC-98F8-EC18E6F09F79@apache.org>

Hi,

It's also inconsistent, because the same type of code works with MethodHandles. If you make something accessible (and succeed doing this), it uses some special internal lookup that has no restrictions for unreflect. 

Uwe

Am 9. Januar 2017 19:43:44 MEZ schrieb Martin Buchholz <martinrb at google.com>:
>Relatedly, I'm writing whitebox jtreg tests and would like to use
>VarHandles to access internal data structures.
>Because I have:
>
> * @modules java.base/java.util.concurrent:open
>
>I can use reflection with setAccessible to obtain a usable Field, but
>if I
>try to turn that into a VarHandle I get:
>
>        Field f = ConcurrentLinkedQueue.class.getDeclaredField("head");
>        f.setAccessible(true);
>        VarHandle v = java.lang.invoke.MethodHandles.lookup()
>            .unreflectVarHandle(f);
>
> java.lang.IllegalAccessException: member is private to package:
>java.util.concurrent.ConcurrentLinkedQueue.head/java.util.concurrent.ConcurrentLinkedQueue$Node/getField,
>from WhiteBox (unnamed module @7a187f14)
>
>It seems wrong to have this sort of mismatch - setAccessible works, but
>the
>strictly more powerful VarHandles remain unobtainable.
>
>I happen to have a workaround, but I don't see a general solution for
>whitebox testing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/461e2bd6/attachment-0001.html>

From paul.sandoz at oracle.com  Mon Jan  9 19:29:21 2017
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Mon, 9 Jan 2017 11:29:21 -0800
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
	<CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
Message-ID: <58DC5C59-1F78-450F-A43A-B8578B77CF9E@oracle.com>

Hi Martin,

Have you tried using:

 MethodHandles.privateLookupIn(ConcurrentLinkedQueue.class, MethodHandles.lookup()).
    findVarHandle(?)

?

I agree there is some inconsistency here, and we wanted to discourage the use of setAccessible. One problem is setAccessible conflates accessibility with stomping on final fields. If we do allow setAccessible access to work for unreflecting i would still not want to allow VarHandles to stomp on final fields, so some inconsistency would remain.

My preference would be to encourage people to use MethodHandles.privateLookupIn.

Paul.


> On 9 Jan 2017, at 10:43, Martin Buchholz <martinrb at google.com> wrote:
> 
> Relatedly, I'm writing whitebox jtreg tests and would like to use VarHandles to access internal data structures.
> Because I have:
> 
>  * @modules java.base/java.util.concurrent:open
> 
> I can use reflection with setAccessible to obtain a usable Field, but if I try to turn that into a VarHandle I get:
> 
>         Field f = ConcurrentLinkedQueue.class.getDeclaredField("head");
>         f.setAccessible(true);
>         VarHandle v = java.lang.invoke.MethodHandles.lookup()
>             .unreflectVarHandle(f);
> 
>  java.lang.IllegalAccessException: member is private to package: java.util.concurrent.ConcurrentLinkedQueue.head/java.util.concurrent.ConcurrentLinkedQueue$Node/getField, from WhiteBox (unnamed module @7a187f14)
> 
> It seems wrong to have this sort of mismatch - setAccessible works, but the strictly more powerful VarHandles remain unobtainable.
> 
> I happen to have a workaround, but I don't see a general solution for whitebox testing.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/e079618f/signature.asc>

From martinrb at google.com  Mon Jan  9 21:27:18 2017
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 9 Jan 2017 13:27:18 -0800
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <58DC5C59-1F78-450F-A43A-B8578B77CF9E@oracle.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
	<CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
	<58DC5C59-1F78-450F-A43A-B8578B77CF9E@oracle.com>
Message-ID: <CA+kOe08MkALjjxR8X+7EjanofCPw2PtxFh_7ZiwEJHtwN1=c0A@mail.gmail.com>

On Mon, Jan 9, 2017 at 11:29 AM, Paul Sandoz <paul.sandoz at oracle.com> wrote:

> Hi Martin,
>
> Have you tried using:
>
>  MethodHandles.privateLookupIn(ConcurrentLinkedQueue.class,
> MethodHandles.lookup()).
>     findVarHandle(?)
>
> ?
>

Thanks!  privateLookupIn was not in my toolbox.  I'm happy with:

    final VarHandle HEAD, TAIL, ITEM, NEXT;

    public WhiteBox() throws ReflectiveOperationException {
        Class<?> qClass = LinkedTransferQueue.class;
        Class<?> nodeClass = Class.forName(qClass.getName() + "$Node");
        MethodHandles.Lookup lookup
            = MethodHandles.privateLookupIn(qClass, MethodHandles.lookup());
        HEAD = lookup.findVarHandle(qClass, "head", nodeClass);
        TAIL = lookup.findVarHandle(qClass, "tail", nodeClass);
        NEXT = lookup.findVarHandle(nodeClass, "next", nodeClass);
        ITEM = lookup.findVarHandle(nodeClass, "item", Object.class);
    }

    Object head(LinkedTransferQueue q) { return HEAD.getVolatile(q); }
    Object tail(LinkedTransferQueue q) { return TAIL.getVolatile(q); }
    Object item(Object node)           { return ITEM.getVolatile(node); }
    Object next(Object node)           { return NEXT.getVolatile(node); }

although programming this way does not feel very Java-esque.  I miss being
able to use a Node type at compile time.

I agree there is some inconsistency here, and we wanted to discourage the
> use of setAccessible. One problem is setAccessible conflates accessibility
> with stomping on final fields. If we do allow setAccessible access to work
> for unreflecting i would still not want to allow VarHandles to stomp on
> final fields, so some inconsistency would remain.
>

I agree with your decision to not give unreflected VarHandles the same
unconstrained superpowers that the Field has.

My whitebox tests tend to use private access only for reading, but I can
imagine cases where it is useful to give testing code stronger access than
for regular VarHandles, maybe even to write final fields, for example for
fuzz testing or reliably creating an internal state that can only happen
with a race.  There's may be a case for providing such a big hammer, if
it's sufficiently hard to get at, but I can live without it.


> My preference would be to encourage people to use
> MethodHandles.privateLookupIn.
>

I'm a convert! It will take a while for privateLookupIn to become "common
knowledge".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/ad7eb98e/attachment.html>

From paul.sandoz at oracle.com  Mon Jan  9 22:07:02 2017
From: paul.sandoz at oracle.com (Paul Sandoz)
Date: Mon, 9 Jan 2017 14:07:02 -0800
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <CA+kOe08MkALjjxR8X+7EjanofCPw2PtxFh_7ZiwEJHtwN1=c0A@mail.gmail.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
	<CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
	<58DC5C59-1F78-450F-A43A-B8578B77CF9E@oracle.com>
	<CA+kOe08MkALjjxR8X+7EjanofCPw2PtxFh_7ZiwEJHtwN1=c0A@mail.gmail.com>
Message-ID: <0E07F1EF-F5D1-464D-8901-B4C86529DA2A@oracle.com>


> On 9 Jan 2017, at 13:27, Martin Buchholz <martinrb at google.com> wrote:
> 
> My whitebox tests tend to use private access only for reading, but I can imagine cases where it is useful to give testing code stronger access than for regular VarHandles, maybe even to write final fields, for example for fuzz testing or reliably creating an internal state that can only happen with a race.  There's may be a case for providing such a big hammer, if it's sufficiently hard to get at, but I can live without it.
> 

That big hammer could be jdk.internal.misc.Unsafe for certain tightly bound tests that really need it.


> My preference would be to encourage people to use MethodHandles.privateLookupIn.
> 
> I'm a convert!

Great!


> It will take a while for privateLookupIn to become "common knowledge".
> 

I shall endeavour to mention it at Devoxx US in March where John and I will present on java.lang.invoke.

Paul.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/c1bb88a8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/c1bb88a8/signature.asc>

From martinrb at google.com  Mon Jan  9 22:45:45 2017
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 9 Jan 2017 14:45:45 -0800
Subject: jdk.internal.reflect.ReflectionFactory and SecurityManager
In-Reply-To: <0E07F1EF-F5D1-464D-8901-B4C86529DA2A@oracle.com>
References: <2a6d2f85-b3c9-0658-6684-34c1a90aa22b@gmail.com>
	<2584fe72-ccf8-30d3-72bc-7cad0a6b7845@oracle.com>
	<0e2b9fdc-8b7d-73c9-907d-ffe987b5b8fd@gmail.com>
	<a380b9e2-7ac8-ad0a-2589-83ea40c42160@oracle.com>
	<EA94713D-F669-4BC3-A872-42372FF38FB2@oracle.com>
	<d78b60b9-2a8d-aea4-4aac-916c91791586@oracle.com>
	<CA+kOe09tSLRR_DRDMf7fzQGRpf9uG4RMm=srKGWEbU3FSTaTbA@mail.gmail.com>
	<58DC5C59-1F78-450F-A43A-B8578B77CF9E@oracle.com>
	<CA+kOe08MkALjjxR8X+7EjanofCPw2PtxFh_7ZiwEJHtwN1=c0A@mail.gmail.com>
	<0E07F1EF-F5D1-464D-8901-B4C86529DA2A@oracle.com>
Message-ID: <CA+kOe0_LEQgjQXw5WvR=BKtDjL5N-P0gDXqGEvCYqNn1bRyvHA@mail.gmail.com>

I followed up on my own suggestion and wrote a race-simulating test without
the big hammer:

    /**
     * Checks that traversal operations collapse a random pattern of
     * dead nodes as could normally only occur with a race.
     */
    @Test(dataProvider = "traversalActions")
    public void traversalOperationsCollapseNodes(
        Consumer<LinkedTransferQueue> traversalAction) {
        LinkedTransferQueue q = new LinkedTransferQueue();
        int n = rnd.nextInt(6);
        for (int i = 0; i < n; i++) q.add(i);
        ArrayList nulledOut = new ArrayList();
        for (Object p = head(q); p != null; p = next(p))
            if (rnd.nextBoolean()) {
                nulledOut.add(item(p));
                ITEM.setVolatile(p, null);
            }
        traversalAction.accept(q);
        if (n == 0)
            assertEquals(nodeCount(q), 0);
        else {
            int c = nodeCount(q);
            assertEquals(q.size(), c - (q.contains(n - 1) ? 0 : 1));
            for (int i = 0; i < n; i++)
                assertTrue(nulledOut.contains(i) ^ q.contains(i));
        }
    }


On Mon, Jan 9, 2017 at 2:07 PM, Paul Sandoz <paul.sandoz at oracle.com> wrote:

>
> On 9 Jan 2017, at 13:27, Martin Buchholz <martinrb at google.com> wrote:
>
> My whitebox tests tend to use private access only for reading, but I can
> imagine cases where it is useful to give testing code stronger access than
> for regular VarHandles, maybe even to write final fields, for example for
> fuzz testing or reliably creating an internal state that can only happen
> with a race.  There's may be a case for providing such a big hammer, if
> it's sufficiently hard to get at, but I can live without it.
>
>
>
> That big hammer could be jdk.internal.misc.Unsafe for certain tightly
> bound tests that really need it.
>
>
> My preference would be to encourage people to use
>> MethodHandles.privateLookupIn.
>>
>
> I'm a convert!
>
>
> Great!
>
>
> It will take a while for privateLookupIn to become "common knowledge".
>
>
> I shall endeavour to mention it at Devoxx US in March where John and I
> will present on java.lang.invoke.
>
> Paul.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170109/33c0e06f/attachment-0001.html>

From christoph.langer at sap.com  Thu Jan 19 08:42:44 2017
From: christoph.langer at sap.com (Langer, Christoph)
Date: Thu, 19 Jan 2017 08:42:44 +0000
Subject: Current version of testng
Message-ID: <26d2ae65d054464ea9945ad59c16f104@derote13de46.global.corp.sap>

Hi,

can you still confirm that the recommended/used testng version for jtreg used by OpenJDK 9 is still 6.9.5, as per [1]? Or has this changed in the meanwhile?

Thanks
Christoph

[1] http://openjdk.java.net/jtreg/build.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/jtreg-use/attachments/20170119/fb29ea10/attachment.html>