List extending Collection/SequencedCollection

[Ryan Ernst]

Hi Joe and Stuart,

Given the inconsistencies mentioned, I see how this change may not be worth the hassle, so I’ll drop it. I appreciate the thoughtful responses to explain your reasoning. 

Thanks!
Ryan 


> On Jul 7, 2023, at 4:21 PM, Stuart Marks <stuart.marks at oracle.com> wrote:
> 
> Hi Ryan,
> 
> Thanks for trying out JDK 21 early access.
> 
> The issue you raise is indeed an inconsistency, but it's not clear which way it should be resolved, or even whether it needs to be resolved, as the consequences are quite minor.
> 
> Specifically, when the Sequenced Collections JEP was integrated, it included these changes (edited for brevity):
> 
> - public interface List<E> extends Collection<E> {
> + public interface List<E> extends SequencedCollection<E> {
> 
> - public interface SortedSet<E> extends Set<E> {
> + public interface SortedSet<E> extends Set<E>, SequencedSet<E> {
> 
> As you observed, on List, SequencedCollection has replaced Collection, but on SortedSet, SequencedSet was added alongside Set. Which way is correct? Should SequencedCollection have been added to List, instead of replacing Collection? Or on SortedSet, should SequencedSet have replaced Set instead of simply being added?
> 
> (I have to admit my first inclination was that the SortedSet change was a mistake, and that SequencedSet ought to have replaced Set. I think the reason it turned out the way it did was that, my earliest prototype, before I published anything, had a single interface OrderedCollection. This was retrofitted onto SortedSet as
> 
>    public interface SortedSet<E> extends Set<E>, OrderedCollection<E> {
> 
> Only later did I decide to add OrderedSet, and so I changed the declaration to
> 
>    public interface SortedSet<E> extends Set<E>, OrderedSet<E> {
> 
> and I didn't notice that the interfaces could be minimized. And of course there were a couple rounds of renaming subsequent to this.)
> 
> In any case, this inconsistency is of little consequence, as the members available to users of the interfaces in question are the same regardless of the way the interfaces are declared. This is true from both a source and binary compatibility standpoint.
> 
> Joe's point about compatibility is that even though such changes are *visible* to reflective code, they aren't *incompatible*. There are many compatible changes possible, such as moving the declaration of a method upward in the hierarchy. From the standpoint of reflection, that method may appear to have been removed from some interface; but that doesn't mean it's incompatible. Reflective code needs to be adjusted to take such things into account. The presence or absence of Collection as a superinterface of List is a similar case.
> 
> Perhaps in some sense the JDK itself ought to be more consistent about this. We have for example this:
> 
>    class HashSet extends AbstractSet implements Set, Cloneable, Serializable
> 
> but also this:
> 
>    class EnumSet extends AbstractSet implements Cloneable, Serializable
> 
> (That is, EnumSet is "missing" Set.)
> 
> Questions about this have been asked on Stack Overflow, and various answers there have made up a rationale about the possibly-redundant interfaces being included explicitly because it expresses intent more clearly, or some such. My own guess is that nobody has paid much attention to this issue because resolving it one way or another hardly makes any practical difference.
> 
> s'marks
> 
>> On 7/7/23 7:25 AM, Ryan Ernst wrote:
>> Thanks for laying out your thinking, Joe. I will watch your talks.
>> If I understood your response correctly, you are ok making such a change, especially since it is semantically equivalent? If that’s the case, is JDK 21 past the point of feature release, or should the change target only 22? It doesn’t matter much to me, but I thought since SequencedCollection is added in 21 it would be good to “fix” this there so as to avoid a gap in behavior.
>>>> On Jun 30, 2023, at 6:13 PM, Joseph D. Darcy <joe.darcy at oracle.com> wrote:
>>> 
>>> Hi Ryan,
>>> 
>>> Apropos of this discussion, I happened to recently give a talk to the JCP that in part covered behavioral compatibility in the JDK:
>>> 
>>> https://jcp.org/aboutJava/communityprocess/ec-public/materials/2023-06-13/JCP-EC-Public-Agenda-June-2023.html
>>> https://jcp.org/aboutJava/communityprocess/ec-public/materials/2023-06-13/Contributing_to_OpenJDK_2023_04_12.pdf
>>> 
>>> There are many observable details of the JDK implementation, including reflective details, timing, etc. there are _not_ part of the interfaces users should rely on.
>>> 
>>> My current evaluation here is that it would set an unfortunate precedent to preclude making the sort of source changes in questions because of (potential) compatibility concerns in reflective modeling, especially in a feature release. (IMO, there is a stronger argument for not making such a change in an update release, but even there as the change is a semantically equivalent refactoring, I think it is generally fine.)
>>> 
>>> HTH,
>>> 
>>> -Joe
>>> 
>>>> On 6/29/2023 11:30 AM, Ryan Ernst wrote:
>>>> Thanks for replying, Joe. First, let me reiterate, we fully admit
>>>> there was a bug in painless, we stopped short in walking the class
>>>> hierarchy. There is no bug in the SequencedCollection hierarchy. But I
>>>> do think there is an inconsistency.
>>>> 
>>>>> The two definition are semantically equivalent
>>>>> ...
>>>>> The JDK 22 javadoc for List shows:
>>>>>> All Superinterfaces:
>>>>>>     Collection<E>, Iterable<E>, SequencedCollection<E>
>>>> While that is true, in practice the reflection API does not give the
>>>> same collapsed view that javadocs do. Calling getInterfaces() on a
>>>> class only returns direct super interfaces, so
>>>> List.class.getInterfaces() no longer returns Collection.class in JDK
>>>> 21.
>>>> 
>>>> The inconsistency I see is that SortedSet.class.getInterfaces() *does*
>>>> still return Set.class. Was that intentional? It seems like either
>>>> SortedSet does not need to extend Set directly, or List should still
>>>> extend Collection directly. And doing the latter would provide an
>>>> extra layer of "compatibility" for applications that may look at
>>>> direct super interfaces, and be surprised that Collection disappeared
>>>> across JDK versions for List.
>>>> 
>>>>> On Wed, Jun 28, 2023 at 6:31 PM Joseph D. Darcy <joe.darcy at oracle.com> wrote:
>>>>> Hello,
>>>>> 
>>>>> What is Painless doing that would fail under
>>>>> 
>>>>>      List extends SequencedCollection ...
>>>>> 
>>>>> but work under
>>>>> 
>>>>>      List extends SequencedCollection, Collection ...
>>>>> 
>>>>> The two definition are semantically equivalent since SequencedCollection
>>>>> itself extends Collection.
>>>>> 
>>>>> The JDK 22 javadoc for List shows:
>>>>> 
>>>>>> All Superinterfaces:
>>>>>>     Collection<E>, Iterable<E>, SequencedCollection<E>
>>>>> There are certainly implementation artifacts concerning the details of
>>>>> how a type was declared that exposed via core reflection (for the
>>>>> javax.lang.model API during annotation processing at compile time), but
>>>>> it is a bug for third party programs to rely on such details.
>>>>> 
>>>>> HTH,
>>>>> 
>>>>> -Joe
>>>>> 
>>>>> On 6/27/2023 7:37 AM, Ryan Ernst wrote:
>>>>>> Hi core-libs-dev,
>>>>>> 
>>>>>> I know various threads have existed over the past few months on
>>>>>> SequencedCollection and its implications on compatibility. I wanted to
>>>>>> raise this semi-related issue we noticed recently after beginning
>>>>>> testing against Java 21.
>>>>>> 
>>>>>> Elasticsearch began testing against Java 21, and noticed a series of
>>>>>> failures in Painless (our builtin scripting language, which
>>>>>> dynamically compiles to bytecode). Most tests that used List failed to
>>>>>> compile, with several unknown methods (eg add). Painless builds a
>>>>>> hierarchy of classes it knows about for method resolution. This
>>>>>> hierarchy didn't know anything about SequencedCollection since we
>>>>>> currently compile against Java 17. Now, this was ultimately a bug in
>>>>>> Painless, because we knew there could be cases where we encounter
>>>>>> unknown classes in the hierarchy (eg a class which is not blessed to
>>>>>> be visible in the language). However, I think the reason we found that
>>>>>> bug (List extending from SequencedCollection) might still cause issues
>>>>>> for some.
>>>>>> 
>>>>>> While debugging the issue, something that struck me as interesting in
>>>>>> the SequencedCollection hierarchy is the difference between List and
>>>>>> SortedSet. Both of these classes were made to be compatible with
>>>>>> sequenced classes by adding the new classes as super interfaces,
>>>>>> SequencedCollection and SequencedSet, respectively. However, while
>>>>>> SortedSet was *added* as a super interface, List had its super
>>>>>> interface Collection *replaced* by SequencedCollection.
>>>>>> 
>>>>>> I don't know enough about the rampdown process to know if this is
>>>>>> still fixable in Java 21. It is probably an extreme edge case that
>>>>>> doesn't matter, since eg instanceof Collection will still work in
>>>>>> existing code. But for consistency, it would seem better to maintain
>>>>>> Collection as a direct super interface of List. Is there any reason
>>>>>> such a change doesn't fit with the collections architecture going
>>>>>> forward?