List extending Collection/SequencedCollection

Fri Jul 7 23:21:46 UTC 2023

Hi Ryan,

Thanks for trying out JDK 21 early access.

The issue you raise is indeed an inconsistency, but it's not clear which way it 
should be resolved, or even whether it needs to be resolved, as the consequences are 
quite minor.

Specifically, when the Sequenced Collections JEP was integrated, it included these 
changes (edited for brevity):

- public interface List<E> extends Collection<E> {
+ public interface List<E> extends SequencedCollection<E> {

- public interface SortedSet<E> extends Set<E> {
+ public interface SortedSet<E> extends Set<E>, SequencedSet<E> {

As you observed, on List, SequencedCollection has replaced Collection, but on 
SortedSet, SequencedSet was added alongside Set. Which way is correct? Should 
SequencedCollection have been added to List, instead of replacing Collection? Or on 
SortedSet, should SequencedSet have replaced Set instead of simply being added?

(I have to admit my first inclination was that the SortedSet change was a mistake, 
and that SequencedSet ought to have replaced Set. I think the reason it turned out 
the way it did was that, my earliest prototype, before I published anything, had a 
single interface OrderedCollection. This was retrofitted onto SortedSet as

     public interface SortedSet<E> extends Set<E>, OrderedCollection<E> {

Only later did I decide to add OrderedSet, and so I changed the declaration to

     public interface SortedSet<E> extends Set<E>, OrderedSet<E> {

and I didn't notice that the interfaces could be minimized. And of course there were 
a couple rounds of renaming subsequent to this.)

In any case, this inconsistency is of little consequence, as the members available 
to users of the interfaces in question are the same regardless of the way the 
interfaces are declared. This is true from both a source and binary compatibility 
standpoint.

Joe's point about compatibility is that even though such changes are *visible* to 
reflective code, they aren't *incompatible*. There are many compatible changes 
possible, such as moving the declaration of a method upward in the hierarchy. From 
the standpoint of reflection, that method may appear to have been removed from some 
interface; but that doesn't mean it's incompatible. Reflective code needs to be 
adjusted to take such things into account. The presence or absence of Collection as 
a superinterface of List is a similar case.

Perhaps in some sense the JDK itself ought to be more consistent about this. We have 
for example this:

     class HashSet extends AbstractSet implements Set, Cloneable, Serializable

but also this:

     class EnumSet extends AbstractSet implements Cloneable, Serializable

(That is, EnumSet is "missing" Set.)

Questions about this have been asked on Stack Overflow, and various answers there 
have made up a rationale about the possibly-redundant interfaces being included 
explicitly because it expresses intent more clearly, or some such. My own guess is 
that nobody has paid much attention to this issue because resolving it one way or 
another hardly makes any practical difference.

s'marks

On 7/7/23 7:25 AM, Ryan Ernst wrote:
> Thanks for laying out your thinking, Joe. I will watch your talks.
> 
> If I understood your response correctly, you are ok making such a change, especially since it is semantically equivalent? If that’s the case, is JDK 21 past the point of feature release, or should the change target only 22? It doesn’t matter much to me, but I thought since SequencedCollection is added in 21 it would be good to “fix” this there so as to avoid a gap in behavior.
> 
> 
> 
>> On Jun 30, 2023, at 6:13 PM, Joseph D. Darcy <joe.darcy at oracle.com> wrote:
>>
>> Hi Ryan,
>>
>> Apropos of this discussion, I happened to recently give a talk to the JCP that in part covered behavioral compatibility in the JDK:
>>
>> https://jcp.org/aboutJava/communityprocess/ec-public/materials/2023-06-13/JCP-EC-Public-Agenda-June-2023.html
>> https://jcp.org/aboutJava/communityprocess/ec-public/materials/2023-06-13/Contributing_to_OpenJDK_2023_04_12.pdf
>>
>> There are many observable details of the JDK implementation, including reflective details, timing, etc. there are _not_ part of the interfaces users should rely on.
>>
>> My current evaluation here is that it would set an unfortunate precedent to preclude making the sort of source changes in questions because of (potential) compatibility concerns in reflective modeling, especially in a feature release. (IMO, there is a stronger argument for not making such a change in an update release, but even there as the change is a semantically equivalent refactoring, I think it is generally fine.)
>>
>> HTH,
>>
>> -Joe
>>
>>> On 6/29/2023 11:30 AM, Ryan Ernst wrote:
>>> Thanks for replying, Joe. First, let me reiterate, we fully admit
>>> there was a bug in painless, we stopped short in walking the class
>>> hierarchy. There is no bug in the SequencedCollection hierarchy. But I
>>> do think there is an inconsistency.
>>>
>>>> The two definition are semantically equivalent
>>>> ...
>>>> The JDK 22 javadoc for List shows:
>>>>> All Superinterfaces:
>>>>>      Collection<E>, Iterable<E>, SequencedCollection<E>
>>> While that is true, in practice the reflection API does not give the
>>> same collapsed view that javadocs do. Calling getInterfaces() on a
>>> class only returns direct super interfaces, so
>>> List.class.getInterfaces() no longer returns Collection.class in JDK
>>> 21.
>>>
>>> The inconsistency I see is that SortedSet.class.getInterfaces() *does*
>>> still return Set.class. Was that intentional? It seems like either
>>> SortedSet does not need to extend Set directly, or List should still
>>> extend Collection directly. And doing the latter would provide an
>>> extra layer of "compatibility" for applications that may look at
>>> direct super interfaces, and be surprised that Collection disappeared
>>> across JDK versions for List.
>>>
>>>> On Wed, Jun 28, 2023 at 6:31 PM Joseph D. Darcy <joe.darcy at oracle.com> wrote:
>>>> Hello,
>>>>
>>>> What is Painless doing that would fail under
>>>>
>>>>       List extends SequencedCollection ...
>>>>
>>>> but work under
>>>>
>>>>       List extends SequencedCollection, Collection ...
>>>>
>>>> The two definition are semantically equivalent since SequencedCollection
>>>> itself extends Collection.
>>>>
>>>> The JDK 22 javadoc for List shows:
>>>>
>>>>> All Superinterfaces:
>>>>>      Collection<E>, Iterable<E>, SequencedCollection<E>
>>>> There are certainly implementation artifacts concerning the details of
>>>> how a type was declared that exposed via core reflection (for the
>>>> javax.lang.model API during annotation processing at compile time), but
>>>> it is a bug for third party programs to rely on such details.
>>>>
>>>> HTH,
>>>>
>>>> -Joe
>>>>
>>>> On 6/27/2023 7:37 AM, Ryan Ernst wrote:
>>>>> Hi core-libs-dev,
>>>>>
>>>>> I know various threads have existed over the past few months on
>>>>> SequencedCollection and its implications on compatibility. I wanted to
>>>>> raise this semi-related issue we noticed recently after beginning
>>>>> testing against Java 21.
>>>>>
>>>>> Elasticsearch began testing against Java 21, and noticed a series of
>>>>> failures in Painless (our builtin scripting language, which
>>>>> dynamically compiles to bytecode). Most tests that used List failed to
>>>>> compile, with several unknown methods (eg add). Painless builds a
>>>>> hierarchy of classes it knows about for method resolution. This
>>>>> hierarchy didn't know anything about SequencedCollection since we
>>>>> currently compile against Java 17. Now, this was ultimately a bug in
>>>>> Painless, because we knew there could be cases where we encounter
>>>>> unknown classes in the hierarchy (eg a class which is not blessed to
>>>>> be visible in the language). However, I think the reason we found that
>>>>> bug (List extending from SequencedCollection) might still cause issues
>>>>> for some.
>>>>>
>>>>> While debugging the issue, something that struck me as interesting in
>>>>> the SequencedCollection hierarchy is the difference between List and
>>>>> SortedSet. Both of these classes were made to be compatible with
>>>>> sequenced classes by adding the new classes as super interfaces,
>>>>> SequencedCollection and SequencedSet, respectively. However, while
>>>>> SortedSet was *added* as a super interface, List had its super
>>>>> interface Collection *replaced* by SequencedCollection.
>>>>>
>>>>> I don't know enough about the rampdown process to know if this is
>>>>> still fixable in Java 21. It is probably an extreme edge case that
>>>>> doesn't matter, since eg instanceof Collection will still work in
>>>>> existing code. But for consistency, it would seem better to maintain
>>>>> Collection as a direct super interface of List. Is there any reason
>>>>> such a change doesn't fit with the collections architecture going
>>>>> forward?