From kevin.l.stern at gmail.com  Thu Apr  1 00:08:12 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Wed, 31 Mar 2010 19:08:12 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
Message-ID: <t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>

I realize that 2 * (2^(k/2) - 1) only works for even numbered superblocks,
the odd numbered superblocks need an additional term added (the number of
data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I also came
across an alternative characterization of superblock in the paper which
states that data blocks are grouped within a superblock when they are the
same size - to me, though, that implies that my example structure below
would be

SB_0: [1]
SB_1: [2][2][2]
SB_2: [4][4][4][4][4][4]

which seems to contradict my understanding of (1) below.  I must be reading
this upside down.

On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:

> What am I missing here?  In "Resizable arrays in optimal time and space"
> the authors define their data structure with the following property:
>
> (1)  "When superblock SB_k is fully allocated, it consists of
> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>
> Since the superblock is zero-based indexed this implies the following
> structure:
>
> SB_0: [1]
> SB_1: [2]
> SB_2: [2][2]
> SB_3: [4][4]
> SB_4: [4][4][4][4]
> [...]
>
> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>
> r = 100 (the binary expansion of i + 1)
> k = |r| - 1 = 2
> p = 2^k - 1 = 3
>
> What concerns me is their statement that p represents "the number of data
> blocks in superblocks prior to SB_k."  There are only two data blocks in
> superblocks prior to SB_2, not three.  Given (1) above, unless I'm
> misinterpreting it, the number of data blocks in superblocks prior to SB_k
> should be:
>
> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>
> This, of course, seems to work out much better in my example above, giving
> the correct answer to my interpretation of their data structure, but I have
> a hard time believing that this is their mistake rather than my
> misinterpretation.
>
> Thoughts?
>
> Kevin
>
>
> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <martinrb at google.com>wrote:
>
>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>> > Hi Martin,
>> >
>> > Thanks much for your feedback.  The first approach that comes to mind to
>> > implement O(1) time front as well as rear insertion is to create a
>> cyclic
>> > list structure with a front/rear pointer - to insert at the front
>> requires
>> > decrementing the front pointer (modulo the size) and to insert at the
>> rear
>> > requires incrementing the rear pointer (modulo the size).  We need to
>> resize
>> > when the two pointers bump into each other.  Could you explain more
>> about
>> > your suggestion of introducing an arraylet that is shared by the front
>> and
>> > the rear?
>>
>> It was a half-baked idea - I don't know if there's a way to turn it into
>> something useful.  I was thinking of the ArrayDeque implementation,
>> where all the elements live in a single array.
>>
>> >  It's not clear to me how that would help and/or be a better
>> > approach than the cyclic list.  Anyhow, the paper that you reference,
>> > "Resizable arrays in optimal time and space", gives a deque so if we
>> take
>> > that approach then the deque is specified.
>>
>> Technically, ArrayList also supports the Deque operations -
>> just not efficiently.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100331/8f364089/attachment.html>

From kevin.l.stern at gmail.com  Thu Apr  1 02:34:15 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Wed, 31 Mar 2010 21:34:15 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
Message-ID: <u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>

I'm almost convinced now that the paper is incorrect.  The code below gives
me the appropriate index into the index array and the offset into the data
block.  That being said, remember when I mentioned that this will include a
bit more work to access an element than a simple bit shift and a bit mask?
Well this is more than a bit more - we'll be doing this each time an index
is requested.  I'll spend some time trying to twiddle the bits to see if I
can eliminate/combine some of the operations.

        for (int r = 1; r < 33; r++) {
            int k = lg(r);
            int floorKO2 = k >> 1;
            int powFloorKO2 = (1 << floorKO2);
            int p = ((1 << floorKO2) - 1) << 1;
            int ceilKO2;
            if ((k & 1) == 1) {
                ceilKO2 = floorKO2 + 1;
                p += powFloorKO2;
            } else {
                ceilKO2 = floorKO2;
            }
            int e = r & ((1 << ceilKO2) - 1);
            int b = (r >> ceilKO2) & (powFloorKO2 - 1);

            System.out.println((r - 1) + " " + (p + b) + " " + e);
        }

Kevin

On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:

> I realize that 2 * (2^(k/2) - 1) only works for even numbered superblocks,
> the odd numbered superblocks need an additional term added (the number of
> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I also came
> across an alternative characterization of superblock in the paper which
> states that data blocks are grouped within a superblock when they are the
> same size - to me, though, that implies that my example structure below
> would be
>
> SB_0: [1]
> SB_1: [2][2][2]
> SB_2: [4][4][4][4][4][4]
>
> which seems to contradict my understanding of (1) below.  I must be reading
> this upside down.
>
>
> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>
>> What am I missing here?  In "Resizable arrays in optimal time and space"
>> the authors define their data structure with the following property:
>>
>> (1)  "When superblock SB_k is fully allocated, it consists of
>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>
>> Since the superblock is zero-based indexed this implies the following
>> structure:
>>
>> SB_0: [1]
>> SB_1: [2]
>> SB_2: [2][2]
>> SB_3: [4][4]
>> SB_4: [4][4][4][4]
>> [...]
>>
>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>
>> r = 100 (the binary expansion of i + 1)
>> k = |r| - 1 = 2
>> p = 2^k - 1 = 3
>>
>> What concerns me is their statement that p represents "the number of data
>> blocks in superblocks prior to SB_k."  There are only two data blocks in
>> superblocks prior to SB_2, not three.  Given (1) above, unless I'm
>> misinterpreting it, the number of data blocks in superblocks prior to SB_k
>> should be:
>>
>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>
>> This, of course, seems to work out much better in my example above, giving
>> the correct answer to my interpretation of their data structure, but I have
>> a hard time believing that this is their mistake rather than my
>> misinterpretation.
>>
>> Thoughts?
>>
>> Kevin
>>
>>
>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <martinrb at google.com>wrote:
>>
>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern <kevin.l.stern at gmail.com>
>>> wrote:
>>> > Hi Martin,
>>> >
>>> > Thanks much for your feedback.  The first approach that comes to mind
>>> to
>>> > implement O(1) time front as well as rear insertion is to create a
>>> cyclic
>>> > list structure with a front/rear pointer - to insert at the front
>>> requires
>>> > decrementing the front pointer (modulo the size) and to insert at the
>>> rear
>>> > requires incrementing the rear pointer (modulo the size).  We need to
>>> resize
>>> > when the two pointers bump into each other.  Could you explain more
>>> about
>>> > your suggestion of introducing an arraylet that is shared by the front
>>> and
>>> > the rear?
>>>
>>> It was a half-baked idea - I don't know if there's a way to turn it into
>>> something useful.  I was thinking of the ArrayDeque implementation,
>>> where all the elements live in a single array.
>>>
>>> >  It's not clear to me how that would help and/or be a better
>>> > approach than the cyclic list.  Anyhow, the paper that you reference,
>>> > "Resizable arrays in optimal time and space", gives a deque so if we
>>> take
>>> > that approach then the deque is specified.
>>>
>>> Technically, ArrayList also supports the Deque operations -
>>> just not efficiently.
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100331/0e3f0369/attachment.html>

From gokdogan at gmail.com  Thu Apr  1 08:42:00 2010
From: gokdogan at gmail.com (Goktug Gokdogan)
Date: Thu, 1 Apr 2010 01:42:00 -0700
Subject: java.util.Pair
In-Reply-To: <4BB28A9C.10705@oracle.com>
References: <EDA0E349-DBE4-45DA-BF05-83C62A786116@Sun.COM>
	<108fcdeb1003301054v211788a8tdb28dcb24aa0d2e4@mail.gmail.com>
	<4BB28A9C.10705@oracle.com>
Message-ID: <x2rf5b3aa371004010142v60795bfdy7045c0932a7ce00e@mail.gmail.com>

I think an object oriented language like Java should not contain a class
where there are so many different candidate names for its fields. It's a
code smell for improper abstraction.
There are lots of classes that are structurally similar but they are defined
in different classes as they are abstractions for different things, like
Point and Size.
Similarly, IMHO, developers should be encouraged to define their own classes
with a proper abstraction, adding classes like Pair will discourage to do
so.

On Tue, Mar 30, 2010 at 4:34 PM, <joe.darcy at oracle.com> wrote:

>
>
> On 3/30/2010 10:54 AM, Kevin Bourrillion wrote:
>
>> Pair is only a partial, flawed solution to a special case (n=2) of a very
>> significant problem: the disproportionate complexity of creating value types
>> in Java.  I support addressing the underlying problem in Java 8, and not
>> littering the API with dead-end solutions like Pair.
>>
>
> While I have sympathy with that conclusion, there is the
> side-effect of littering many APIs with the flotsam of lots of different
> classes named "Pair."  My inclination would be to produce one adequate
> Pair class in the JDK to prevent the proliferation of yet more Pair classes
> in other code bases.
>
> I should know better than to take the bait, below is a first cut at
> java.util.Pair.
>
> -Joe
>
> package java.util;
>
> import java.util.Objects;
>
> /**
>  * An immutable pair of values.  The values may be null.  The values
>  * themselves may be mutable.
>  *
>  * @param <A> the type of the first element of the pair
>  * @param <B> the type of the second element of the pair
>  *
>  * @since 1.7
>  */
> public final class Pair<A, B> {
>    private final A a;
>    private final B b;
>
>    private Pair(A a, B b) {
>        this.a = a;
>        this.b = b;
>    }
>
>    /**
>     * Returns a pair whose elements are the first and second
>     * arguments, respectively.
>     * @return a pair constructed from the arguments
>     */
>    public static <C, D> Pair<C, D> valueOf(C c, D d) {
>        // Don't mandate new values.
>        return new Pair<C, D>(c, d);
>    }
>
>    /**
>     * Returns the value of the first element of the pair.
>     * @return the value of the first element of the pair
>     */
>    public A getA() {
>        return a;
>    }
>
>    /**
>     * Returns the value of the second element of the pair.
>     * @return the value of the second element of the pair
>     */
>    public B getB() {
>        return b;
>    }
>
>    /**
>     * TBD
>     */
>    @Override
>    public String toString() {
>        return "[" + Objects.toString(a) + ", " + Objects.toString(b) + "]";
>    }
>
>    /**
>     * TBD
>     */
>    @Override
>    public boolean equals(Object x) {
>        if (!(x instanceof Pair))
>            return false;
>        else {
>            Pair<?,?> that = (Pair<?,?>) x;
>            return
>                Objects.equals(this.a, that.a) &&
>                Objects.equals(this.b, that.b);
>        }
>    }
>
>    /**
>     * TBD
>     */
>    @Override
>    public int hashCode() {
>        return Objects.hash(a, b);
>    }
> }
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100401/24bdd4e1/attachment.html>

From Ulf.Zibis at gmx.de  Thu Apr  1 12:26:05 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Thu, 01 Apr 2010 14:26:05 +0200
Subject: Swap should be better done native?
In-Reply-To: <g2x1ccfd1c11003311354re190b57fw2c3ef288dd8e40e5@mail.gmail.com>
References: <4BB20D1E.4090301@gmx.de> <87ljd9sm6o.fsf@mid.deneb.enyo.de>	
	<4BB39315.1020005@gmx.de>	
	<1ccfd1c11003311140u51df1dcbld7155ca4b1f1cfea@mail.gmail.com>	
	<4BB39C08.5040003@gmx.de>	
	<1ccfd1c11003311206l23bb0783k11296eb6370c536@mail.gmail.com>	
	<4BB3B390.8040806@gmx.de>
	<g2x1ccfd1c11003311354re190b57fw2c3ef288dd8e40e5@mail.gmail.com>
Message-ID: <4BB490DD.2060809@gmx.de>

Am 31.03.2010 22:54, schrieb Martin Buchholz:
> On Wed, Mar 31, 2010 at 13:41, Ulf Zibis<Ulf.Zibis at gmx.de>  wrote:
>    
>> You remember on UTF-8 twiddling:
>>
>> Am 16.03.2010 22:51, schrieb Ulf Zibis:
>>      
>>> Am 16.03.2010 21:57, schrieb Martin Buchholz:
>>>        
>>>> On Tue, Mar 16, 2010 at 12:48, Ulf Zibis<Ulf.Zibis at gmx.de>    wrote:
>>>>          
>>>>>>>   8-bit shift + compare would allow HotSpot to compile to smart 1-byte
>>>>>>> immediate op-codes.
>>>>>>> In encodeBufferLoop() you could use putChar(), putInt() instead put().
>>>>>>> Should perform better.
>>>>>>>
>>>>>>>                
>>>>>> I'm not convinced.  You would need to assemble bytes into an
>>>>>> int, and then break them apart into bytes on the other side?
>>>>>>
>>>>>>              
>>>>> Some time ago, I disassembled such code. I could see, that the int was
>>>>> copied directly to memory by one 32-bit move instruction.
>>>>> In case of using put(byte), I saw 4 8-bit move instructions.
>>>>>            
>>>> Ulf, I'd like to understand this better.
>>>>
>>>> How are you generating the machine code
>>>> (pointer to docs?)?
>>>>          
>>> I must prepare it. Takes some time.
>>>        
>> Now you can see, that putInt(int) is done at once, so faster as 4 put(byte).
>>      
> Leider versteh ich dass immer noch nicht.
>    

Thanks for some little German. :-D

> Ich weiss auch noch nicht,
> how to generate disassembled code.
>    

Sorry about my flippancy/ignorance.
I started here: http://wikis.sun.com/display/HotSpotInternals/PrintAssembly
On Windows I use a hsdis-i386.dll compiled by Andreas Sch?sser and 
provided from Volker Simonis <volker.simonis at gmail.com>.

> How can 4 put(byte) be converted into one put(int)?
>    
See the following code snippets ...

===================================================
Codesnippet from EUC_TW$Encoder:
             dst.put(SS2);
             dst.put((byte)(0xa0 | p));
             dst.put((byte)(db >> 8));
             dst.put((byte)db);

becomes (124 bytes):
   0x00b94ab2: mov    %edx,%ebx          ;*invokevirtual put
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ab4: mov    0x14(%ebx),%ecx    ;*getfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 1 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ab7: mov    0x18(%ebx),%edx    ;*getfield limit
                                         ; - 
java.nio.Buffer::nextPutIndex at 5 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94aba: cmp    %edx,%ecx
   0x00b94abc: jge    0x00b94b79         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ac2: mov    %ebx,0x14(%esp)
   0x00b94ac6: mov    %ebx,%esi
   0x00b94ac8: mov    0x8(%esi),%ebp
   0x00b94acb: mov    0xc(%esi),%edi
   0x00b94ace: mov    0x8(%esp),%eax
   0x00b94ad2: or     $0xa0,%eax         ;*ior  ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 107 (line 267)
   0x00b94ad8: mov    %eax,0x8(%esp)
   0x00b94adc: mov    %ebp,%eax
   0x00b94ade: mov    %ecx,%ebx
   0x00b94ae0: inc    %ebx               ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ae1: mov    %ebx,0x14(%esi)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ae4: add    %ecx,%eax
   0x00b94ae6: movb   $0x8e,(%eax)       ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 97 (line 266)
   0x00b94ae9: cmp    %edx,%ebx
   0x00b94aeb: jge    0x00b94b8d         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 109 (line 267)
   0x00b94af1: mov    0x8(%esp),%ebx
   0x00b94af5: mov    %bl,0x1(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 109 (line 267)
   0x00b94af8: mov    %ecx,%ebx
   0x00b94afa: add    $0x2,%ebx          ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 109 (line 267)
   0x00b94afd: mov    %ebx,0x14(%esi)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 109 (line 267)
   0x00b94b00: cmp    %edx,%ebx
   0x00b94b02: jge    0x00b94b9d         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 119 (line 268)
   0x00b94b08: mov    0x18(%esp),%ebx
   0x00b94b0c: mov    %bl,0x2(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 119 (line 268)
   0x00b94b0f: mov    %ecx,%ebx
   0x00b94b11: add    $0x3,%ebx          ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 119 (line 268)
   0x00b94b14: mov    %esi,%ebp
   0x00b94b16: mov    %ebx,0x14(%ebp)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 119 (line 268)
   0x00b94b19: cmp    %edx,%ebx
   0x00b94b1b: jge    0x00b94ba9         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 126 (line 269)
   0x00b94b21: mov    0x4(%esp),%edx
   0x00b94b25: mov    %dl,0x3(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 126 (line 269)
   0x00b94b28: add    $0x4,%ecx
   0x00b94b2b: mov    %ecx,0x14(%ebp)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer0::encode at 126 (line 269)

===================================================
Alternative 1 codesnippet:
             dst.putInt((SS2 << 24) | (0xa0 << 16) | (p << 16) | db);

becomes (63 bytes):
   0x00b95d51: shl    $0x10,%ebx
   0x00b95d54: or     %edi,%ebx
   0x00b95d56: or     $0x8ea00000,%ebx   ;*ior  ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 93 (line 265)
   0x00b95d5c: cmp    $0x282c61d8,%ebp   ;   
{oop(&apos;java/nio/DirectByteBuffer&apos;)}
   0x00b95d62: jne    0x00b95dcd         ;*invokevirtual putInt
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d64: mov    0x18(%edx),%edi
   0x00b95d67: mov    0x14(%edx),%ecx    ;*getfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 5 (line 518)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 4 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d6a: sub    %ecx,%edi
   0x00b95d6c: cmp    $0x4,%edi
   0x00b95d6f: jl     0x00b95de1         ;*if_icmpge
                                         ; - 
java.nio.Buffer::nextPutIndex at 10 (line 518)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 4 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d71: movzbl 0x26(%edx),%eax
   0x00b95d75: test   %eax,%eax
   0x00b95d77: jne    0x00b95d7b         ;*ifeq
                                         ; - 
java.nio.DirectByteBuffer::putInt at 17 (line 664)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 11 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d79: bswap  %ebx               ;*invokevirtual putInt
                                         ; - 
java.nio.DirectByteBuffer::putInt at 30 (line 664)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 11 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d7b: mov    %edx,%esi
   0x00b95d7d: mov    0x8(%esi),%ebp
   0x00b95d80: mov    0xc(%esi),%edi     ;*getfield address
                                         ; - 
java.nio.DirectByteBuffer::ix at 1 (line 225)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 7 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d83: mov    %ecx,%eax
   0x00b95d85: add    $0x4,%eax
   0x00b95d88: mov    %eax,0x14(%edx)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 33 (line 521)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 4 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)
   0x00b95d8b: mov    %ebp,%eax
   0x00b95d8d: mov    %ebx,(%eax,%ecx,1)  ;*invokevirtual putInt
                                         ; - 
java.nio.DirectByteBuffer::putInt at 30 (line 664)
                                         ; - 
java.nio.DirectByteBuffer::putInt at 11 (line 676)
                                         ; - 
sun.nio.cs.ext.E_31_d_n_codeToBuffer1::encode at 94 (line 265)

===================================================
On big endian machines, additionally the swapping becomes omitted.

===================================================
Alternative 2 codesnippet:
             bb[0] = SS2;
             bb[1] = (byte)(0xa0 | p);
             bb[2] = (byte)(db >> 8);
             bb[3] = (byte)db;
             dst.put(bb, 0, 4);

becomes (149 bytes):
   0x00b95f68: mov    0x30(%esp),%ebp    ;*invokevirtual put
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f6c: mov    %ebp,0x30(%esp)
   0x00b95f70: add    $0xfffffffc,%ecx
   0x00b95f73: or     $0x4,%ecx
   0x00b95f76: test   %ecx,%ecx
   0x00b95f78: jl     0x00b96120         ;*ifge
                                         ; - 
java.nio.Buffer::checkBounds at 13 (line 551)
                                         ; - java.nio.ByteBuffer::put at 4 
(line 803)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f7e: mov    0x14(%ebp),%ecx    ;*getfield position
                                         ; - 
java.nio.Buffer::remaining at 5 (line 383)
                                         ; - java.nio.ByteBuffer::put at 9 
(line 804)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f81: mov    0x18(%ebp),%ebx    ;*getfield limit
                                         ; - 
java.nio.Buffer::remaining at 1 (line 383)
                                         ; - java.nio.ByteBuffer::put at 9 
(line 804)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f84: mov    %ebx,%ebp
   0x00b95f86: sub    %ecx,%ebp
   0x00b95f88: cmp    $0x4,%ebp
   0x00b95f8b: jl     0x00b9612d         ;*if_icmple
                                         ; - java.nio.ByteBuffer::put at 12 
(line 804)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f91: cmp    %ebx,%ecx
   0x00b95f93: jge    0x00b96139         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95f99: mov    %eax,0x18(%esp)
   0x00b95f9d: mov    %edx,0x14(%esp)
   0x00b95fa1: mov    0x30(%esp),%esi
   0x00b95fa5: mov    0x8(%esi),%ebp
   0x00b95fa8: mov    0xc(%esi),%edi
   0x00b95fab: mov    %ecx,%edx
   0x00b95fad: inc    %edx               ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fae: mov    %edx,0x14(%esi)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fb1: mov    %ebp,%eax
   0x00b95fb3: add    %ecx,%eax
   0x00b95fb5: movb   $0x8e,(%eax)       ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fb8: cmp    %ebx,%edx
   0x00b95fba: jge    0x00b96155         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fc0: mov    0x8(%esp),%edx
   0x00b95fc4: mov    %dl,0x1(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fc7: mov    %ecx,%ebp
   0x00b95fc9: add    $0x2,%ebp          ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fcc: mov    %ebp,0x14(%esi)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fcf: cmp    %ebx,%ebp
   0x00b95fd1: jge    0x00b96142         ;*if_icmplt
                                         ; - 
java.nio.Buffer::nextPutIndex at 8 (line 512)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fd7: mov    0x18(%esp),%edx
   0x00b95fdb: mov    %dl,0x2(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fde: mov    %ecx,%ebp
   0x00b95fe0: add    $0x3,%ebp          ;*iadd
                                         ; - 
java.nio.Buffer::nextPutIndex at 26 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fe3: mov    %ebp,0x14(%esi)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fe6: cmp    %ebx,%ebp
   0x00b95fe8: jge    0x00b96161         ;*aload_0
                                         ; - java.nio.ByteBuffer::put at 38 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95fee: mov    0x4(%esp),%ebx
   0x00b95ff2: mov    %bl,0x3(%eax)      ;*invokevirtual putByte
                                         ; - 
java.nio.DirectByteBuffer::put at 12 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)
   0x00b95ff5: add    $0x4,%ecx
   0x00b95ff8: mov    %esi,%ebp
   0x00b95ffa: mov    %ecx,0x14(%ebp)    ;*putfield position
                                         ; - 
java.nio.Buffer::nextPutIndex at 27 (line 514)
                                         ; - 
java.nio.DirectByteBuffer::put at 5 (line 271)
                                         ; - java.nio.ByteBuffer::put at 43 
(line 808)
                                         ; - 
java.nio.DirectByteBuffer::put at 117 (line 349)
                                         ; - 
sun.nio.cs.ext.E_30_d_n_codeToBuffer01::encode at 136 (line 272)

===================================================

I believe, alternative 2 could become much better from HotSpot side 
(enable -XX:+DoEscapeAnalysis didn't work here).

-Ulf





From martinrb at google.com  Thu Apr  1 22:36:41 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 1 Apr 2010 15:36:41 -0700
Subject: Wording improvements for String.indexOf, String.lastIndexOf
In-Reply-To: <4BB5119E.7010208@oracle.com>
References: <s2j1ccfd1c11003311606je42e5119z27f975960c2747f9@mail.gmail.com>
	<4BB5119E.7010208@oracle.com>
Message-ID: <p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>

On Thu, Apr 1, 2010 at 14:35, Xueming Shen <xueming.shen at oracle.com> wrote:
>
> CR 6940381 Created, P4 java/classes_lang Wording improvements for
> String.indexOf, String.lastIndexOf
>
> Since it's a simple rewording, I would not expect a CCC is necessary.
>
> Please do it separately.
>
> Seems like the "is <code>true</code>." line is missing from the new wording
> (in
> all cases), is it purposely? Not a native, but I would assume we still need
> those 2 words to make the sentence complete.

It is true that "is true"
would make it a grammatically complete sentence,
but I think it's best to not be quite that pedantic and lawyerly.
In general, the JDK makes a pretty good compromise between
precision and readability.

Martin


From martinrb at google.com  Thu Apr  1 22:38:43 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 1 Apr 2010 15:38:43 -0700
Subject: Wording improvements for String.indexOf, String.lastIndexOf
In-Reply-To: <p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>
References: <s2j1ccfd1c11003311606je42e5119z27f975960c2747f9@mail.gmail.com>
	<4BB5119E.7010208@oracle.com>
	<p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>
Message-ID: <p2z1ccfd1c11004011538ie728951dj4c2aed61732f1f4f@mail.gmail.com>

Renamed to IndexOf-javadoc
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/IndexOf-javadoc/

Martin

On Thu, Apr 1, 2010 at 15:36, Martin Buchholz <martinrb at google.com> wrote:
> On Thu, Apr 1, 2010 at 14:35, Xueming Shen <xueming.shen at oracle.com> wrote:
>>
>> CR 6940381 Created, P4 java/classes_lang Wording improvements for
>> String.indexOf, String.lastIndexOf
>>
>> Since it's a simple rewording, I would not expect a CCC is necessary.
>>
>> Please do it separately.
>>
>> Seems like the "is <code>true</code>." line is missing from the new wording
>> (in
>> all cases), is it purposely? Not a native, but I would assume we still need
>> those 2 words to make the sentence complete.
>
> It is true that "is true"
> would make it a grammatically complete sentence,
> but I think it's best to not be quite that pedantic and lawyerly.
> In general, the JDK makes a pretty good compromise between
> precision and readability.
>
> Martin
>


From martinrb at google.com  Fri Apr  2 03:28:18 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 1 Apr 2010 20:28:18 -0700
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
Message-ID: <l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>

Hi Kevin,

You're probably the only one on this list who has
seriously read the paper.  It is not surprising that
taking a research paper into production would
discover bugs - the research never had to undergo
rigorous testing.  (I like the Java culture of
combining spec + implementation + test suite)

I suggest you ask the authors directly about the bug.
They would probably also be interested to hear
about your implementation.

Are you aware of Integer.numberOfLeadingZeros?
http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)

Martin

On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
> I'm almost convinced now that the paper is incorrect.? The code below gives
> me the appropriate index into the index array and the offset into the data
> block.? That being said, remember when I mentioned that this will include a
> bit more work to access an element than a simple bit shift and a bit mask?
> Well this is more than a bit more - we'll be doing this each time an index
> is requested.? I'll spend some time trying to twiddle the bits to see if I
> can eliminate/combine some of the operations.
>
> ??? ??? for (int r = 1; r < 33; r++) {
> ??? ??? ??? int k = lg(r);
> ??? ??? ??? int floorKO2 = k >> 1;
> ??? ??? ??? int powFloorKO2 = (1 << floorKO2);
> ??? ??? ??? int p = ((1 << floorKO2) - 1) << 1;
> ??? ??? ??? int ceilKO2;
> ??? ??? ??? if ((k & 1) == 1) {
> ??? ??? ??? ??? ceilKO2 = floorKO2 + 1;
> ??? ??? ??? ??? p += powFloorKO2;
> ??? ??? ??? } else {
> ??? ??? ??? ??? ceilKO2 = floorKO2;
> ??? ??? ??? }
> ??? ??? ??? int e = r & ((1 << ceilKO2) - 1);
> ??? ??? ??? int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>
> ??? ??? ??? System.out.println((r - 1) + " " + (p + b) + " " + e);
> ??? ??? }
>
> Kevin
>
> On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern <kevin.l.stern at gmail.com>
> wrote:
>>
>> I realize that 2 * (2^(k/2) - 1) only works for even numbered superblocks,
>> the odd numbered superblocks need an additional term added (the number of
>> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I also came
>> across an alternative characterization of superblock in the paper which
>> states that data blocks are grouped within a superblock when they are the
>> same size - to me, though, that implies that my example structure below
>> would be
>>
>> SB_0: [1]
>> SB_1: [2][2][2]
>> SB_2: [4][4][4][4][4][4]
>>
>> which seems to contradict my understanding of (1) below.? I must be
>> reading this upside down.
>>
>> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>>>
>>> What am I missing here?? In "Resizable arrays in optimal time and space"
>>> the authors define their data structure with the following property:
>>>
>>> (1)? "When superblock SB_k is fully allocated, it consists of
>>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>
>>> Since the superblock is zero-based indexed this implies the following
>>> structure:
>>>
>>> SB_0: [1]
>>> SB_1: [2]
>>> SB_2: [2][2]
>>> SB_3: [4][4]
>>> SB_4: [4][4][4][4]
>>> [...]
>>>
>>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>
>>> r = 100 (the binary expansion of i + 1)
>>> k = |r| - 1 = 2
>>> p = 2^k - 1 = 3
>>>
>>> What concerns me is their statement that p represents "the number of data
>>> blocks in superblocks prior to SB_k."? There are only two data blocks in
>>> superblocks prior to SB_2, not three.? Given (1) above, unless I'm
>>> misinterpreting it, the number of data blocks in superblocks prior to SB_k
>>> should be:
>>>
>>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>
>>> This, of course, seems to work out much better in my example above,
>>> giving the correct answer to my interpretation of their data structure, but
>>> I have a hard time believing that this is their mistake rather than my
>>> misinterpretation.
>>>
>>> Thoughts?
>>>
>>> Kevin
>>>
>>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <martinrb at google.com>
>>> wrote:
>>>>
>>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>> wrote:
>>>> > Hi Martin,
>>>> >
>>>> > Thanks much for your feedback.? The first approach that comes to mind
>>>> > to
>>>> > implement O(1) time front as well as rear insertion is to create a
>>>> > cyclic
>>>> > list structure with a front/rear pointer - to insert at the front
>>>> > requires
>>>> > decrementing the front pointer (modulo the size) and to insert at the
>>>> > rear
>>>> > requires incrementing the rear pointer (modulo the size).? We need to
>>>> > resize
>>>> > when the two pointers bump into each other.? Could you explain more
>>>> > about
>>>> > your suggestion of introducing an arraylet that is shared by the front
>>>> > and
>>>> > the rear?
>>>>
>>>> It was a half-baked idea - I don't know if there's a way to turn it into
>>>> something useful. ?I was thinking of the ArrayDeque implementation,
>>>> where all the elements live in a single array.
>>>>
>>>> > ?It's not clear to me how that would help and/or be a better
>>>> > approach than the cyclic list.? Anyhow, the paper that you reference,
>>>> > "Resizable arrays in optimal time and space", gives a deque so if we
>>>> > take
>>>> > that approach then the deque is specified.
>>>>
>>>> Technically, ArrayList also supports the Deque operations -
>>>> just not efficiently.
>>>
>>
>
>


From martinrb at google.com  Fri Apr  2 20:01:18 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 2 Apr 2010 13:01:18 -0700
Subject: review request 6933322 - Add methods highSurrogate(), 
	lowSurrogate() to class Character
In-Reply-To: <4BB45C72.6070700@sun.com>
References: <1ccfd1c11003201136u78e159ew88724bfa5a9e28c0@mail.gmail.com>
	<4BA56749.8020506@gmx.de>
	<1ccfd1c11003240124v24db88c0wd8c05396a92a6fef@mail.gmail.com>
	<4BABCAC0.4010704@gmx.de> <4BABCD1A.2040205@gmx.de>
	<4BB45C72.6070700@sun.com>
Message-ID: <j2n1ccfd1c11004021301s419c7961s88d3131c4984f665@mail.gmail.com>

Hi Masayoshi,

Writing good spec is hard and annoying, but important.

I've improved the spec in various ways, mostly as you suggested.
Please see the updated webrev
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/highSurrogate/

Martin

On Thu, Apr 1, 2010 at 01:42, Masayoshi Okutsu <Masayoshi.Okutsu at sun.com> wrote:
> Hi Martin,
>
> Here are my comments on the 6933322 changes.
>
> - I'd suggest that the Unicode terms be used instead of "the first part" and
> "the second part", something like "the high surrogate (also known as leading
> surrogate) code unit of the surrogate pair." If you want to emphasize the
> order, "the leading surrogate (high surrogate) code unit" should be OK.
> Actually there were some discussions about high/low vs. leading/trailing in
> JSR 204, and we decided to use high/low to follow the (main) Unicode terms.
>
> http://www.unicode.org/glossary/#high_surrogate_code_unit
> http://www.unicode.org/glossary/#leading_surrogate
>
> - @param, @return and @since are missing. There should be @see for the
> counterpart.
>
> Otherwise, the changes look good to me, assuming that CCC would approve the
> API change.
>
> Thanks,
> Masayoshi
>
> On 3/26/2010 5:52 AM, Ulf Zibis wrote:
>>
>> Updated topic.
>>
>> -Ulf
>>
>>
>> Am 25.03.2010 21:42, schrieb Ulf Zibis:
>>>
>>> Am 24.03.2010 09:24, schrieb Martin Buchholz:
>>>>
>>>> Ulf, Sherman, Masayoshi,
>>>> here are changes for you to review.
>>>> Only the patch highSurrogate needs a separate bug filed
>>>> (and CCC, please)
>>>
>>> I had just filed it 2 weeks ago, see:
>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6933322
>>>
>>> -Ulf
>


From Masayoshi.Okutsu at Sun.COM  Thu Apr  1 08:42:26 2010
From: Masayoshi.Okutsu at Sun.COM (Masayoshi Okutsu)
Date: Thu, 01 Apr 2010 17:42:26 +0900
Subject: review request 6933322 - Add methods highSurrogate(),
	lowSurrogate() to class Character
In-Reply-To: <4BABCD1A.2040205@gmx.de>
References: <1ccfd1c11003201136u78e159ew88724bfa5a9e28c0@mail.gmail.com>
	<4BA56749.8020506@gmx.de>
	<1ccfd1c11003240124v24db88c0wd8c05396a92a6fef@mail.gmail.com>
	<4BABCAC0.4010704@gmx.de> <4BABCD1A.2040205@gmx.de>
Message-ID: <4BB45C72.6070700@sun.com>

Hi Martin,

Here are my comments on the 6933322 changes.

- I'd suggest that the Unicode terms be used instead of "the first part" 
and "the second part", something like "the high surrogate (also known as 
leading surrogate) code unit of the surrogate pair." If you want to 
emphasize the order, "the leading surrogate (high surrogate) code unit" 
should be OK. Actually there were some discussions about high/low vs. 
leading/trailing in JSR 204, and we decided to use high/low to follow 
the (main) Unicode terms.

http://www.unicode.org/glossary/#high_surrogate_code_unit
http://www.unicode.org/glossary/#leading_surrogate

- @param, @return and @since are missing. There should be @see for the 
counterpart.

Otherwise, the changes look good to me, assuming that CCC would approve 
the API change.

Thanks,
Masayoshi

On 3/26/2010 5:52 AM, Ulf Zibis wrote:
> Updated topic.
>
> -Ulf
>
>
> Am 25.03.2010 21:42, schrieb Ulf Zibis:
>> Am 24.03.2010 09:24, schrieb Martin Buchholz:
>>> Ulf, Sherman, Masayoshi,
>>> here are changes for you to review.
>>> Only the patch highSurrogate needs a separate bug filed
>>> (and CCC, please)
>>
>> I had just filed it 2 weeks ago, see:
>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6933322
>>
>> -Ulf


From Ulf.Zibis at gmx.de  Sat Apr  3 00:54:24 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Sat, 03 Apr 2010 02:54:24 +0200
Subject: review request 6933322 - Add methods highSurrogate(),
	lowSurrogate() to class Character
In-Reply-To: <j2n1ccfd1c11004021301s419c7961s88d3131c4984f665@mail.gmail.com>
References: <1ccfd1c11003201136u78e159ew88724bfa5a9e28c0@mail.gmail.com>	
	<4BA56749.8020506@gmx.de>	
	<1ccfd1c11003240124v24db88c0wd8c05396a92a6fef@mail.gmail.com>	
	<4BABCAC0.4010704@gmx.de> <4BABCD1A.2040205@gmx.de>	
	<4BB45C72.6070700@sun.com>
	<j2n1ccfd1c11004021301s419c7961s88d3131c4984f665@mail.gmail.com>
Message-ID: <4BB691C0.8090800@gmx.de>

Looks good to me, but what about integrating **/highSurrogate2?

Am 02.04.2010 22:01, schrieb Martin Buchholz:
> Hi Masayoshi,
>
> Writing good spec is hard and annoying, but important.
>    

I think you have done a little bit too much in the 2nd paragraphs. I 
rarely have seen such detailedness on current javadocs.
Good exercise, the sophisticated use of {@link }. :-)

-Ulf

> I've improved the spec in various ways, mostly as you suggested.
> Please see the updated webrev
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/highSurrogate/
>
> Martin
>
> On Thu, Apr 1, 2010 at 01:42, Masayoshi Okutsu<Masayoshi.Okutsu at sun.com>  wrote:
>    
>> Hi Martin,
>>
>> Here are my comments on the 6933322 changes.
>>
>> - I'd suggest that the Unicode terms be used instead of "the first part" and
>> "the second part", something like "the high surrogate (also known as leading
>> surrogate) code unit of the surrogate pair." If you want to emphasize the
>> order, "the leading surrogate (high surrogate) code unit" should be OK.
>> Actually there were some discussions about high/low vs. leading/trailing in
>> JSR 204, and we decided to use high/low to follow the (main) Unicode terms.
>>
>> http://www.unicode.org/glossary/#high_surrogate_code_unit
>> http://www.unicode.org/glossary/#leading_surrogate
>>
>> - @param, @return and @since are missing. There should be @see for the
>> counterpart.
>>
>> Otherwise, the changes look good to me, assuming that CCC would approve the
>> API change.
>>
>> Thanks,
>> Masayoshi
>>
>> On 3/26/2010 5:52 AM, Ulf Zibis wrote:
>>      
>>> Updated topic.
>>>
>>> -Ulf
>>>
>>>
>>> Am 25.03.2010 21:42, schrieb Ulf Zibis:
>>>        
>>>> Am 24.03.2010 09:24, schrieb Martin Buchholz:
>>>>          
>>>>> Ulf, Sherman, Masayoshi,
>>>>> here are changes for you to review.
>>>>> Only the patch highSurrogate needs a separate bug filed
>>>>> (and CCC, please)
>>>>>            
>>>> I had just filed it 2 weeks ago, see:
>>>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6933322
>>>>
>>>> -Ulf
>>>>          
>>      
>
>    



From martinrb at google.com  Sat Apr  3 01:02:38 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 2 Apr 2010 18:02:38 -0700
Subject: review request 6933322 - Add methods highSurrogate(), 
	lowSurrogate() to class Character
In-Reply-To: <4BB691C0.8090800@gmx.de>
References: <1ccfd1c11003201136u78e159ew88724bfa5a9e28c0@mail.gmail.com>
	<4BA56749.8020506@gmx.de>
	<1ccfd1c11003240124v24db88c0wd8c05396a92a6fef@mail.gmail.com>
	<4BABCAC0.4010704@gmx.de> <4BABCD1A.2040205@gmx.de>
	<4BB45C72.6070700@sun.com>
	<j2n1ccfd1c11004021301s419c7961s88d3131c4984f665@mail.gmail.com>
	<4BB691C0.8090800@gmx.de>
Message-ID: <i2n1ccfd1c11004021802y37dd008euac86c3e0ebe80545@mail.gmail.com>

On Fri, Apr 2, 2010 at 17:54, Ulf Zibis <Ulf.Zibis at gmx.de> wrote:
> Looks good to me, but what about integrating **/highSurrogate2?

I plan to do that just before I commit.

> I think you have done a little bit too much in the 2nd paragraphs. I rarely
> have seen such detailedness on current javadocs.

The source is now less readable.
Hopefully the output is more readable.

Martin


From kevin.l.stern at gmail.com  Sun Apr  4 11:12:24 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Sun, 4 Apr 2010 06:12:24 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
Message-ID: <r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>

The data structure is available at the second link that I originally
provided (once again, it is
https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en).
This does not have O(1) time insertion at the front as yet as it was unclear
to me whether or not it was agreed upon:
_________________
From: Osvaldo Doederlein <opinali at gmail.com>
Date: Mon, Mar 29, 2010 at 10:08 AM
Subject: Re: A List implementation backed by multiple small arrays rather
than the traditional single large array.
To: Martin Buchholz <martinrb at google.com>
Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
core-libs-dev at openjdk.java.net

Initially, it would be good enough to replace only java.util.ArrayList with
minimal overhead. ArrayList does not support efficient add-at-front or other
enhancements of ArrayDeque; but ArrayList is still a much more important and
popular collection, it's the primary "straight replacement for primitive
arrrays" and I guess it should continue with that role.
_________________

As a disclaimer, I'm still tinkering with this so I'll be updating the
document at the provided link as I find improvements.

Thoughts?

Thanks,

Kevin

On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <martinrb at google.com>wrote:

> Hi Kevin,
>
> You're probably the only one on this list who has
> seriously read the paper.  It is not surprising that
> taking a research paper into production would
> discover bugs - the research never had to undergo
> rigorous testing.  (I like the Java culture of
> combining spec + implementation + test suite)
>
> I suggest you ask the authors directly about the bug.
> They would probably also be interested to hear
> about your implementation.
>
> Are you aware of Integer.numberOfLeadingZeros?
>
> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>
> Martin
>
> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <kevin.l.stern at gmail.com>
> wrote:
> > I'm almost convinced now that the paper is incorrect.  The code below
> gives
> > me the appropriate index into the index array and the offset into the
> data
> > block.  That being said, remember when I mentioned that this will include
> a
> > bit more work to access an element than a simple bit shift and a bit
> mask?
> > Well this is more than a bit more - we'll be doing this each time an
> index
> > is requested.  I'll spend some time trying to twiddle the bits to see if
> I
> > can eliminate/combine some of the operations.
> >
> >         for (int r = 1; r < 33; r++) {
> >             int k = lg(r);
> >             int floorKO2 = k >> 1;
> >             int powFloorKO2 = (1 << floorKO2);
> >             int p = ((1 << floorKO2) - 1) << 1;
> >             int ceilKO2;
> >             if ((k & 1) == 1) {
> >                 ceilKO2 = floorKO2 + 1;
> >                 p += powFloorKO2;
> >             } else {
> >                 ceilKO2 = floorKO2;
> >             }
> >             int e = r & ((1 << ceilKO2) - 1);
> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
> >
> >             System.out.println((r - 1) + " " + (p + b) + " " + e);
> >         }
> >
> > Kevin
> >
> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern <kevin.l.stern at gmail.com
> >
> > wrote:
> >>
> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
> superblocks,
> >> the odd numbered superblocks need an additional term added (the number
> of
> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I also
> came
> >> across an alternative characterization of superblock in the paper which
> >> states that data blocks are grouped within a superblock when they are
> the
> >> same size - to me, though, that implies that my example structure below
> >> would be
> >>
> >> SB_0: [1]
> >> SB_1: [2][2][2]
> >> SB_2: [4][4][4][4][4][4]
> >>
> >> which seems to contradict my understanding of (1) below.  I must be
> >> reading this upside down.
> >>
> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern <
> kevin.l.stern at gmail.com>
> >> wrote:
> >>>
> >>> What am I missing here?  In "Resizable arrays in optimal time and
> space"
> >>> the authors define their data structure with the following property:
> >>>
> >>> (1)  "When superblock SB_k is fully allocated, it consists of
> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
> >>>
> >>> Since the superblock is zero-based indexed this implies the following
> >>> structure:
> >>>
> >>> SB_0: [1]
> >>> SB_1: [2]
> >>> SB_2: [2][2]
> >>> SB_3: [4][4]
> >>> SB_4: [4][4][4][4]
> >>> [...]
> >>>
> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
> >>>
> >>> r = 100 (the binary expansion of i + 1)
> >>> k = |r| - 1 = 2
> >>> p = 2^k - 1 = 3
> >>>
> >>> What concerns me is their statement that p represents "the number of
> data
> >>> blocks in superblocks prior to SB_k."  There are only two data blocks
> in
> >>> superblocks prior to SB_2, not three.  Given (1) above, unless I'm
> >>> misinterpreting it, the number of data blocks in superblocks prior to
> SB_k
> >>> should be:
> >>>
> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
> >>>
> >>> This, of course, seems to work out much better in my example above,
> >>> giving the correct answer to my interpretation of their data structure,
> but
> >>> I have a hard time believing that this is their mistake rather than my
> >>> misinterpretation.
> >>>
> >>> Thoughts?
> >>>
> >>> Kevin
> >>>
> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <martinrb at google.com>
> >>> wrote:
> >>>>
> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern <
> kevin.l.stern at gmail.com>
> >>>> wrote:
> >>>> > Hi Martin,
> >>>> >
> >>>> > Thanks much for your feedback.  The first approach that comes to
> mind
> >>>> > to
> >>>> > implement O(1) time front as well as rear insertion is to create a
> >>>> > cyclic
> >>>> > list structure with a front/rear pointer - to insert at the front
> >>>> > requires
> >>>> > decrementing the front pointer (modulo the size) and to insert at
> the
> >>>> > rear
> >>>> > requires incrementing the rear pointer (modulo the size).  We need
> to
> >>>> > resize
> >>>> > when the two pointers bump into each other.  Could you explain more
> >>>> > about
> >>>> > your suggestion of introducing an arraylet that is shared by the
> front
> >>>> > and
> >>>> > the rear?
> >>>>
> >>>> It was a half-baked idea - I don't know if there's a way to turn it
> into
> >>>> something useful.  I was thinking of the ArrayDeque implementation,
> >>>> where all the elements live in a single array.
> >>>>
> >>>> >  It's not clear to me how that would help and/or be a better
> >>>> > approach than the cyclic list.  Anyhow, the paper that you
> reference,
> >>>> > "Resizable arrays in optimal time and space", gives a deque so if we
> >>>> > take
> >>>> > that approach then the deque is specified.
> >>>>
> >>>> Technically, ArrayList also supports the Deque operations -
> >>>> just not efficiently.
> >>>
> >>
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100404/fbd6cdd5/attachment.html>

From xueming.shen at sun.com  Sun Apr  4 01:41:07 2010
From: xueming.shen at sun.com (xueming.shen at sun.com)
Date: Sun, 04 Apr 2010 01:41:07 +0000
Subject: hg: jdk7/tl/jdk: 4947220: (process)Runtime.exec() cannot invoke
	applications with unicode parameters(win)
Message-ID: <20100404014206.ADA4E44084@hg.openjdk.java.net>

Changeset: 1105276dbd6a
Author:    sherman
Date:      2010-04-03 18:29 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/1105276dbd6a

4947220: (process)Runtime.exec() cannot invoke applications with unicode parameters(win)
Summary: to use CreateProcessW on Windowns platform
Reviewed-by: martin

! src/share/native/java/lang/System.c
! src/share/native/java/lang/java_props.h
! src/solaris/native/java/lang/java_props_md.c
! src/windows/native/java/lang/ProcessImpl_md.c
! src/windows/native/java/lang/java_props_md.c
! test/java/lang/ProcessBuilder/Basic.java



From abhijit.saha at sun.com  Mon Apr  5 22:49:57 2010
From: abhijit.saha at sun.com (abhijit.saha at sun.com)
Date: Mon, 05 Apr 2010 22:49:57 +0000
Subject: hg: jdk7/tl/hotspot: 6 new changesets
Message-ID: <20100405225025.EE1B544332@hg.openjdk.java.net>

Changeset: b5d78a3b8843
Author:    kvn
Date:      2009-12-03 14:20 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/b5d78a3b8843

6892265: System.arraycopy unable to reference elements beyond Integer.MAX_VALUE bytes
Summary: Use size_t type cast to widen int values in typeArrayKlass::copy_array().
Reviewed-by: never, jcoomes

! src/share/vm/oops/typeArrayKlass.cpp
+ test/compiler/6892265/Test.java

Changeset: ae4032fb0a5b
Author:    kvn
Date:      2010-01-21 10:07 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/ae4032fb0a5b

6894807: No ClassCastException for HashAttributeSet constructors if run with -Xcomp
Summary: Return interface klass type if it is exact.
Reviewed-by: never

! src/share/vm/opto/cfgnode.cpp
! src/share/vm/opto/type.cpp

Changeset: 0c3f888b7636
Author:    acorn
Date:      2010-01-19 16:03 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/0c3f888b7636

6626217: Fixed loader constraint array handling
Summary: Loader constraints track array elements, not arrays themselves.
Reviewed-by: dcubed, kevinw

! src/share/vm/ci/ciEnv.cpp
! src/share/vm/classfile/loaderConstraints.cpp
! src/share/vm/classfile/loaderConstraints.hpp
! src/share/vm/classfile/systemDictionary.cpp

Changeset: 3d531bbe5917
Author:    acorn
Date:      2010-01-28 13:59 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3d531bbe5917

Merge


Changeset: f5dd08ad65df
Author:    acorn
Date:      2010-03-15 15:51 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/f5dd08ad65df

6932480: Fix crash in CompilerThread/Parser. Unloaded array klass?
Summary: Restore code deleted in 6626217
Reviewed-by: asaha, kevinw

! src/share/vm/ci/ciEnv.cpp

Changeset: 09ac706c2623
Author:    asaha
Date:      2010-03-24 17:16 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/09ac706c2623

Merge

! src/share/vm/ci/ciEnv.cpp
! src/share/vm/classfile/loaderConstraints.cpp
! src/share/vm/classfile/loaderConstraints.hpp
! src/share/vm/classfile/systemDictionary.cpp
- src/share/vm/gc_implementation/g1/ptrQueue.inline.hpp
! src/share/vm/opto/type.cpp



From abhijit.saha at sun.com  Mon Apr  5 23:17:13 2010
From: abhijit.saha at sun.com (abhijit.saha at sun.com)
Date: Mon, 05 Apr 2010 23:17:13 +0000
Subject: hg: jdk7/tl/jdk: 30 new changesets
Message-ID: <20100405232703.B483D4433B@hg.openjdk.java.net>

Changeset: d3309aae68ef
Author:    dl
Date:      2009-10-06 12:20 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/d3309aae68ef

6888149: AtomicReferenceArray causes SIGSEGV -> SEGV_MAPERR error
Summary: Avoid integer overflow by using long arithmetic
Reviewed-by: martin, dholmes

! src/share/classes/java/util/concurrent/atomic/AtomicIntegerArray.java
! src/share/classes/java/util/concurrent/atomic/AtomicLongArray.java
! src/share/classes/java/util/concurrent/atomic/AtomicReferenceArray.java

Changeset: 08f57141c305
Author:    asaha
Date:      2009-11-20 14:24 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/08f57141c305

Merge

- test/sun/tools/native2ascii/test2

Changeset: b1e8f41ed755
Author:    chegar
Date:      2009-11-23 12:40 +0000
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/b1e8f41ed755

6639665: ThreadGroup finalizer allows creation of false root ThreadGroups
Reviewed-by: alanb, hawtin

! src/share/classes/java/lang/ThreadGroup.java

Changeset: e943f6b0b0e9
Author:    alanb
Date:      2009-11-25 10:02 +0000
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/e943f6b0b0e9

6736390: File TOCTOU deserialization vulnerability
Reviewed-by: hawtin

! src/share/classes/java/io/File.java

Changeset: ff9c2f53594e
Author:    sherman
Date:      2009-11-25 11:29 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ff9c2f53594e

6745393: Inflater/Deflater clone issue
Summary: To use explicit lobk object.
Reviewed-by: alanb

! src/share/classes/java/util/zip/Deflater.java
! src/share/classes/java/util/zip/Inflater.java
! src/share/native/java/util/zip/Deflater.c
! src/share/native/java/util/zip/Inflater.c

Changeset: d893f890b4dd
Author:    sherman
Date:      2009-11-25 12:51 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/d893f890b4dd

6904925: Changeset for 6745393 for jdk7 ssr forest was incomplete
Summary: To add, commit and push back the ZStreamRef.java
Reviewed-by: alanb

+ src/share/classes/java/util/zip/ZStreamRef.java

Changeset: df3091222715
Author:    mchung
Date:      2009-11-25 09:09 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/df3091222715

6893947: Deserialization of RMIConnectionImpl objects should enforce stricter checks [ZDI-CAN-588]
Summary: narrow the doPrivileged block to only set context ClassLoader
Reviewed-by: hawtin, emcmanus

! src/share/classes/javax/management/remote/rmi/RMIConnectionImpl.java

Changeset: bc309e9233ce
Author:    mchung
Date:      2009-11-25 11:19 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/bc309e9233ce

Merge


Changeset: 621edf6b03fc
Author:    mchung
Date:      2009-11-25 16:02 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/621edf6b03fc

Merge


Changeset: 338c8775f0a3
Author:    asaha
Date:      2009-11-26 07:17 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/338c8775f0a3

Merge


Changeset: f0b63b6d9709
Author:    asaha
Date:      2009-12-01 08:55 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/f0b63b6d9709

Merge

- test/tools/launcher/SolarisDataModel.sh
- test/tools/launcher/SolarisRunpath.sh
- test/tools/launcher/libraryCaller.c
- test/tools/launcher/libraryCaller.h
- test/tools/launcher/libraryCaller.java

Changeset: 121fa73c7185
Author:    michaelm
Date:      2009-12-02 12:17 +0000
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/121fa73c7185

6893954: Subclasses of InetAddress may incorrectly interpret network addresses
Summary: runtime type checks and deserialization check
Reviewed-by: chegar, alanb, jccollet

! src/share/classes/java/net/DatagramSocket.java
! src/share/classes/java/net/InetAddress.java
! src/share/classes/java/net/MulticastSocket.java
! src/share/classes/java/net/NetworkInterface.java
! src/share/classes/java/net/Socket.java
! src/share/classes/sun/nio/ch/Net.java

Changeset: edaa7e2efd63
Author:    asaha
Date:      2009-12-04 10:23 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/edaa7e2efd63

Merge

- make/tools/CharsetMapping/DoubleByte-X.java
- make/tools/CharsetMapping/SingleByte-X.java
- src/share/classes/sun/util/CoreResourceBundleControl-XLocales.java
- src/share/classes/sun/util/LocaleDataMetaInfo-XLocales.java
- test/java/util/Formatter/Basic-X.java

Changeset: 3598d6eb087c
Author:    xuelei
Date:      2009-12-07 21:16 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/3598d6eb087c

6898739: TLS renegotiation issue
Summary: the interim fix disables TLS/SSL renegotiation
Reviewed-by: mullan, chegar, wetmore

! src/share/classes/sun/security/ssl/ClientHandshaker.java
! src/share/classes/sun/security/ssl/Handshaker.java
! src/share/classes/sun/security/ssl/SSLEngineImpl.java
! src/share/classes/sun/security/ssl/SSLSocketImpl.java
! src/share/classes/sun/security/ssl/ServerHandshaker.java
! test/sun/security/ssl/com/sun/net/ssl/internal/ssl/SSLSocketImpl/InvalidateServerSessionRenegotiate.java
! test/sun/security/ssl/javax/net/ssl/NewAPIs/JSSERenegotiate.java
! test/sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/CheckStatus.java
! test/sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/ConnectionTest.java
! test/sun/security/ssl/javax/net/ssl/NewAPIs/SSLEngine/NoAuthClientAuth.java

Changeset: 91a4840fa9b4
Author:    mullan
Date:      2009-12-08 15:58 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/91a4840fa9b4

6633872: Policy/PolicyFile leak dynamic ProtectionDomains.
Reviewed-by: hawtin

! src/share/classes/java/security/Policy.java
! src/share/classes/java/security/ProtectionDomain.java
+ src/share/classes/sun/misc/JavaSecurityProtectionDomainAccess.java
! src/share/classes/sun/misc/SharedSecrets.java
! src/share/classes/sun/security/provider/PolicyFile.java

Changeset: 7a60d100ffa5
Author:    mullan
Date:      2009-12-18 09:09 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/7a60d100ffa5

6904162: Add new VeriSign root CA certificates to JRE and remove some old/unused ones
Reviewed-by: asaha

- test/lib/security/cacerts/VerifyCACerts.java

Changeset: 3dabb7d5be98
Author:    malenkov
Date:      2009-12-22 17:56 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/3dabb7d5be98

6904691: Java Applet Trusted Methods Chaining Privilege Escalation Vulnerability
Reviewed-by: hawtin, peterz

! src/share/classes/java/beans/EventHandler.java
! src/share/classes/java/beans/Statement.java
! test/java/beans/EventHandler/Test6277246.java
! test/java/beans/EventHandler/Test6277266.java

Changeset: c80b6350de63
Author:    michaelm
Date:      2010-01-12 12:13 +0000
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/c80b6350de63

6910590: Application can modify command array, in ProcessBuilder
Summary: clone array returned by List.toArray()
Reviewed-by: chegar, alanb

! src/share/classes/java/lang/ProcessBuilder.java

Changeset: 0667ab707c48
Author:    bae
Date:      2010-02-17 12:49 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0667ab707c48

6914866: Sun JRE ImagingLib arbitrary code execution vulnerability
Reviewed-by: prr, hawtin

! src/share/native/sun/awt/medialib/awt_ImagingLib.c
! src/share/native/sun/awt/medialib/safe_alloc.h

Changeset: 494aea51f26f
Author:    bae
Date:      2010-02-17 13:10 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/494aea51f26f

6914823: Java AWT Library Invalid Index Vulnerability
Reviewed-by: flar, hawtin

! src/share/classes/sun/awt/image/ImageRepresentation.java

Changeset: 45ead4a2c48b
Author:    bae
Date:      2010-02-17 13:32 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/45ead4a2c48b

6909597: Sun Java Runtime Environment JPEGImageReader stepX Integer Overflow Vulnerability
Reviewed-by: igor

! src/share/native/sun/awt/image/jpeg/imageioJPEG.c

Changeset: 1ff19af7b735
Author:    bae
Date:      2010-02-19 22:30 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/1ff19af7b735

6899653: Sun Java Runtime CMM readMabCurveData Buffer Overflow Vulnerability
Reviewed-by: prr, hawtin

! src/share/native/sun/java2d/cmm/lcms/cmsio1.c
! src/share/native/sun/java2d/cmm/lcms/cmsxform.c

Changeset: cda01c4b091c
Author:    ksrini
Date:      2010-02-22 14:33 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/cda01c4b091c

6902299: Java JAR "unpack200" must verify input parameters
Summary: Added several checks for addition of values before memory allocation
Reviewed-by: asaha

! src/share/native/com/sun/java/util/jar/pack/bytes.cpp
! src/share/native/com/sun/java/util/jar/pack/unpack.cpp

Changeset: 7a6b3cc68e92
Author:    denis
Date:      2010-02-26 03:54 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/7a6b3cc68e92

6887703: Unsigned applet can retrieve the dragged information before drop action occur
Reviewed-by: uta

! src/share/classes/sun/awt/dnd/SunDropTargetContextPeer.java

Changeset: c5c6f8fa92ae
Author:    denis
Date:      2010-03-06 03:37 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/c5c6f8fa92ae

6932659: JTreg test files were missed in push of 6887703
Reviewed-by: uta

! test/java/awt/regtesthelpers/process/ProcessCommunicator.java

Changeset: 2805db6e6ff6
Author:    asaha
Date:      2010-03-24 14:16 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/2805db6e6ff6

Merge

- make/java/redist/FILES.gmk
- make/java/text/FILES_java.gmk
- make/sun/nio/FILES_java.gmk
! src/share/classes/java/beans/Statement.java
! src/share/classes/java/util/zip/Deflater.java
- src/share/classes/javax/swing/plaf/synth/DefaultMenuLayout.java
- src/share/classes/sun/awt/ComponentAccessor.java
- src/share/classes/sun/awt/WindowAccessor.java
- src/share/classes/sun/dyn/util/BytecodeSignature.java
- src/share/classes/sun/security/provider/IdentityDatabase.java
! src/share/classes/sun/security/provider/PolicyFile.java
- src/share/classes/sun/security/provider/SystemIdentity.java
- src/share/classes/sun/security/provider/SystemSigner.java
- src/share/classes/sun/security/x509/X500Signer.java
- src/share/classes/sun/security/x509/X509Cert.java
- src/share/classes/sun/swing/plaf/synth/SynthUI.java
- src/share/classes/sun/tools/jar/JarVerifierStream.java
- src/solaris/classes/sun/nio/ch/SctpSocketDispatcher.java
! test/java/awt/regtesthelpers/process/ProcessCommunicator.java
- test/java/net/Socket/FDClose.java

Changeset: 1dccfa00dc64
Author:    asaha
Date:      2010-03-24 17:32 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/1dccfa00dc64

Merge

! src/share/classes/sun/security/ssl/SSLSocketImpl.java

Changeset: 6ec14b5ede77
Author:    asaha
Date:      2010-03-25 07:12 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/6ec14b5ede77

Merge


Changeset: 3ef9b3446677
Author:    asaha
Date:      2010-03-29 07:17 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/3ef9b3446677

Merge


Changeset: a9fdd143a58e
Author:    asaha
Date:      2010-04-05 16:11 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/a9fdd143a58e

Merge

- make/tools/src/build/tools/charsetmapping/CharsetMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateDBCS.java
- make/tools/src/build/tools/charsetmapping/GenerateEUC_TW.java
- make/tools/src/build/tools/charsetmapping/GenerateMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateSBCS.java
- src/share/classes/sun/io/ByteToCharHKSCS.java
- src/share/classes/sun/io/ByteToCharHKSCS_2001.java
- src/share/classes/sun/io/CharToByteHKSCS.java
- src/share/classes/sun/io/CharToByteHKSCS_2001.java
- src/share/classes/sun/nio/cs/ext/Big5.java
- src/share/classes/sun/nio/cs/ext/HKSCS_2001.java



From christopher.hegarty at sun.com  Tue Apr  6 12:54:25 2010
From: christopher.hegarty at sun.com (christopher.hegarty at sun.com)
Date: Tue, 06 Apr 2010 12:54:25 +0000
Subject: hg: jdk7/tl/jdk: 6648001: Cancelling HTTP authentication causes
	subsequent deadlocks
Message-ID: <20100406125524.CE3BB44401@hg.openjdk.java.net>

Changeset: 69002275e0e2
Author:    chegar
Date:      2010-04-06 13:47 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/69002275e0e2

6648001: Cancelling HTTP authentication causes subsequent deadlocks
Reviewed-by: michaelm

! src/share/classes/sun/net/www/protocol/http/AuthenticationInfo.java
! src/share/classes/sun/net/www/protocol/http/HttpURLConnection.java
+ test/java/net/Authenticator/Deadlock.java



From christopher.hegarty at sun.com  Tue Apr  6 14:50:51 2010
From: christopher.hegarty at sun.com (christopher.hegarty at sun.com)
Date: Tue, 06 Apr 2010 14:50:51 +0000
Subject: hg: jdk7/tl/jdk: 6921111: NullPointerException in
	PlainDatagramSocketImpl.socketSetOption
Message-ID: <20100406145132.DB24C44413@hg.openjdk.java.net>

Changeset: 495ba30cf02f
Author:    chegar
Date:      2010-04-06 15:44 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/495ba30cf02f

6921111: NullPointerException in PlainDatagramSocketImpl.socketSetOption
Reviewed-by: alanb

! src/solaris/native/java/net/PlainDatagramSocketImpl.c
! src/windows/native/java/net/TwoStacksPlainDatagramSocketImpl.c



From xueming.shen at sun.com  Tue Apr  6 22:52:51 2010
From: xueming.shen at sun.com (xueming.shen at sun.com)
Date: Tue, 06 Apr 2010 22:52:51 +0000
Subject: hg: jdk7/tl/jdk: 6717164: FilterInputStream.skip incorrectly inherits
	wording specifying how the InputStream.skip works
Message-ID: <20100406225348.066E744421@hg.openjdk.java.net>

Changeset: 0b7f10901f30
Author:    sherman
Date:      2010-04-06 15:45 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0b7f10901f30

6717164: FilterInputStream.skip incorrectly inherits wording specifying how the InputStream.skip works
Summary: restoring the javadoc
Reviewed-by: alanb

! src/share/classes/java/io/FilterInputStream.java



From martinrb at google.com  Wed Apr  7 20:08:28 2010
From: martinrb at google.com (martinrb at google.com)
Date: Wed, 07 Apr 2010 20:08:28 +0000
Subject: hg: jdk7/tl/jdk: 6941130: Semaphore should throw if number of permits
	overflows or underflows
Message-ID: <20100407200947.A579D44445@hg.openjdk.java.net>

Changeset: fc7c38b2584c
Author:    martin
Date:      2010-04-07 12:30 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/fc7c38b2584c

6941130: Semaphore should throw if number of permits overflows or underflows
Summary: Check if release could make number of permits negative
Reviewed-by: dl, dholmes

! src/share/classes/java/util/concurrent/Semaphore.java
+ test/java/util/concurrent/Semaphore/PermitOverflow.java



From sean.mullan at sun.com  Fri Apr  9 14:28:07 2010
From: sean.mullan at sun.com (sean.mullan at sun.com)
Date: Fri, 09 Apr 2010 14:28:07 +0000
Subject: hg: jdk7/tl/jdk: 6909281: 6u19 b99(pit):Error loading first applet in
	browser session( both FF && IE, windows ): NPE is thrown
Message-ID: <20100409142906.C3952444B1@hg.openjdk.java.net>

Changeset: 710c4493902f
Author:    mullan
Date:      2010-04-09 07:21 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/710c4493902f

6909281: 6u19 b99(pit):Error loading first applet in browser session( both FF && IE, windows ): NPE is thrown
Summary: Fix for 6633872 causes NPE due to uninitialised ProtectionDomain class
Reviewed-by: andrew

! src/share/classes/sun/misc/SharedSecrets.java



From jonathan.gibbons at sun.com  Fri Apr  9 22:42:07 2010
From: jonathan.gibbons at sun.com (jonathan.gibbons at sun.com)
Date: Fri, 09 Apr 2010 22:42:07 +0000
Subject: hg: jdk7/tl/langtools: 6942649: add hidden option to identify
	location and version of javac classes
Message-ID: <20100409224217.EEF8E444C2@hg.openjdk.java.net>

Changeset: 96072ad00783
Author:    jjg
Date:      2010-04-09 15:39 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/96072ad00783

6942649: add hidden option to identify location and version of javac classes
Reviewed-by: darcy

! src/share/classes/com/sun/tools/javac/main/Main.java
+ test/tools/javac/T6942649.java



From martinrb at google.com  Sat Apr 10 00:42:02 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 9 Apr 2010 17:42:02 -0700
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
Message-ID: <q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>

My feeling on whether to support O(1) at both ends
is that any flavor of this that ends up in the JDK eventually
should really do this.  My idea is that we can
wholeheartedly recommend this collection class
for overall good behavior without any of the surprising
performance traps of existing collection classes.

But for the preliminary version, it makes sense to
support only O(1) at one end, if it simplifies the
implementation.  Random access will of course
be worse than ArrayList, but by how much?
We can do some benchmarking and look for
micro-optimizations now.

Kevin, what is you own personal feeling?
Is the algorithm correct, and efficient enough?
Do you think your new collection belongs in java.util?

Martin

On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
> The data structure is available at the second link that I originally
> provided (once again, it is
> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en).
> This does not have O(1) time insertion at the front as yet as it was unclear
> to me whether or not it was agreed upon:
> _________________
> From: Osvaldo Doederlein <opinali at gmail.com>
> Date: Mon, Mar 29, 2010 at 10:08 AM
> Subject: Re: A List implementation backed by multiple small arrays rather
> than the traditional single large array.
> To: Martin Buchholz <martinrb at google.com>
> Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
> core-libs-dev at openjdk.java.net
>
> Initially, it would be good enough to replace only java.util.ArrayList with
> minimal overhead. ArrayList does not support efficient add-at-front or other
> enhancements of ArrayDeque; but ArrayList is still a much more important and
> popular collection, it's the primary "straight replacement for primitive
> arrrays" and I guess it should continue with that role.
> _________________
>
> As a disclaimer, I'm still tinkering with this so I'll be updating the
> document at the provided link as I find improvements.
>
> Thoughts?
>
> Thanks,
>
> Kevin
>
> On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <martinrb at google.com>
> wrote:
>>
>> Hi Kevin,
>>
>> You're probably the only one on this list who has
>> seriously read the paper. ?It is not surprising that
>> taking a research paper into production would
>> discover bugs - the research never had to undergo
>> rigorous testing. ?(I like the Java culture of
>> combining spec + implementation + test suite)
>>
>> I suggest you ask the authors directly about the bug.
>> They would probably also be interested to hear
>> about your implementation.
>>
>> Are you aware of Integer.numberOfLeadingZeros?
>>
>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)
>>
>> Martin
>>
>> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>> > I'm almost convinced now that the paper is incorrect.? The code below
>> > gives
>> > me the appropriate index into the index array and the offset into the
>> > data
>> > block.? That being said, remember when I mentioned that this will
>> > include a
>> > bit more work to access an element than a simple bit shift and a bit
>> > mask?
>> > Well this is more than a bit more - we'll be doing this each time an
>> > index
>> > is requested.? I'll spend some time trying to twiddle the bits to see if
>> > I
>> > can eliminate/combine some of the operations.
>> >
>> > ??? ??? for (int r = 1; r < 33; r++) {
>> > ??? ??? ??? int k = lg(r);
>> > ??? ??? ??? int floorKO2 = k >> 1;
>> > ??? ??? ??? int powFloorKO2 = (1 << floorKO2);
>> > ??? ??? ??? int p = ((1 << floorKO2) - 1) << 1;
>> > ??? ??? ??? int ceilKO2;
>> > ??? ??? ??? if ((k & 1) == 1) {
>> > ??? ??? ??? ??? ceilKO2 = floorKO2 + 1;
>> > ??? ??? ??? ??? p += powFloorKO2;
>> > ??? ??? ??? } else {
>> > ??? ??? ??? ??? ceilKO2 = floorKO2;
>> > ??? ??? ??? }
>> > ??? ??? ??? int e = r & ((1 << ceilKO2) - 1);
>> > ??? ??? ??? int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>> >
>> > ??? ??? ??? System.out.println((r - 1) + " " + (p + b) + " " + e);
>> > ??? ??? }
>> >
>> > Kevin
>> >
>> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>> > <kevin.l.stern at gmail.com>
>> > wrote:
>> >>
>> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>> >> superblocks,
>> >> the odd numbered superblocks need an additional term added (the number
>> >> of
>> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I also
>> >> came
>> >> across an alternative characterization of superblock in the paper which
>> >> states that data blocks are grouped within a superblock when they are
>> >> the
>> >> same size - to me, though, that implies that my example structure below
>> >> would be
>> >>
>> >> SB_0: [1]
>> >> SB_1: [2][2][2]
>> >> SB_2: [4][4][4][4][4][4]
>> >>
>> >> which seems to contradict my understanding of (1) below.? I must be
>> >> reading this upside down.
>> >>
>> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>> >> <kevin.l.stern at gmail.com>
>> >> wrote:
>> >>>
>> >>> What am I missing here?? In "Resizable arrays in optimal time and
>> >>> space"
>> >>> the authors define their data structure with the following property:
>> >>>
>> >>> (1)? "When superblock SB_k is fully allocated, it consists of
>> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>> >>>
>> >>> Since the superblock is zero-based indexed this implies the following
>> >>> structure:
>> >>>
>> >>> SB_0: [1]
>> >>> SB_1: [2]
>> >>> SB_2: [2][2]
>> >>> SB_3: [4][4]
>> >>> SB_4: [4][4][4][4]
>> >>> [...]
>> >>>
>> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>> >>>
>> >>> r = 100 (the binary expansion of i + 1)
>> >>> k = |r| - 1 = 2
>> >>> p = 2^k - 1 = 3
>> >>>
>> >>> What concerns me is their statement that p represents "the number of
>> >>> data
>> >>> blocks in superblocks prior to SB_k."? There are only two data blocks
>> >>> in
>> >>> superblocks prior to SB_2, not three.? Given (1) above, unless I'm
>> >>> misinterpreting it, the number of data blocks in superblocks prior to
>> >>> SB_k
>> >>> should be:
>> >>>
>> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>> >>>
>> >>> This, of course, seems to work out much better in my example above,
>> >>> giving the correct answer to my interpretation of their data
>> >>> structure, but
>> >>> I have a hard time believing that this is their mistake rather than my
>> >>> misinterpretation.
>> >>>
>> >>> Thoughts?
>> >>>
>> >>> Kevin
>> >>>
>> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <martinrb at google.com>
>> >>> wrote:
>> >>>>
>> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>> >>>> <kevin.l.stern at gmail.com>
>> >>>> wrote:
>> >>>> > Hi Martin,
>> >>>> >
>> >>>> > Thanks much for your feedback.? The first approach that comes to
>> >>>> > mind
>> >>>> > to
>> >>>> > implement O(1) time front as well as rear insertion is to create a
>> >>>> > cyclic
>> >>>> > list structure with a front/rear pointer - to insert at the front
>> >>>> > requires
>> >>>> > decrementing the front pointer (modulo the size) and to insert at
>> >>>> > the
>> >>>> > rear
>> >>>> > requires incrementing the rear pointer (modulo the size).? We need
>> >>>> > to
>> >>>> > resize
>> >>>> > when the two pointers bump into each other.? Could you explain more
>> >>>> > about
>> >>>> > your suggestion of introducing an arraylet that is shared by the
>> >>>> > front
>> >>>> > and
>> >>>> > the rear?
>> >>>>
>> >>>> It was a half-baked idea - I don't know if there's a way to turn it
>> >>>> into
>> >>>> something useful. ?I was thinking of the ArrayDeque implementation,
>> >>>> where all the elements live in a single array.
>> >>>>
>> >>>> > ?It's not clear to me how that would help and/or be a better
>> >>>> > approach than the cyclic list.? Anyhow, the paper that you
>> >>>> > reference,
>> >>>> > "Resizable arrays in optimal time and space", gives a deque so if
>> >>>> > we
>> >>>> > take
>> >>>> > that approach then the deque is specified.
>> >>>>
>> >>>> Technically, ArrayList also supports the Deque operations -
>> >>>> just not efficiently.
>> >>>
>> >>
>> >
>> >
>
>


From xuelei.fan at sun.com  Sat Apr 10 01:21:42 2010
From: xuelei.fan at sun.com (xuelei.fan at sun.com)
Date: Sat, 10 Apr 2010 01:21:42 +0000
Subject: hg: jdk7/tl/jdk: 6941936: Broken pipe error of test case
	DNSIdentities.java
Message-ID: <20100410012232.76A4C444C5@hg.openjdk.java.net>

Changeset: 89f4ec9e4b33
Author:    xuelei
Date:      2010-04-10 09:13 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/89f4ec9e4b33

6941936: Broken pipe error of test case DNSIdentities.java
Reviewed-by: chegar

! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/DNSIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/HttpsPost.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPAddressDNSIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPAddressIPIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/Identities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/Redirect.java



From dl at cs.oswego.edu  Sat Apr 10 15:02:25 2010
From: dl at cs.oswego.edu (Doug Lea)
Date: Sat, 10 Apr 2010 11:02:25 -0400
Subject: Concurrent calls to new Random() not random enough
In-Reply-To: <4BAA4EF2.1080306@sun.com>
References: <1ccfd1c11003231550x5bc5509dy3061513adafd0e40@mail.gmail.com>
	<1ccfd1c11003231917j13b11b2dl46d3a801ecd05919@mail.gmail.com>
	<4BAA4EF2.1080306@sun.com>
Message-ID: <4BC09301.3040406@cs.oswego.edu>

On 03/24/10 13:42, Xueming Shen wrote:
> 6937857: Concurrent calls to new Random() not random enough
>
> Martin Buchholz wrote:
>> [+fy, jeremymanson]
>>
>> Here's a much better test case,
>> and a proposed fix:
>>
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/RandomSeedCollisions
>>
>> This adds some initialization overhead, but also removes some
>> since
>> new Random()
>> no longer invokes a synchronized method.

Sorry for the long delay in looking at this, but if
this is still in need of a reviewer OK, then I think
it is fine.

-Doug


>>
>> ----
>> import java.util.*;
>>
>> public class RandomSeed {
>> public static void main(String[] args) throws Throwable {
>> class RandomCollector implements Runnable {
>> long[] randoms = new long[1<<22];
>> public void run() {
>> for (int i = 0; i < randoms.length; i++)
>> randoms[i] = new Random().nextLong();
>> }};
>> final int threadCount = 2;
>> List<RandomCollector> collectors = new ArrayList<RandomCollector>();
>> List<Thread> threads = new ArrayList<Thread>();
>> for (int i = 0; i < threadCount; i++) {
>> RandomCollector r = new RandomCollector();
>> collectors.add(r);
>> threads.add(new Thread(r));
>> }
>> for (Thread thread : threads)
>> thread.start();
>> for (Thread thread : threads)
>> thread.join();
>> int collisions = 0;
>> HashSet<Long> s = new HashSet<Long>();
>> for (RandomCollector r : collectors) {
>> for (long x : r.randoms) {
>> if (s.contains(x))
>> collisions++;
>> s.add(x);
>> }
>> }
>> System.out.printf("collisions=%d%n", collisions);
>> }
>> }
>>
>>
>> On Tue, Mar 23, 2010 at 15:50, Martin Buchholz <martinrb at google.com>
>> wrote:
>>> Hi Sherman,
>>>
>>> This is a bug report (sorry, no fix this time)
>>>
>>> Synopsis: Concurrent calls to new Random() not random enough
>>> Description:
>>> new Random() promises this:
>>> /**
>>> * Creates a new random number generator. This constructor sets
>>> * the seed of the random number generator to a value very likely
>>> * to be distinct from any other invocation of this constructor.
>>> */
>>>
>>> but if there are concurrent calls to new Random(), it does not
>>> do very well at fulfilling its contract.
>>>
>>> The following program should print out a number much closer to 0.
>>>
>>> import java.util.*;
>>>
>>> public class RandomSeed {
>>> public static void main(String[] args) throws Throwable {
>>> class RandomCollector implements Runnable {
>>> int[] randoms = new int[1<<21];
>>> public void run() {
>>> for (int i = 0; i < randoms.length; i++)
>>> randoms[i] = new Random().nextInt();
>>> }};
>>> final int threadCount = 2;
>>> List<RandomCollector> collectors = new ArrayList<RandomCollector>();
>>> List<Thread> threads = new ArrayList<Thread>();
>>> for (int i = 0; i < threadCount; i++) {
>>> RandomCollector r = new RandomCollector();
>>> collectors.add(r);
>>> threads.add(new Thread(r));
>>> }
>>> for (Thread thread : threads)
>>> thread.start();
>>> for (Thread thread : threads)
>>> thread.join();
>>> int collisions = 0;
>>> HashSet<Integer> s = new HashSet<Integer>();
>>> for (RandomCollector r : collectors) {
>>> for (int x : r.randoms) {
>>> if (s.contains(x))
>>> collisions++;
>>> s.add(x);
>>> }
>>> }
>>> System.out.println(collisions);
>>> }
>>> }
>>> ---
>>> ==> javac -source 1.6 -Xlint:all RandomSeed.java
>>> ==> java -esa -ea RandomSeed
>>> 876
>>>
>



From kevin.l.stern at gmail.com  Sun Apr 11 16:32:05 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Sun, 11 Apr 2010 11:32:05 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003290023u5c59f926o8ceb79fe0d3bbc6f@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
Message-ID: <y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>

Hello Martin,

I spent some time this weekend trying to bring out bugs in the
implementation; I believe the latest version to be in decent shape.  I have
also gathered some data on the performance of ChunkedArrayList over
ArrayList using the latest 1.6 JDK, which I've included below (note that the
numbers represent the time spent performing the specified operation with
ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
equivalent performance, < 1.00 indicates that ChunkedArrayList is less
costly and > 1.00 indicates that ArrayList is less costly).  I've noticed
relatively significant variability in a few of the numbers when I switch
hardware; though, these data do seem to represent rough performance
expectations.  For my test I generated x elements and then timed the process
of adding them to ArrayList/ChunkedArrayList, then I performed a get
operation on each for indices 0 through x-1 and finally I used the iterator
mechanism to retrieve the first through xth element (of course, I performed
each of these operations multiple times throwing away the timing for the
first few iterations to warm up the JVM).

Regarding the question of whether or not this belongs in java.util, I would
suggest that if it is desirable from a GC point of view to eliminate the
large backing array from ArrayList then your suggestion of achieving this by
way of a data structure that is both time and space optimal is a
particularly elegant solution as it not only guarantees that no backing
array will be larger than sqrt(n) elements but it also provides dynamic
shrinking behavior, has less maximum memory overhead than ArrayList, and
copies (asymptotically) fewer elements during a resize than ArrayList.  Of
course, this data structure does not do everything better than ArrayList; in
particular, indexed access is more costly, due to the required decomposition
of the index into backing array index and offset and the additional memory
indirection, and insertion-at-an-index is more costly, due to the multiple
array copies necessary to complete the shift.  That being said, I think that
the additional cost of indexed access is partially mitigated by the
availability of iterator and listIterator, whose implementations do not use
the index decomposition procedure, and the additional cost of
insertion-at-an-index is partially mitigated by the fact that
insertion-at-an-index is already an undesirable operation on ArrayList due
to its linear time complexity.

Kevin

1000000 elements:
Client JVM:
Add to ChunkedArrayList over ArrayList: 1.30
Indexed access ChunkedArrayList over ArrayList: 1.80
Iterator ChunkedArrayList over ArrayList: 0.52

Server JVM:
Add to ChunkedArrayList over ArrayList: 0.81
Indexed access ChunkedArrayList over ArrayList: 2.87
Iterator ChunkedArrayList over ArrayList: 1.31

100000 elements:
Client JVM:
Add to ChunkedArrayList over ArrayList: 0.96
Indexed access ChunkedArrayList over ArrayList: 1.86
Iterator ChunkedArrayList over ArrayList: 0.48

Server JVM:
Add to ChunkedArrayList over ArrayList: 0.96
Indexed access ChunkedArrayList over ArrayList: 1.89
Iterator ChunkedArrayList over ArrayList: 2.68

10000 elements:
Client JVM:
Add to ChunkedArrayList over ArrayList: 1.04
Indexed access ChunkedArrayList over ArrayList: 2.33
Iterator ChunkedArrayList over ArrayList: 0.53

Server JVM:
Add to ChunkedArrayList over ArrayList: 0.97
Indexed access ChunkedArrayList over ArrayList: 2.45
Iterator ChunkedArrayList over ArrayList: 2.52

1000 elements:
Client JVM:
Add to ChunkedArrayList over ArrayList: 0.99
Indexed access ChunkedArrayList over ArrayList: 2.27
Iterator ChunkedArrayList over ArrayList: 0.54

Server JVM:
Add to ChunkedArrayList over ArrayList: 0.84
Indexed access ChunkedArrayList over ArrayList: 1.23
Iterator ChunkedArrayList over ArrayList: 1.11


On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com> wrote:

> My feeling on whether to support O(1) at both ends
> is that any flavor of this that ends up in the JDK eventually
> should really do this.  My idea is that we can
> wholeheartedly recommend this collection class
> for overall good behavior without any of the surprising
> performance traps of existing collection classes.
>
> But for the preliminary version, it makes sense to
> support only O(1) at one end, if it simplifies the
> implementation.  Random access will of course
> be worse than ArrayList, but by how much?
> We can do some benchmarking and look for
> micro-optimizations now.
>
> Kevin, what is you own personal feeling?
> Is the algorithm correct, and efficient enough?
> Do you think your new collection belongs in java.util?
>
> Martin
>
> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com>
> wrote:
> > The data structure is available at the second link that I originally
> > provided (once again, it is
> >
> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
> ).
> > This does not have O(1) time insertion at the front as yet as it was
> unclear
> > to me whether or not it was agreed upon:
> > _________________
> > From: Osvaldo Doederlein <opinali at gmail.com>
> > Date: Mon, Mar 29, 2010 at 10:08 AM
> > Subject: Re: A List implementation backed by multiple small arrays rather
> > than the traditional single large array.
> > To: Martin Buchholz <martinrb at google.com>
> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
> > core-libs-dev at openjdk.java.net
> >
> > Initially, it would be good enough to replace only java.util.ArrayList
> with
> > minimal overhead. ArrayList does not support efficient add-at-front or
> other
> > enhancements of ArrayDeque; but ArrayList is still a much more important
> and
> > popular collection, it's the primary "straight replacement for primitive
> > arrrays" and I guess it should continue with that role.
> > _________________
> >
> > As a disclaimer, I'm still tinkering with this so I'll be updating the
> > document at the provided link as I find improvements.
> >
> > Thoughts?
> >
> > Thanks,
> >
> > Kevin
> >
> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <martinrb at google.com>
> > wrote:
> >>
> >> Hi Kevin,
> >>
> >> You're probably the only one on this list who has
> >> seriously read the paper.  It is not surprising that
> >> taking a research paper into production would
> >> discover bugs - the research never had to undergo
> >> rigorous testing.  (I like the Java culture of
> >> combining spec + implementation + test suite)
> >>
> >> I suggest you ask the authors directly about the bug.
> >> They would probably also be interested to hear
> >> about your implementation.
> >>
> >> Are you aware of Integer.numberOfLeadingZeros?
> >>
> >>
> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
> >>
> >> Martin
> >>
> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <kevin.l.stern at gmail.com>
> >> wrote:
> >> > I'm almost convinced now that the paper is incorrect.  The code below
> >> > gives
> >> > me the appropriate index into the index array and the offset into the
> >> > data
> >> > block.  That being said, remember when I mentioned that this will
> >> > include a
> >> > bit more work to access an element than a simple bit shift and a bit
> >> > mask?
> >> > Well this is more than a bit more - we'll be doing this each time an
> >> > index
> >> > is requested.  I'll spend some time trying to twiddle the bits to see
> if
> >> > I
> >> > can eliminate/combine some of the operations.
> >> >
> >> >         for (int r = 1; r < 33; r++) {
> >> >             int k = lg(r);
> >> >             int floorKO2 = k >> 1;
> >> >             int powFloorKO2 = (1 << floorKO2);
> >> >             int p = ((1 << floorKO2) - 1) << 1;
> >> >             int ceilKO2;
> >> >             if ((k & 1) == 1) {
> >> >                 ceilKO2 = floorKO2 + 1;
> >> >                 p += powFloorKO2;
> >> >             } else {
> >> >                 ceilKO2 = floorKO2;
> >> >             }
> >> >             int e = r & ((1 << ceilKO2) - 1);
> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
> >> >
> >> >             System.out.println((r - 1) + " " + (p + b) + " " + e);
> >> >         }
> >> >
> >> > Kevin
> >> >
> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
> >> > <kevin.l.stern at gmail.com>
> >> > wrote:
> >> >>
> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
> >> >> superblocks,
> >> >> the odd numbered superblocks need an additional term added (the
> number
> >> >> of
> >> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I
> also
> >> >> came
> >> >> across an alternative characterization of superblock in the paper
> which
> >> >> states that data blocks are grouped within a superblock when they are
> >> >> the
> >> >> same size - to me, though, that implies that my example structure
> below
> >> >> would be
> >> >>
> >> >> SB_0: [1]
> >> >> SB_1: [2][2][2]
> >> >> SB_2: [4][4][4][4][4][4]
> >> >>
> >> >> which seems to contradict my understanding of (1) below.  I must be
> >> >> reading this upside down.
> >> >>
> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
> >> >> <kevin.l.stern at gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> What am I missing here?  In "Resizable arrays in optimal time and
> >> >>> space"
> >> >>> the authors define their data structure with the following property:
> >> >>>
> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
> >> >>>
> >> >>> Since the superblock is zero-based indexed this implies the
> following
> >> >>> structure:
> >> >>>
> >> >>> SB_0: [1]
> >> >>> SB_1: [2]
> >> >>> SB_2: [2][2]
> >> >>> SB_3: [4][4]
> >> >>> SB_4: [4][4][4][4]
> >> >>> [...]
> >> >>>
> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
> >> >>>
> >> >>> r = 100 (the binary expansion of i + 1)
> >> >>> k = |r| - 1 = 2
> >> >>> p = 2^k - 1 = 3
> >> >>>
> >> >>> What concerns me is their statement that p represents "the number of
> >> >>> data
> >> >>> blocks in superblocks prior to SB_k."  There are only two data
> blocks
> >> >>> in
> >> >>> superblocks prior to SB_2, not three.  Given (1) above, unless I'm
> >> >>> misinterpreting it, the number of data blocks in superblocks prior
> to
> >> >>> SB_k
> >> >>> should be:
> >> >>>
> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
> >> >>>
> >> >>> This, of course, seems to work out much better in my example above,
> >> >>> giving the correct answer to my interpretation of their data
> >> >>> structure, but
> >> >>> I have a hard time believing that this is their mistake rather than
> my
> >> >>> misinterpretation.
> >> >>>
> >> >>> Thoughts?
> >> >>>
> >> >>> Kevin
> >> >>>
> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz <
> martinrb at google.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
> >> >>>> <kevin.l.stern at gmail.com>
> >> >>>> wrote:
> >> >>>> > Hi Martin,
> >> >>>> >
> >> >>>> > Thanks much for your feedback.  The first approach that comes to
> >> >>>> > mind
> >> >>>> > to
> >> >>>> > implement O(1) time front as well as rear insertion is to create
> a
> >> >>>> > cyclic
> >> >>>> > list structure with a front/rear pointer - to insert at the front
> >> >>>> > requires
> >> >>>> > decrementing the front pointer (modulo the size) and to insert at
> >> >>>> > the
> >> >>>> > rear
> >> >>>> > requires incrementing the rear pointer (modulo the size).  We
> need
> >> >>>> > to
> >> >>>> > resize
> >> >>>> > when the two pointers bump into each other.  Could you explain
> more
> >> >>>> > about
> >> >>>> > your suggestion of introducing an arraylet that is shared by the
> >> >>>> > front
> >> >>>> > and
> >> >>>> > the rear?
> >> >>>>
> >> >>>> It was a half-baked idea - I don't know if there's a way to turn it
> >> >>>> into
> >> >>>> something useful.  I was thinking of the ArrayDeque implementation,
> >> >>>> where all the elements live in a single array.
> >> >>>>
> >> >>>> >  It's not clear to me how that would help and/or be a better
> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
> >> >>>> > reference,
> >> >>>> > "Resizable arrays in optimal time and space", gives a deque so if
> >> >>>> > we
> >> >>>> > take
> >> >>>> > that approach then the deque is specified.
> >> >>>>
> >> >>>> Technically, ArrayList also supports the Deque operations -
> >> >>>> just not efficiently.
> >> >>>
> >> >>
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100411/1f750f66/attachment.html>

From martinrb at google.com  Sun Apr 11 21:17:44 2010
From: martinrb at google.com (Martin Buchholz)
Date: Sun, 11 Apr 2010 14:17:44 -0700
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1704b7a21003300425i7dd1ef7he28728ad3cdb60e2@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
Message-ID: <h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>

Hi Kevin,

Thanks for your continuing work on this.

I like the test results, and agree with your analysis.
I'm especially happy that you're beating
ArrayList at some operations.

I'd like to see O(1) addition at the beginning,
implement both List and Deque (I regret
our not having done this with ArrayDeque).

An additional property that would be nice to
have (but don't try too hard)
is to provide some kind of real-time
guarantees on the cost of an individual operation,
not just amortized time.  E.g. ArrayList.add
is worst-case O(n), making it unsuitable for use
in some real-time applications.

I will help get your changes into the obvious
software distributions.  I assume you're happy
with having this class included in any of
Doug Lea's jsr166, guava-libraries, or the JDK itself.
You should sign a Sun contributor agreement,
or whatever the Oracle equivalent is,
if you have not done so yet.

Doug Lea likes public domain,
guava-libraries likes the Apache license.

We should get various people a chance to give
a thumbs up on the design of this class -
Doug Lea, Josh Bloch.

Martin

On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
> Hello Martin,
>
> I spent some time this weekend trying to bring out bugs in the
> implementation; I believe the latest version to be in decent shape.? I have
> also gathered some data on the performance of ChunkedArrayList over
> ArrayList using the latest 1.6 JDK, which I've included below (note that the
> numbers represent the time spent performing the specified operation with
> ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
> equivalent performance, < 1.00 indicates that ChunkedArrayList is less
> costly and > 1.00 indicates that ArrayList is less costly).? I've noticed
> relatively significant variability in a few of the numbers when I switch
> hardware; though, these data do seem to represent rough performance
> expectations.? For my test I generated x elements and then timed the process
> of adding them to ArrayList/ChunkedArrayList, then I performed a get
> operation on each for indices 0 through x-1 and finally I used the iterator
> mechanism to retrieve the first through xth element (of course, I performed
> each of these operations multiple times throwing away the timing for the
> first few iterations to warm up the JVM).
>
> Regarding the question of whether or not this belongs in java.util, I would
> suggest that if it is desirable from a GC point of view to eliminate the
> large backing array from ArrayList then your suggestion of achieving this by
> way of a data structure that is both time and space optimal is a
> particularly elegant solution as it not only guarantees that no backing
> array will be larger than sqrt(n) elements but it also provides dynamic
> shrinking behavior, has less maximum memory overhead than ArrayList, and
> copies (asymptotically) fewer elements during a resize than ArrayList.? Of
> course, this data structure does not do everything better than ArrayList; in
> particular, indexed access is more costly, due to the required decomposition
> of the index into backing array index and offset and the additional memory
> indirection, and insertion-at-an-index is more costly, due to the multiple
> array copies necessary to complete the shift.? That being said, I think that
> the additional cost of indexed access is partially mitigated by the
> availability of iterator and listIterator, whose implementations do not use
> the index decomposition procedure, and the additional cost of
> insertion-at-an-index is partially mitigated by the fact that
> insertion-at-an-index is already an undesirable operation on ArrayList due
> to its linear time complexity.
>
> Kevin
>
> 1000000 elements:
> Client JVM:
> Add to ChunkedArrayList over ArrayList: 1.30
> Indexed access ChunkedArrayList over ArrayList: 1.80
> Iterator ChunkedArrayList over ArrayList: 0.52
>
> Server JVM:
> Add to ChunkedArrayList over ArrayList: 0.81
> Indexed access ChunkedArrayList over ArrayList: 2.87
> Iterator ChunkedArrayList over ArrayList: 1.31
>
> 100000 elements:
> Client JVM:
> Add to ChunkedArrayList over ArrayList: 0.96
> Indexed access ChunkedArrayList over ArrayList: 1.86
> Iterator ChunkedArrayList over ArrayList: 0.48
>
> Server JVM:
> Add to ChunkedArrayList over ArrayList: 0.96
> Indexed access ChunkedArrayList over ArrayList: 1.89
> Iterator ChunkedArrayList over ArrayList: 2.68
>
> 10000 elements:
> Client JVM:
> Add to ChunkedArrayList over ArrayList: 1.04
> Indexed access ChunkedArrayList over ArrayList: 2.33
> Iterator ChunkedArrayList over ArrayList: 0.53
>
> Server JVM:
> Add to ChunkedArrayList over ArrayList: 0.97
> Indexed access ChunkedArrayList over ArrayList: 2.45
> Iterator ChunkedArrayList over ArrayList: 2.52
>
> 1000 elements:
> Client JVM:
> Add to ChunkedArrayList over ArrayList: 0.99
> Indexed access ChunkedArrayList over ArrayList: 2.27
> Iterator ChunkedArrayList over ArrayList: 0.54
>
> Server JVM:
> Add to ChunkedArrayList over ArrayList: 0.84
> Indexed access ChunkedArrayList over ArrayList: 1.23
> Iterator ChunkedArrayList over ArrayList: 1.11
>
>
> On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com> wrote:
>>
>> My feeling on whether to support O(1) at both ends
>> is that any flavor of this that ends up in the JDK eventually
>> should really do this. ?My idea is that we can
>> wholeheartedly recommend this collection class
>> for overall good behavior without any of the surprising
>> performance traps of existing collection classes.
>>
>> But for the preliminary version, it makes sense to
>> support only O(1) at one end, if it simplifies the
>> implementation. ?Random access will of course
>> be worse than ArrayList, but by how much?
>> We can do some benchmarking and look for
>> micro-optimizations now.
>>
>> Kevin, what is you own personal feeling?
>> Is the algorithm correct, and efficient enough?
>> Do you think your new collection belongs in java.util?
>>
>> Martin
>>
>> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>> > The data structure is available at the second link that I originally
>> > provided (once again, it is
>> >
>> > https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en).
>> > This does not have O(1) time insertion at the front as yet as it was
>> > unclear
>> > to me whether or not it was agreed upon:
>> > _________________
>> > From: Osvaldo Doederlein <opinali at gmail.com>
>> > Date: Mon, Mar 29, 2010 at 10:08 AM
>> > Subject: Re: A List implementation backed by multiple small arrays
>> > rather
>> > than the traditional single large array.
>> > To: Martin Buchholz <martinrb at google.com>
>> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>> > core-libs-dev at openjdk.java.net
>> >
>> > Initially, it would be good enough to replace only java.util.ArrayList
>> > with
>> > minimal overhead. ArrayList does not support efficient add-at-front or
>> > other
>> > enhancements of ArrayDeque; but ArrayList is still a much more important
>> > and
>> > popular collection, it's the primary "straight replacement for primitive
>> > arrrays" and I guess it should continue with that role.
>> > _________________
>> >
>> > As a disclaimer, I'm still tinkering with this so I'll be updating the
>> > document at the provided link as I find improvements.
>> >
>> > Thoughts?
>> >
>> > Thanks,
>> >
>> > Kevin
>> >
>> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <martinrb at google.com>
>> > wrote:
>> >>
>> >> Hi Kevin,
>> >>
>> >> You're probably the only one on this list who has
>> >> seriously read the paper. ?It is not surprising that
>> >> taking a research paper into production would
>> >> discover bugs - the research never had to undergo
>> >> rigorous testing. ?(I like the Java culture of
>> >> combining spec + implementation + test suite)
>> >>
>> >> I suggest you ask the authors directly about the bug.
>> >> They would probably also be interested to hear
>> >> about your implementation.
>> >>
>> >> Are you aware of Integer.numberOfLeadingZeros?
>> >>
>> >>
>> >> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)
>> >>
>> >> Martin
>> >>
>> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <kevin.l.stern at gmail.com>
>> >> wrote:
>> >> > I'm almost convinced now that the paper is incorrect.? The code below
>> >> > gives
>> >> > me the appropriate index into the index array and the offset into the
>> >> > data
>> >> > block.? That being said, remember when I mentioned that this will
>> >> > include a
>> >> > bit more work to access an element than a simple bit shift and a bit
>> >> > mask?
>> >> > Well this is more than a bit more - we'll be doing this each time an
>> >> > index
>> >> > is requested.? I'll spend some time trying to twiddle the bits to see
>> >> > if
>> >> > I
>> >> > can eliminate/combine some of the operations.
>> >> >
>> >> > ??? ??? for (int r = 1; r < 33; r++) {
>> >> > ??? ??? ??? int k = lg(r);
>> >> > ??? ??? ??? int floorKO2 = k >> 1;
>> >> > ??? ??? ??? int powFloorKO2 = (1 << floorKO2);
>> >> > ??? ??? ??? int p = ((1 << floorKO2) - 1) << 1;
>> >> > ??? ??? ??? int ceilKO2;
>> >> > ??? ??? ??? if ((k & 1) == 1) {
>> >> > ??? ??? ??? ??? ceilKO2 = floorKO2 + 1;
>> >> > ??? ??? ??? ??? p += powFloorKO2;
>> >> > ??? ??? ??? } else {
>> >> > ??? ??? ??? ??? ceilKO2 = floorKO2;
>> >> > ??? ??? ??? }
>> >> > ??? ??? ??? int e = r & ((1 << ceilKO2) - 1);
>> >> > ??? ??? ??? int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>> >> >
>> >> > ??? ??? ??? System.out.println((r - 1) + " " + (p + b) + " " + e);
>> >> > ??? ??? }
>> >> >
>> >> > Kevin
>> >> >
>> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>> >> > <kevin.l.stern at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>> >> >> superblocks,
>> >> >> the odd numbered superblocks need an additional term added (the
>> >> >> number
>> >> >> of
>> >> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I
>> >> >> also
>> >> >> came
>> >> >> across an alternative characterization of superblock in the paper
>> >> >> which
>> >> >> states that data blocks are grouped within a superblock when they
>> >> >> are
>> >> >> the
>> >> >> same size - to me, though, that implies that my example structure
>> >> >> below
>> >> >> would be
>> >> >>
>> >> >> SB_0: [1]
>> >> >> SB_1: [2][2][2]
>> >> >> SB_2: [4][4][4][4][4][4]
>> >> >>
>> >> >> which seems to contradict my understanding of (1) below.? I must be
>> >> >> reading this upside down.
>> >> >>
>> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>> >> >> <kevin.l.stern at gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> What am I missing here?? In "Resizable arrays in optimal time and
>> >> >>> space"
>> >> >>> the authors define their data structure with the following
>> >> >>> property:
>> >> >>>
>> >> >>> (1)? "When superblock SB_k is fully allocated, it consists of
>> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>> >> >>>
>> >> >>> Since the superblock is zero-based indexed this implies the
>> >> >>> following
>> >> >>> structure:
>> >> >>>
>> >> >>> SB_0: [1]
>> >> >>> SB_1: [2]
>> >> >>> SB_2: [2][2]
>> >> >>> SB_3: [4][4]
>> >> >>> SB_4: [4][4][4][4]
>> >> >>> [...]
>> >> >>>
>> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>> >> >>>
>> >> >>> r = 100 (the binary expansion of i + 1)
>> >> >>> k = |r| - 1 = 2
>> >> >>> p = 2^k - 1 = 3
>> >> >>>
>> >> >>> What concerns me is their statement that p represents "the number
>> >> >>> of
>> >> >>> data
>> >> >>> blocks in superblocks prior to SB_k."? There are only two data
>> >> >>> blocks
>> >> >>> in
>> >> >>> superblocks prior to SB_2, not three.? Given (1) above, unless I'm
>> >> >>> misinterpreting it, the number of data blocks in superblocks prior
>> >> >>> to
>> >> >>> SB_k
>> >> >>> should be:
>> >> >>>
>> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>> >> >>>
>> >> >>> This, of course, seems to work out much better in my example above,
>> >> >>> giving the correct answer to my interpretation of their data
>> >> >>> structure, but
>> >> >>> I have a hard time believing that this is their mistake rather than
>> >> >>> my
>> >> >>> misinterpretation.
>> >> >>>
>> >> >>> Thoughts?
>> >> >>>
>> >> >>> Kevin
>> >> >>>
>> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>> >> >>> <martinrb at google.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>> >> >>>> <kevin.l.stern at gmail.com>
>> >> >>>> wrote:
>> >> >>>> > Hi Martin,
>> >> >>>> >
>> >> >>>> > Thanks much for your feedback.? The first approach that comes to
>> >> >>>> > mind
>> >> >>>> > to
>> >> >>>> > implement O(1) time front as well as rear insertion is to create
>> >> >>>> > a
>> >> >>>> > cyclic
>> >> >>>> > list structure with a front/rear pointer - to insert at the
>> >> >>>> > front
>> >> >>>> > requires
>> >> >>>> > decrementing the front pointer (modulo the size) and to insert
>> >> >>>> > at
>> >> >>>> > the
>> >> >>>> > rear
>> >> >>>> > requires incrementing the rear pointer (modulo the size).? We
>> >> >>>> > need
>> >> >>>> > to
>> >> >>>> > resize
>> >> >>>> > when the two pointers bump into each other.? Could you explain
>> >> >>>> > more
>> >> >>>> > about
>> >> >>>> > your suggestion of introducing an arraylet that is shared by the
>> >> >>>> > front
>> >> >>>> > and
>> >> >>>> > the rear?
>> >> >>>>
>> >> >>>> It was a half-baked idea - I don't know if there's a way to turn
>> >> >>>> it
>> >> >>>> into
>> >> >>>> something useful. ?I was thinking of the ArrayDeque
>> >> >>>> implementation,
>> >> >>>> where all the elements live in a single array.
>> >> >>>>
>> >> >>>> > ?It's not clear to me how that would help and/or be a better
>> >> >>>> > approach than the cyclic list.? Anyhow, the paper that you
>> >> >>>> > reference,
>> >> >>>> > "Resizable arrays in optimal time and space", gives a deque so
>> >> >>>> > if
>> >> >>>> > we
>> >> >>>> > take
>> >> >>>> > that approach then the deque is specified.
>> >> >>>>
>> >> >>>> Technically, ArrayList also supports the Deque operations -
>> >> >>>> just not efficiently.
>> >> >>>
>> >> >>
>> >> >
>> >> >
>> >
>> >
>
>


From xueming.shen at sun.com  Mon Apr 12 18:07:51 2010
From: xueming.shen at sun.com (xueming.shen at sun.com)
Date: Mon, 12 Apr 2010 18:07:51 +0000
Subject: hg: jdk7/tl/jdk: 6942707: nwe Big5.map for HKSCS2008 missed 5
	codepoints
Message-ID: <20100412180821.4F91D444F4@hg.openjdk.java.net>

Changeset: 507cd94489e7
Author:    sherman
Date:      2010-04-12 10:57 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/507cd94489e7

6942707: nwe Big5.map for HKSCS2008 missed 5 codepoints
Summary: updated the Big5.map table to add 5 entries
Reviewed-by: okutsu

- make/tools/CharsetMapping/Big5.c2b
! make/tools/CharsetMapping/Big5.map



From yu-ching.peng at sun.com  Mon Apr 12 23:03:03 2010
From: yu-ching.peng at sun.com (yu-ching.peng at sun.com)
Date: Mon, 12 Apr 2010 23:03:03 +0000
Subject: hg: jdk7/tl/jdk: 2 new changesets
Message-ID: <20100412230342.80E38444F8@hg.openjdk.java.net>

Changeset: 6b641c576e77
Author:    valeriep
Date:      2010-04-07 17:20 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/6b641c576e77

6918573: sun.security.pkcs11.P11RSACipher.finalize() is a scalability blocker
Summary: Removed the finalize() methods and use PhantomReference in Session to do auto clean up.
Reviewed-by: wetmore

! src/share/classes/sun/security/pkcs11/P11Cipher.java
! src/share/classes/sun/security/pkcs11/P11Digest.java
! src/share/classes/sun/security/pkcs11/P11Key.java
! src/share/classes/sun/security/pkcs11/P11Mac.java
! src/share/classes/sun/security/pkcs11/P11RSACipher.java
! src/share/classes/sun/security/pkcs11/P11Signature.java
! src/share/classes/sun/security/pkcs11/Session.java
! src/share/classes/sun/security/pkcs11/SessionManager.java

Changeset: a45217204978
Author:    valeriep
Date:      2010-04-12 15:53 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/a45217204978

Merge

- make/tools/CharsetMapping/Big5.c2b



From brucechapman at paradise.net.nz  Tue Apr 13 08:18:27 2010
From: brucechapman at paradise.net.nz (Bruce Chapman)
Date: Tue, 13 Apr 2010 20:18:27 +1200
Subject: Disambiguating empty catch blocks
Message-ID: <4BC428D3.1060004@paradise.net.nz>

Empty catch blocks are ambiguous and can mean one of

- I want to ignore this exception - the code works correctly when the 
catch block executes.
- I know that this exception will never actually occur in this 
particular case - the catch block never executes.
- The catch block is incomplete - a bug.



To enable programmers to disambiguate these situations I propose adding 
the following methods to Exception:

/**
This method does absolutely nothing. Calling it in a catch clause can be 
used to explicitly indicate that the exception itself is being 
deliberately ignored.
*/
public void ignore() {
}



and


     /**
     This method throws an AssertionError with this Exception as its
     cause.
     @throws an AssertionError
     */
     public void impossible() {
         AssertionError aPaddy =
             new AssertionError("The impossible happened");
         aPaddy.initCause(this);
         throw aPaddy;
     }



example:

     byte[] rawMessage= ...;
     String msg = null;
     try {
         msg = new String(rawMessage,"UTF-8");
     } catch (UnsupportedEncodingException e) {
         e.impossible();
     }

These two methods provide a means to make the programmer's intent 
explicit for the first two meanings of an empty catch block and 
satisfies tools (IDEs and static anaylsis) that the catch block isn't 
just declaring an unused variable, thus enabling a more robust 
interpretation of an empty catch block (or unused exception variable in 
a catch block)  as a problem to be rectified.



Some questions:

Is Exception the right home for this, or is there any value in hoisting 
it into Throwable?

Any problems which I have overlooked?

Comments?

With a little encouragement (like someone offering to mentor this), I 
will file an RFE for this, and start work on a changeset against 
openJDK7 to implement it. Yes I realise that the javadoc needs a little 
more development.

regards

Bruce Chapman





From chris.hegarty at oracle.com  Tue Apr 13 11:03:16 2010
From: chris.hegarty at oracle.com (chris.hegarty at oracle.com)
Date: Tue, 13 Apr 2010 11:03:16 +0000
Subject: hg: jdk7/tl/jdk: 6706251:
	api/java_net/NetworkInterface/index.html#misc: getDisplayName()
	returned non null but empty String
Message-ID: <20100413110423.C693744528@hg.openjdk.java.net>

Changeset: 1672f0212f02
Author:    chegar
Date:      2010-04-13 12:02 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/1672f0212f02

6706251: api/java_net/NetworkInterface/index.html#misc: getDisplayName() returned non null but empty String
Reviewed-by: alanb, michaelm, andrew

! src/share/classes/java/net/NetworkInterface.java



From kevin.l.stern at gmail.com  Tue Apr 13 11:18:48 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Tue, 13 Apr 2010 06:18:48 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<1ccfd1c11003301520g564876fehfce57def62f6d6b3@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
Message-ID: <q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>

Hi Martin,

It's interesting to note that the old circular list trick will not suffice
to turn this data structure into a deque since we might be copying all n
elements back to the front = 0 position every n^(1/2) operations (add
wouldn't amortize to O(1)).  We could use the old two stacks trick (push
elements onto one stack, flip (the bottom) half (of) the elements to the
'other' stack when the 'other' stack becomes empty), mentioned in
[Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
[Brodnik99resizablearrays] the authors suggest a method for making all
blocks roughly the same size, allowing us to expand/shrink capacity at the
beginning or the end; this is the approach that I will take to create a
deque.

The FAQ for the Sun Contributor Agreement Q3 (
http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
indicates that one should check with the project to determine where the SCA
should be sent.  Do you know where I would find this information?

Kevin

@MISC{Brodnik99resizablearrays,
    author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine and J.
Ian Munro and Robert Sedgewick},
    title = {Resizable Arrays in Optimal Time and Space},
    year = {1999}
}

On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:

> Hi Kevin,
>
> Thanks for your continuing work on this.
>
> I like the test results, and agree with your analysis.
> I'm especially happy that you're beating
> ArrayList at some operations.
>
> I'd like to see O(1) addition at the beginning,
> implement both List and Deque (I regret
> our not having done this with ArrayDeque).
>
> An additional property that would be nice to
> have (but don't try too hard)
> is to provide some kind of real-time
> guarantees on the cost of an individual operation,
> not just amortized time.  E.g. ArrayList.add
> is worst-case O(n), making it unsuitable for use
> in some real-time applications.
>
> I will help get your changes into the obvious
> software distributions.  I assume you're happy
> with having this class included in any of
> Doug Lea's jsr166, guava-libraries, or the JDK itself.
> You should sign a Sun contributor agreement,
> or whatever the Oracle equivalent is,
> if you have not done so yet.
>
> Doug Lea likes public domain,
> guava-libraries likes the Apache license.
>
> We should get various people a chance to give
> a thumbs up on the design of this class -
> Doug Lea, Josh Bloch.
>
> Martin
>
> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com>
> wrote:
> > Hello Martin,
> >
> > I spent some time this weekend trying to bring out bugs in the
> > implementation; I believe the latest version to be in decent shape.  I
> have
> > also gathered some data on the performance of ChunkedArrayList over
> > ArrayList using the latest 1.6 JDK, which I've included below (note that
> the
> > numbers represent the time spent performing the specified operation with
> > ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
> > equivalent performance, < 1.00 indicates that ChunkedArrayList is less
> > costly and > 1.00 indicates that ArrayList is less costly).  I've noticed
> > relatively significant variability in a few of the numbers when I switch
> > hardware; though, these data do seem to represent rough performance
> > expectations.  For my test I generated x elements and then timed the
> process
> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
> > operation on each for indices 0 through x-1 and finally I used the
> iterator
> > mechanism to retrieve the first through xth element (of course, I
> performed
> > each of these operations multiple times throwing away the timing for the
> > first few iterations to warm up the JVM).
> >
> > Regarding the question of whether or not this belongs in java.util, I
> would
> > suggest that if it is desirable from a GC point of view to eliminate the
> > large backing array from ArrayList then your suggestion of achieving this
> by
> > way of a data structure that is both time and space optimal is a
> > particularly elegant solution as it not only guarantees that no backing
> > array will be larger than sqrt(n) elements but it also provides dynamic
> > shrinking behavior, has less maximum memory overhead than ArrayList, and
> > copies (asymptotically) fewer elements during a resize than ArrayList.
> Of
> > course, this data structure does not do everything better than ArrayList;
> in
> > particular, indexed access is more costly, due to the required
> decomposition
> > of the index into backing array index and offset and the additional
> memory
> > indirection, and insertion-at-an-index is more costly, due to the
> multiple
> > array copies necessary to complete the shift.  That being said, I think
> that
> > the additional cost of indexed access is partially mitigated by the
> > availability of iterator and listIterator, whose implementations do not
> use
> > the index decomposition procedure, and the additional cost of
> > insertion-at-an-index is partially mitigated by the fact that
> > insertion-at-an-index is already an undesirable operation on ArrayList
> due
> > to its linear time complexity.
> >
> > Kevin
> >
> > 1000000 elements:
> > Client JVM:
> > Add to ChunkedArrayList over ArrayList: 1.30
> > Indexed access ChunkedArrayList over ArrayList: 1.80
> > Iterator ChunkedArrayList over ArrayList: 0.52
> >
> > Server JVM:
> > Add to ChunkedArrayList over ArrayList: 0.81
> > Indexed access ChunkedArrayList over ArrayList: 2.87
> > Iterator ChunkedArrayList over ArrayList: 1.31
> >
> > 100000 elements:
> > Client JVM:
> > Add to ChunkedArrayList over ArrayList: 0.96
> > Indexed access ChunkedArrayList over ArrayList: 1.86
> > Iterator ChunkedArrayList over ArrayList: 0.48
> >
> > Server JVM:
> > Add to ChunkedArrayList over ArrayList: 0.96
> > Indexed access ChunkedArrayList over ArrayList: 1.89
> > Iterator ChunkedArrayList over ArrayList: 2.68
> >
> > 10000 elements:
> > Client JVM:
> > Add to ChunkedArrayList over ArrayList: 1.04
> > Indexed access ChunkedArrayList over ArrayList: 2.33
> > Iterator ChunkedArrayList over ArrayList: 0.53
> >
> > Server JVM:
> > Add to ChunkedArrayList over ArrayList: 0.97
> > Indexed access ChunkedArrayList over ArrayList: 2.45
> > Iterator ChunkedArrayList over ArrayList: 2.52
> >
> > 1000 elements:
> > Client JVM:
> > Add to ChunkedArrayList over ArrayList: 0.99
> > Indexed access ChunkedArrayList over ArrayList: 2.27
> > Iterator ChunkedArrayList over ArrayList: 0.54
> >
> > Server JVM:
> > Add to ChunkedArrayList over ArrayList: 0.84
> > Indexed access ChunkedArrayList over ArrayList: 1.23
> > Iterator ChunkedArrayList over ArrayList: 1.11
> >
> >
> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com>
> wrote:
> >>
> >> My feeling on whether to support O(1) at both ends
> >> is that any flavor of this that ends up in the JDK eventually
> >> should really do this.  My idea is that we can
> >> wholeheartedly recommend this collection class
> >> for overall good behavior without any of the surprising
> >> performance traps of existing collection classes.
> >>
> >> But for the preliminary version, it makes sense to
> >> support only O(1) at one end, if it simplifies the
> >> implementation.  Random access will of course
> >> be worse than ArrayList, but by how much?
> >> We can do some benchmarking and look for
> >> micro-optimizations now.
> >>
> >> Kevin, what is you own personal feeling?
> >> Is the algorithm correct, and efficient enough?
> >> Do you think your new collection belongs in java.util?
> >>
> >> Martin
> >>
> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com>
> >> wrote:
> >> > The data structure is available at the second link that I originally
> >> > provided (once again, it is
> >> >
> >> >
> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
> ).
> >> > This does not have O(1) time insertion at the front as yet as it was
> >> > unclear
> >> > to me whether or not it was agreed upon:
> >> > _________________
> >> > From: Osvaldo Doederlein <opinali at gmail.com>
> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
> >> > Subject: Re: A List implementation backed by multiple small arrays
> >> > rather
> >> > than the traditional single large array.
> >> > To: Martin Buchholz <martinrb at google.com>
> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
> >> > core-libs-dev at openjdk.java.net
> >> >
> >> > Initially, it would be good enough to replace only java.util.ArrayList
> >> > with
> >> > minimal overhead. ArrayList does not support efficient add-at-front or
> >> > other
> >> > enhancements of ArrayDeque; but ArrayList is still a much more
> important
> >> > and
> >> > popular collection, it's the primary "straight replacement for
> primitive
> >> > arrrays" and I guess it should continue with that role.
> >> > _________________
> >> >
> >> > As a disclaimer, I'm still tinkering with this so I'll be updating the
> >> > document at the provided link as I find improvements.
> >> >
> >> > Thoughts?
> >> >
> >> > Thanks,
> >> >
> >> > Kevin
> >> >
> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <martinrb at google.com
> >
> >> > wrote:
> >> >>
> >> >> Hi Kevin,
> >> >>
> >> >> You're probably the only one on this list who has
> >> >> seriously read the paper.  It is not surprising that
> >> >> taking a research paper into production would
> >> >> discover bugs - the research never had to undergo
> >> >> rigorous testing.  (I like the Java culture of
> >> >> combining spec + implementation + test suite)
> >> >>
> >> >> I suggest you ask the authors directly about the bug.
> >> >> They would probably also be interested to hear
> >> >> about your implementation.
> >> >>
> >> >> Are you aware of Integer.numberOfLeadingZeros?
> >> >>
> >> >>
> >> >>
> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
> >> >>
> >> >> Martin
> >> >>
> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
> kevin.l.stern at gmail.com>
> >> >> wrote:
> >> >> > I'm almost convinced now that the paper is incorrect.  The code
> below
> >> >> > gives
> >> >> > me the appropriate index into the index array and the offset into
> the
> >> >> > data
> >> >> > block.  That being said, remember when I mentioned that this will
> >> >> > include a
> >> >> > bit more work to access an element than a simple bit shift and a
> bit
> >> >> > mask?
> >> >> > Well this is more than a bit more - we'll be doing this each time
> an
> >> >> > index
> >> >> > is requested.  I'll spend some time trying to twiddle the bits to
> see
> >> >> > if
> >> >> > I
> >> >> > can eliminate/combine some of the operations.
> >> >> >
> >> >> >         for (int r = 1; r < 33; r++) {
> >> >> >             int k = lg(r);
> >> >> >             int floorKO2 = k >> 1;
> >> >> >             int powFloorKO2 = (1 << floorKO2);
> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
> >> >> >             int ceilKO2;
> >> >> >             if ((k & 1) == 1) {
> >> >> >                 ceilKO2 = floorKO2 + 1;
> >> >> >                 p += powFloorKO2;
> >> >> >             } else {
> >> >> >                 ceilKO2 = floorKO2;
> >> >> >             }
> >> >> >             int e = r & ((1 << ceilKO2) - 1);
> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
> >> >> >
> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " + e);
> >> >> >         }
> >> >> >
> >> >> > Kevin
> >> >> >
> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
> >> >> > <kevin.l.stern at gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
> >> >> >> superblocks,
> >> >> >> the odd numbered superblocks need an additional term added (the
> >> >> >> number
> >> >> >> of
> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow, I
> >> >> >> also
> >> >> >> came
> >> >> >> across an alternative characterization of superblock in the paper
> >> >> >> which
> >> >> >> states that data blocks are grouped within a superblock when they
> >> >> >> are
> >> >> >> the
> >> >> >> same size - to me, though, that implies that my example structure
> >> >> >> below
> >> >> >> would be
> >> >> >>
> >> >> >> SB_0: [1]
> >> >> >> SB_1: [2][2][2]
> >> >> >> SB_2: [4][4][4][4][4][4]
> >> >> >>
> >> >> >> which seems to contradict my understanding of (1) below.  I must
> be
> >> >> >> reading this upside down.
> >> >> >>
> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
> >> >> >> <kevin.l.stern at gmail.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> What am I missing here?  In "Resizable arrays in optimal time and
> >> >> >>> space"
> >> >> >>> the authors define their data structure with the following
> >> >> >>> property:
> >> >> >>>
> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
> >> >> >>>
> >> >> >>> Since the superblock is zero-based indexed this implies the
> >> >> >>> following
> >> >> >>> structure:
> >> >> >>>
> >> >> >>> SB_0: [1]
> >> >> >>> SB_1: [2]
> >> >> >>> SB_2: [2][2]
> >> >> >>> SB_3: [4][4]
> >> >> >>> SB_4: [4][4][4][4]
> >> >> >>> [...]
> >> >> >>>
> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
> >> >> >>>
> >> >> >>> r = 100 (the binary expansion of i + 1)
> >> >> >>> k = |r| - 1 = 2
> >> >> >>> p = 2^k - 1 = 3
> >> >> >>>
> >> >> >>> What concerns me is their statement that p represents "the number
> >> >> >>> of
> >> >> >>> data
> >> >> >>> blocks in superblocks prior to SB_k."  There are only two data
> >> >> >>> blocks
> >> >> >>> in
> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above, unless
> I'm
> >> >> >>> misinterpreting it, the number of data blocks in superblocks
> prior
> >> >> >>> to
> >> >> >>> SB_k
> >> >> >>> should be:
> >> >> >>>
> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
> >> >> >>>
> >> >> >>> This, of course, seems to work out much better in my example
> above,
> >> >> >>> giving the correct answer to my interpretation of their data
> >> >> >>> structure, but
> >> >> >>> I have a hard time believing that this is their mistake rather
> than
> >> >> >>> my
> >> >> >>> misinterpretation.
> >> >> >>>
> >> >> >>> Thoughts?
> >> >> >>>
> >> >> >>> Kevin
> >> >> >>>
> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
> >> >> >>> <martinrb at google.com>
> >> >> >>> wrote:
> >> >> >>>>
> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
> >> >> >>>> <kevin.l.stern at gmail.com>
> >> >> >>>> wrote:
> >> >> >>>> > Hi Martin,
> >> >> >>>> >
> >> >> >>>> > Thanks much for your feedback.  The first approach that comes
> to
> >> >> >>>> > mind
> >> >> >>>> > to
> >> >> >>>> > implement O(1) time front as well as rear insertion is to
> create
> >> >> >>>> > a
> >> >> >>>> > cyclic
> >> >> >>>> > list structure with a front/rear pointer - to insert at the
> >> >> >>>> > front
> >> >> >>>> > requires
> >> >> >>>> > decrementing the front pointer (modulo the size) and to insert
> >> >> >>>> > at
> >> >> >>>> > the
> >> >> >>>> > rear
> >> >> >>>> > requires incrementing the rear pointer (modulo the size).  We
> >> >> >>>> > need
> >> >> >>>> > to
> >> >> >>>> > resize
> >> >> >>>> > when the two pointers bump into each other.  Could you explain
> >> >> >>>> > more
> >> >> >>>> > about
> >> >> >>>> > your suggestion of introducing an arraylet that is shared by
> the
> >> >> >>>> > front
> >> >> >>>> > and
> >> >> >>>> > the rear?
> >> >> >>>>
> >> >> >>>> It was a half-baked idea - I don't know if there's a way to turn
> >> >> >>>> it
> >> >> >>>> into
> >> >> >>>> something useful.  I was thinking of the ArrayDeque
> >> >> >>>> implementation,
> >> >> >>>> where all the elements live in a single array.
> >> >> >>>>
> >> >> >>>> >  It's not clear to me how that would help and/or be a better
> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
> >> >> >>>> > reference,
> >> >> >>>> > "Resizable arrays in optimal time and space", gives a deque so
> >> >> >>>> > if
> >> >> >>>> > we
> >> >> >>>> > take
> >> >> >>>> > that approach then the deque is specified.
> >> >> >>>>
> >> >> >>>> Technically, ArrayList also supports the Deque operations -
> >> >> >>>> just not efficiently.
> >> >> >>>
> >> >> >>
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100413/0cdc7b9b/attachment.html>

From kevin.l.stern at gmail.com  Tue Apr 13 11:27:15 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Tue, 13 Apr 2010 06:27:15 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
Message-ID: <x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>

Hi Martin,

I had intended to address your request for absolute O(1) operations in the
previous email.  The approach to achieving this suggested in
[Brodnik99resizablearrays] is tantamount to making ArrayList operations
absolute O(1) by keeping around an array of size (3/2)*n and filling it with
a constant number of entries from the main array each time add is called.
Although this distributes the work done during a resize across the n
operations required to enter a resize-required state, it is at the expense
of additional memory usage and slower add operations.  My thought is that
this would be a fine approach for a real-time application that requires hard
guarantees on performance but would be a liability in so many Java
applications that do not require these hard guarantees.  I look forward to
hearing your thoughts on the matter, though.

Kevin

On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:

> Hi Martin,
>
> It's interesting to note that the old circular list trick will not suffice
> to turn this data structure into a deque since we might be copying all n
> elements back to the front = 0 position every n^(1/2) operations (add
> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
> elements onto one stack, flip (the bottom) half (of) the elements to the
> 'other' stack when the 'other' stack becomes empty), mentioned in
> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
> [Brodnik99resizablearrays] the authors suggest a method for making all
> blocks roughly the same size, allowing us to expand/shrink capacity at the
> beginning or the end; this is the approach that I will take to create a
> deque.
>
> The FAQ for the Sun Contributor Agreement Q3 (
> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
> indicates that one should check with the project to determine where the SCA
> should be sent.  Do you know where I would find this information?
>
> Kevin
>
> @MISC{Brodnik99resizablearrays,
>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine and J.
> Ian Munro and Robert Sedgewick},
>     title = {Resizable Arrays in Optimal Time and Space},
>     year = {1999}
>
> }
>
> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:
>
>> Hi Kevin,
>>
>> Thanks for your continuing work on this.
>>
>> I like the test results, and agree with your analysis.
>> I'm especially happy that you're beating
>> ArrayList at some operations.
>>
>> I'd like to see O(1) addition at the beginning,
>> implement both List and Deque (I regret
>> our not having done this with ArrayDeque).
>>
>> An additional property that would be nice to
>> have (but don't try too hard)
>> is to provide some kind of real-time
>> guarantees on the cost of an individual operation,
>> not just amortized time.  E.g. ArrayList.add
>> is worst-case O(n), making it unsuitable for use
>> in some real-time applications.
>>
>> I will help get your changes into the obvious
>> software distributions.  I assume you're happy
>> with having this class included in any of
>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>> You should sign a Sun contributor agreement,
>> or whatever the Oracle equivalent is,
>> if you have not done so yet.
>>
>> Doug Lea likes public domain,
>> guava-libraries likes the Apache license.
>>
>> We should get various people a chance to give
>> a thumbs up on the design of this class -
>> Doug Lea, Josh Bloch.
>>
>> Martin
>>
>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>> > Hello Martin,
>> >
>> > I spent some time this weekend trying to bring out bugs in the
>> > implementation; I believe the latest version to be in decent shape.  I
>> have
>> > also gathered some data on the performance of ChunkedArrayList over
>> > ArrayList using the latest 1.6 JDK, which I've included below (note that
>> the
>> > numbers represent the time spent performing the specified operation with
>> > ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is less
>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>> noticed
>> > relatively significant variability in a few of the numbers when I switch
>> > hardware; though, these data do seem to represent rough performance
>> > expectations.  For my test I generated x elements and then timed the
>> process
>> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
>> > operation on each for indices 0 through x-1 and finally I used the
>> iterator
>> > mechanism to retrieve the first through xth element (of course, I
>> performed
>> > each of these operations multiple times throwing away the timing for the
>> > first few iterations to warm up the JVM).
>> >
>> > Regarding the question of whether or not this belongs in java.util, I
>> would
>> > suggest that if it is desirable from a GC point of view to eliminate the
>> > large backing array from ArrayList then your suggestion of achieving
>> this by
>> > way of a data structure that is both time and space optimal is a
>> > particularly elegant solution as it not only guarantees that no backing
>> > array will be larger than sqrt(n) elements but it also provides dynamic
>> > shrinking behavior, has less maximum memory overhead than ArrayList, and
>> > copies (asymptotically) fewer elements during a resize than ArrayList.
>> Of
>> > course, this data structure does not do everything better than
>> ArrayList; in
>> > particular, indexed access is more costly, due to the required
>> decomposition
>> > of the index into backing array index and offset and the additional
>> memory
>> > indirection, and insertion-at-an-index is more costly, due to the
>> multiple
>> > array copies necessary to complete the shift.  That being said, I think
>> that
>> > the additional cost of indexed access is partially mitigated by the
>> > availability of iterator and listIterator, whose implementations do not
>> use
>> > the index decomposition procedure, and the additional cost of
>> > insertion-at-an-index is partially mitigated by the fact that
>> > insertion-at-an-index is already an undesirable operation on ArrayList
>> due
>> > to its linear time complexity.
>> >
>> > Kevin
>> >
>> > 1000000 elements:
>> > Client JVM:
>> > Add to ChunkedArrayList over ArrayList: 1.30
>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>> > Iterator ChunkedArrayList over ArrayList: 0.52
>> >
>> > Server JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.81
>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>> > Iterator ChunkedArrayList over ArrayList: 1.31
>> >
>> > 100000 elements:
>> > Client JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.96
>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>> > Iterator ChunkedArrayList over ArrayList: 0.48
>> >
>> > Server JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.96
>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>> > Iterator ChunkedArrayList over ArrayList: 2.68
>> >
>> > 10000 elements:
>> > Client JVM:
>> > Add to ChunkedArrayList over ArrayList: 1.04
>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>> > Iterator ChunkedArrayList over ArrayList: 0.53
>> >
>> > Server JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.97
>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>> > Iterator ChunkedArrayList over ArrayList: 2.52
>> >
>> > 1000 elements:
>> > Client JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.99
>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>> > Iterator ChunkedArrayList over ArrayList: 0.54
>> >
>> > Server JVM:
>> > Add to ChunkedArrayList over ArrayList: 0.84
>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>> > Iterator ChunkedArrayList over ArrayList: 1.11
>> >
>> >
>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com>
>> wrote:
>> >>
>> >> My feeling on whether to support O(1) at both ends
>> >> is that any flavor of this that ends up in the JDK eventually
>> >> should really do this.  My idea is that we can
>> >> wholeheartedly recommend this collection class
>> >> for overall good behavior without any of the surprising
>> >> performance traps of existing collection classes.
>> >>
>> >> But for the preliminary version, it makes sense to
>> >> support only O(1) at one end, if it simplifies the
>> >> implementation.  Random access will of course
>> >> be worse than ArrayList, but by how much?
>> >> We can do some benchmarking and look for
>> >> micro-optimizations now.
>> >>
>> >> Kevin, what is you own personal feeling?
>> >> Is the algorithm correct, and efficient enough?
>> >> Do you think your new collection belongs in java.util?
>> >>
>> >> Martin
>> >>
>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com>
>> >> wrote:
>> >> > The data structure is available at the second link that I originally
>> >> > provided (once again, it is
>> >> >
>> >> >
>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>> ).
>> >> > This does not have O(1) time insertion at the front as yet as it was
>> >> > unclear
>> >> > to me whether or not it was agreed upon:
>> >> > _________________
>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>> >> > Subject: Re: A List implementation backed by multiple small arrays
>> >> > rather
>> >> > than the traditional single large array.
>> >> > To: Martin Buchholz <martinrb at google.com>
>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>> >> > core-libs-dev at openjdk.java.net
>> >> >
>> >> > Initially, it would be good enough to replace only
>> java.util.ArrayList
>> >> > with
>> >> > minimal overhead. ArrayList does not support efficient add-at-front
>> or
>> >> > other
>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>> important
>> >> > and
>> >> > popular collection, it's the primary "straight replacement for
>> primitive
>> >> > arrrays" and I guess it should continue with that role.
>> >> > _________________
>> >> >
>> >> > As a disclaimer, I'm still tinkering with this so I'll be updating
>> the
>> >> > document at the provided link as I find improvements.
>> >> >
>> >> > Thoughts?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Kevin
>> >> >
>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>> martinrb at google.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Kevin,
>> >> >>
>> >> >> You're probably the only one on this list who has
>> >> >> seriously read the paper.  It is not surprising that
>> >> >> taking a research paper into production would
>> >> >> discover bugs - the research never had to undergo
>> >> >> rigorous testing.  (I like the Java culture of
>> >> >> combining spec + implementation + test suite)
>> >> >>
>> >> >> I suggest you ask the authors directly about the bug.
>> >> >> They would probably also be interested to hear
>> >> >> about your implementation.
>> >> >>
>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>> >> >>
>> >> >>
>> >> >>
>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>> >> >>
>> >> >> Martin
>> >> >>
>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>> kevin.l.stern at gmail.com>
>> >> >> wrote:
>> >> >> > I'm almost convinced now that the paper is incorrect.  The code
>> below
>> >> >> > gives
>> >> >> > me the appropriate index into the index array and the offset into
>> the
>> >> >> > data
>> >> >> > block.  That being said, remember when I mentioned that this will
>> >> >> > include a
>> >> >> > bit more work to access an element than a simple bit shift and a
>> bit
>> >> >> > mask?
>> >> >> > Well this is more than a bit more - we'll be doing this each time
>> an
>> >> >> > index
>> >> >> > is requested.  I'll spend some time trying to twiddle the bits to
>> see
>> >> >> > if
>> >> >> > I
>> >> >> > can eliminate/combine some of the operations.
>> >> >> >
>> >> >> >         for (int r = 1; r < 33; r++) {
>> >> >> >             int k = lg(r);
>> >> >> >             int floorKO2 = k >> 1;
>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>> >> >> >             int ceilKO2;
>> >> >> >             if ((k & 1) == 1) {
>> >> >> >                 ceilKO2 = floorKO2 + 1;
>> >> >> >                 p += powFloorKO2;
>> >> >> >             } else {
>> >> >> >                 ceilKO2 = floorKO2;
>> >> >> >             }
>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>> >> >> >
>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " + e);
>> >> >> >         }
>> >> >> >
>> >> >> > Kevin
>> >> >> >
>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>> >> >> > <kevin.l.stern at gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>> >> >> >> superblocks,
>> >> >> >> the odd numbered superblocks need an additional term added (the
>> >> >> >> number
>> >> >> >> of
>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow,
>> I
>> >> >> >> also
>> >> >> >> came
>> >> >> >> across an alternative characterization of superblock in the paper
>> >> >> >> which
>> >> >> >> states that data blocks are grouped within a superblock when they
>> >> >> >> are
>> >> >> >> the
>> >> >> >> same size - to me, though, that implies that my example structure
>> >> >> >> below
>> >> >> >> would be
>> >> >> >>
>> >> >> >> SB_0: [1]
>> >> >> >> SB_1: [2][2][2]
>> >> >> >> SB_2: [4][4][4][4][4][4]
>> >> >> >>
>> >> >> >> which seems to contradict my understanding of (1) below.  I must
>> be
>> >> >> >> reading this upside down.
>> >> >> >>
>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>> >> >> >> <kevin.l.stern at gmail.com>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal time
>> and
>> >> >> >>> space"
>> >> >> >>> the authors define their data structure with the following
>> >> >> >>> property:
>> >> >> >>>
>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>> >> >> >>>
>> >> >> >>> Since the superblock is zero-based indexed this implies the
>> >> >> >>> following
>> >> >> >>> structure:
>> >> >> >>>
>> >> >> >>> SB_0: [1]
>> >> >> >>> SB_1: [2]
>> >> >> >>> SB_2: [2][2]
>> >> >> >>> SB_3: [4][4]
>> >> >> >>> SB_4: [4][4][4][4]
>> >> >> >>> [...]
>> >> >> >>>
>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>> >> >> >>>
>> >> >> >>> r = 100 (the binary expansion of i + 1)
>> >> >> >>> k = |r| - 1 = 2
>> >> >> >>> p = 2^k - 1 = 3
>> >> >> >>>
>> >> >> >>> What concerns me is their statement that p represents "the
>> number
>> >> >> >>> of
>> >> >> >>> data
>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two data
>> >> >> >>> blocks
>> >> >> >>> in
>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above, unless
>> I'm
>> >> >> >>> misinterpreting it, the number of data blocks in superblocks
>> prior
>> >> >> >>> to
>> >> >> >>> SB_k
>> >> >> >>> should be:
>> >> >> >>>
>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>> >> >> >>>
>> >> >> >>> This, of course, seems to work out much better in my example
>> above,
>> >> >> >>> giving the correct answer to my interpretation of their data
>> >> >> >>> structure, but
>> >> >> >>> I have a hard time believing that this is their mistake rather
>> than
>> >> >> >>> my
>> >> >> >>> misinterpretation.
>> >> >> >>>
>> >> >> >>> Thoughts?
>> >> >> >>>
>> >> >> >>> Kevin
>> >> >> >>>
>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>> >> >> >>> <martinrb at google.com>
>> >> >> >>> wrote:
>> >> >> >>>>
>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>> >> >> >>>> <kevin.l.stern at gmail.com>
>> >> >> >>>> wrote:
>> >> >> >>>> > Hi Martin,
>> >> >> >>>> >
>> >> >> >>>> > Thanks much for your feedback.  The first approach that comes
>> to
>> >> >> >>>> > mind
>> >> >> >>>> > to
>> >> >> >>>> > implement O(1) time front as well as rear insertion is to
>> create
>> >> >> >>>> > a
>> >> >> >>>> > cyclic
>> >> >> >>>> > list structure with a front/rear pointer - to insert at the
>> >> >> >>>> > front
>> >> >> >>>> > requires
>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>> insert
>> >> >> >>>> > at
>> >> >> >>>> > the
>> >> >> >>>> > rear
>> >> >> >>>> > requires incrementing the rear pointer (modulo the size).  We
>> >> >> >>>> > need
>> >> >> >>>> > to
>> >> >> >>>> > resize
>> >> >> >>>> > when the two pointers bump into each other.  Could you
>> explain
>> >> >> >>>> > more
>> >> >> >>>> > about
>> >> >> >>>> > your suggestion of introducing an arraylet that is shared by
>> the
>> >> >> >>>> > front
>> >> >> >>>> > and
>> >> >> >>>> > the rear?
>> >> >> >>>>
>> >> >> >>>> It was a half-baked idea - I don't know if there's a way to
>> turn
>> >> >> >>>> it
>> >> >> >>>> into
>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>> >> >> >>>> implementation,
>> >> >> >>>> where all the elements live in a single array.
>> >> >> >>>>
>> >> >> >>>> >  It's not clear to me how that would help and/or be a better
>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
>> >> >> >>>> > reference,
>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a deque
>> so
>> >> >> >>>> > if
>> >> >> >>>> > we
>> >> >> >>>> > take
>> >> >> >>>> > that approach then the deque is specified.
>> >> >> >>>>
>> >> >> >>>> Technically, ArrayList also supports the Deque operations -
>> >> >> >>>> just not efficiently.
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100413/848d812d/attachment.html>

From joe.j.kearney at googlemail.com  Tue Apr 13 11:52:01 2010
From: joe.j.kearney at googlemail.com (Joe Kearney)
Date: Tue, 13 Apr 2010 12:52:01 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com> 
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com> 
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com> 
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com> 
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com> 
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com> 
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com> 
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com> 
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com> 
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
Message-ID: <m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>

Hi Kevin, Martin,

To add another discussion point, I've been writing a draft/proof-of-concept
of retrofitting the List interface onto ArrayDeque. This works over the raw
array, it doesn't use the fancier structures being discussed elsewhere on
this list that deal with splitting huge arrays into arraylets, or that
provide for O(1) insert in the middle.

http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java

I'd be interested if you have any comments in the context of this
discussion. The code is not entirely ready yet, a couple of tests fail
(6/789) because of a corner case I haven't nailed yet, but the idea is there
at least. I'd like to add array shrinking later, when the size dips below
capacity*0.4 perhaps, to avoid flickering up and down around...

Tests show performance to be close to ArrayList for the O(1)
operations. Timings for indexed reads and writes showed
no discernible difference between implementations last time I ran the
tests. I don't understand at the moment why the iterator add at index
size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
operations that become O(1) in a circular implementation (that are
implemented and tested here) are faster than in ArrayList. Insert/remove in
the middle are somewhat faster than ArrayList because we only have to copy
at most half of the elements, except when resizing the array.

Kevin, I don't fully understand your point about not amortizing to O(1).
Certainly that's true for insert not at head or tail. Otherwise this
implementation only moves array elements to the front on an array resize
operation which happens every O(ln n) operations at most, if we do lots of
adds, maybe a little more if we add array shrinking too.  This is the same
as ArrayList. Are you just referring to the add-in-the-middle case?

Some performance results below, code for these is in the repository above
too. This was the second run, after a warmup.

Thanks,
Joe

------------------------------------------------ CircularArrayList
------------------------------------------------
  size         add         get         set   iterAdd/3   iterAdd/2
insert(5)   removeRnd   removeMid   remove(0)
    10          20          67          70         125         102
 90         240         191         138
   100          19          67          70         166         138
 94         230         194         118
  1000          28          64          67         681         538
 91         324         382         119
 10000          30          65          67        5884        4425
 94        1296        2330         124
---------------------------------------------------- ArrayList
----------------------------------------------------
  size         add         get         set   iterAdd/3   iterAdd/2
insert(5)   removeRnd   removeMid   remove(0)
    10          23          68          70         100          69
32913         162         130         105
   100          20          67          70         129         104
21944         169         134         135
  1000          29          63          67         651         506
 9602         364         333         526
 10000          30          63          66        5878        4414
 9947        2312        2280        4437

2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>

> Hi Martin,
>
> I had intended to address your request for absolute O(1) operations in the
> previous email.  The approach to achieving this suggested in
> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
> a constant number of entries from the main array each time add is called.
> Although this distributes the work done during a resize across the n
> operations required to enter a resize-required state, it is at the expense
> of additional memory usage and slower add operations.  My thought is that
> this would be a fine approach for a real-time application that requires hard
> guarantees on performance but would be a liability in so many Java
> applications that do not require these hard guarantees.  I look forward to
> hearing your thoughts on the matter, though.
>
> Kevin
>
>
> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>
>> Hi Martin,
>>
>> It's interesting to note that the old circular list trick will not suffice
>> to turn this data structure into a deque since we might be copying all n
>> elements back to the front = 0 position every n^(1/2) operations (add
>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>> elements onto one stack, flip (the bottom) half (of) the elements to the
>> 'other' stack when the 'other' stack becomes empty), mentioned in
>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>> [Brodnik99resizablearrays] the authors suggest a method for making all
>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>> beginning or the end; this is the approach that I will take to create a
>> deque.
>>
>> The FAQ for the Sun Contributor Agreement Q3 (
>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>> indicates that one should check with the project to determine where the SCA
>> should be sent.  Do you know where I would find this information?
>>
>> Kevin
>>
>> @MISC{Brodnik99resizablearrays,
>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine and
>> J. Ian Munro and Robert Sedgewick},
>>     title = {Resizable Arrays in Optimal Time and Space},
>>     year = {1999}
>>
>> }
>>
>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:
>>
>>> Hi Kevin,
>>>
>>> Thanks for your continuing work on this.
>>>
>>> I like the test results, and agree with your analysis.
>>> I'm especially happy that you're beating
>>> ArrayList at some operations.
>>>
>>> I'd like to see O(1) addition at the beginning,
>>> implement both List and Deque (I regret
>>> our not having done this with ArrayDeque).
>>>
>>> An additional property that would be nice to
>>> have (but don't try too hard)
>>> is to provide some kind of real-time
>>> guarantees on the cost of an individual operation,
>>> not just amortized time.  E.g. ArrayList.add
>>> is worst-case O(n), making it unsuitable for use
>>> in some real-time applications.
>>>
>>> I will help get your changes into the obvious
>>> software distributions.  I assume you're happy
>>> with having this class included in any of
>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>> You should sign a Sun contributor agreement,
>>> or whatever the Oracle equivalent is,
>>> if you have not done so yet.
>>>
>>> Doug Lea likes public domain,
>>> guava-libraries likes the Apache license.
>>>
>>> We should get various people a chance to give
>>> a thumbs up on the design of this class -
>>> Doug Lea, Josh Bloch.
>>>
>>> Martin
>>>
>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com>
>>> wrote:
>>> > Hello Martin,
>>> >
>>> > I spent some time this weekend trying to bring out bugs in the
>>> > implementation; I believe the latest version to be in decent shape.  I
>>> have
>>> > also gathered some data on the performance of ChunkedArrayList over
>>> > ArrayList using the latest 1.6 JDK, which I've included below (note
>>> that the
>>> > numbers represent the time spent performing the specified operation
>>> with
>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is less
>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>> noticed
>>> > relatively significant variability in a few of the numbers when I
>>> switch
>>> > hardware; though, these data do seem to represent rough performance
>>> > expectations.  For my test I generated x elements and then timed the
>>> process
>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
>>> > operation on each for indices 0 through x-1 and finally I used the
>>> iterator
>>> > mechanism to retrieve the first through xth element (of course, I
>>> performed
>>> > each of these operations multiple times throwing away the timing for
>>> the
>>> > first few iterations to warm up the JVM).
>>> >
>>> > Regarding the question of whether or not this belongs in java.util, I
>>> would
>>> > suggest that if it is desirable from a GC point of view to eliminate
>>> the
>>> > large backing array from ArrayList then your suggestion of achieving
>>> this by
>>> > way of a data structure that is both time and space optimal is a
>>> > particularly elegant solution as it not only guarantees that no backing
>>> > array will be larger than sqrt(n) elements but it also provides dynamic
>>> > shrinking behavior, has less maximum memory overhead than ArrayList,
>>> and
>>> > copies (asymptotically) fewer elements during a resize than ArrayList.
>>> Of
>>> > course, this data structure does not do everything better than
>>> ArrayList; in
>>> > particular, indexed access is more costly, due to the required
>>> decomposition
>>> > of the index into backing array index and offset and the additional
>>> memory
>>> > indirection, and insertion-at-an-index is more costly, due to the
>>> multiple
>>> > array copies necessary to complete the shift.  That being said, I think
>>> that
>>> > the additional cost of indexed access is partially mitigated by the
>>> > availability of iterator and listIterator, whose implementations do not
>>> use
>>> > the index decomposition procedure, and the additional cost of
>>> > insertion-at-an-index is partially mitigated by the fact that
>>> > insertion-at-an-index is already an undesirable operation on ArrayList
>>> due
>>> > to its linear time complexity.
>>> >
>>> > Kevin
>>> >
>>> > 1000000 elements:
>>> > Client JVM:
>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>> >
>>> > Server JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>> >
>>> > 100000 elements:
>>> > Client JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>> >
>>> > Server JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>> >
>>> > 10000 elements:
>>> > Client JVM:
>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>> >
>>> > Server JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>> >
>>> > 1000 elements:
>>> > Client JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>> >
>>> > Server JVM:
>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>> >
>>> >
>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com>
>>> wrote:
>>> >>
>>> >> My feeling on whether to support O(1) at both ends
>>> >> is that any flavor of this that ends up in the JDK eventually
>>> >> should really do this.  My idea is that we can
>>> >> wholeheartedly recommend this collection class
>>> >> for overall good behavior without any of the surprising
>>> >> performance traps of existing collection classes.
>>> >>
>>> >> But for the preliminary version, it makes sense to
>>> >> support only O(1) at one end, if it simplifies the
>>> >> implementation.  Random access will of course
>>> >> be worse than ArrayList, but by how much?
>>> >> We can do some benchmarking and look for
>>> >> micro-optimizations now.
>>> >>
>>> >> Kevin, what is you own personal feeling?
>>> >> Is the algorithm correct, and efficient enough?
>>> >> Do you think your new collection belongs in java.util?
>>> >>
>>> >> Martin
>>> >>
>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <kevin.l.stern at gmail.com
>>> >
>>> >> wrote:
>>> >> > The data structure is available at the second link that I originally
>>> >> > provided (once again, it is
>>> >> >
>>> >> >
>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>> ).
>>> >> > This does not have O(1) time insertion at the front as yet as it was
>>> >> > unclear
>>> >> > to me whether or not it was agreed upon:
>>> >> > _________________
>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>> >> > Subject: Re: A List implementation backed by multiple small arrays
>>> >> > rather
>>> >> > than the traditional single large array.
>>> >> > To: Martin Buchholz <martinrb at google.com>
>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>> >> > core-libs-dev at openjdk.java.net
>>> >> >
>>> >> > Initially, it would be good enough to replace only
>>> java.util.ArrayList
>>> >> > with
>>> >> > minimal overhead. ArrayList does not support efficient add-at-front
>>> or
>>> >> > other
>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>> important
>>> >> > and
>>> >> > popular collection, it's the primary "straight replacement for
>>> primitive
>>> >> > arrrays" and I guess it should continue with that role.
>>> >> > _________________
>>> >> >
>>> >> > As a disclaimer, I'm still tinkering with this so I'll be updating
>>> the
>>> >> > document at the provided link as I find improvements.
>>> >> >
>>> >> > Thoughts?
>>> >> >
>>> >> > Thanks,
>>> >> >
>>> >> > Kevin
>>> >> >
>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>> martinrb at google.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi Kevin,
>>> >> >>
>>> >> >> You're probably the only one on this list who has
>>> >> >> seriously read the paper.  It is not surprising that
>>> >> >> taking a research paper into production would
>>> >> >> discover bugs - the research never had to undergo
>>> >> >> rigorous testing.  (I like the Java culture of
>>> >> >> combining spec + implementation + test suite)
>>> >> >>
>>> >> >> I suggest you ask the authors directly about the bug.
>>> >> >> They would probably also be interested to hear
>>> >> >> about your implementation.
>>> >> >>
>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>> >> >>
>>> >> >>
>>> >> >>
>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>> >> >>
>>> >> >> Martin
>>> >> >>
>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>> kevin.l.stern at gmail.com>
>>> >> >> wrote:
>>> >> >> > I'm almost convinced now that the paper is incorrect.  The code
>>> below
>>> >> >> > gives
>>> >> >> > me the appropriate index into the index array and the offset into
>>> the
>>> >> >> > data
>>> >> >> > block.  That being said, remember when I mentioned that this will
>>> >> >> > include a
>>> >> >> > bit more work to access an element than a simple bit shift and a
>>> bit
>>> >> >> > mask?
>>> >> >> > Well this is more than a bit more - we'll be doing this each time
>>> an
>>> >> >> > index
>>> >> >> > is requested.  I'll spend some time trying to twiddle the bits to
>>> see
>>> >> >> > if
>>> >> >> > I
>>> >> >> > can eliminate/combine some of the operations.
>>> >> >> >
>>> >> >> >         for (int r = 1; r < 33; r++) {
>>> >> >> >             int k = lg(r);
>>> >> >> >             int floorKO2 = k >> 1;
>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>> >> >> >             int ceilKO2;
>>> >> >> >             if ((k & 1) == 1) {
>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>> >> >> >                 p += powFloorKO2;
>>> >> >> >             } else {
>>> >> >> >                 ceilKO2 = floorKO2;
>>> >> >> >             }
>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>> >> >> >
>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " +
>>> e);
>>> >> >> >         }
>>> >> >> >
>>> >> >> > Kevin
>>> >> >> >
>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>> >> >> > <kevin.l.stern at gmail.com>
>>> >> >> > wrote:
>>> >> >> >>
>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>>> >> >> >> superblocks,
>>> >> >> >> the odd numbered superblocks need an additional term added (the
>>> >> >> >> number
>>> >> >> >> of
>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation; anyhow,
>>> I
>>> >> >> >> also
>>> >> >> >> came
>>> >> >> >> across an alternative characterization of superblock in the
>>> paper
>>> >> >> >> which
>>> >> >> >> states that data blocks are grouped within a superblock when
>>> they
>>> >> >> >> are
>>> >> >> >> the
>>> >> >> >> same size - to me, though, that implies that my example
>>> structure
>>> >> >> >> below
>>> >> >> >> would be
>>> >> >> >>
>>> >> >> >> SB_0: [1]
>>> >> >> >> SB_1: [2][2][2]
>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>> >> >> >>
>>> >> >> >> which seems to contradict my understanding of (1) below.  I must
>>> be
>>> >> >> >> reading this upside down.
>>> >> >> >>
>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>> >> >> >> <kevin.l.stern at gmail.com>
>>> >> >> >> wrote:
>>> >> >> >>>
>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal time
>>> and
>>> >> >> >>> space"
>>> >> >> >>> the authors define their data structure with the following
>>> >> >> >>> property:
>>> >> >> >>>
>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>> >> >> >>>
>>> >> >> >>> Since the superblock is zero-based indexed this implies the
>>> >> >> >>> following
>>> >> >> >>> structure:
>>> >> >> >>>
>>> >> >> >>> SB_0: [1]
>>> >> >> >>> SB_1: [2]
>>> >> >> >>> SB_2: [2][2]
>>> >> >> >>> SB_3: [4][4]
>>> >> >> >>> SB_4: [4][4][4][4]
>>> >> >> >>> [...]
>>> >> >> >>>
>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>> >> >> >>>
>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>> >> >> >>> k = |r| - 1 = 2
>>> >> >> >>> p = 2^k - 1 = 3
>>> >> >> >>>
>>> >> >> >>> What concerns me is their statement that p represents "the
>>> number
>>> >> >> >>> of
>>> >> >> >>> data
>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two data
>>> >> >> >>> blocks
>>> >> >> >>> in
>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above, unless
>>> I'm
>>> >> >> >>> misinterpreting it, the number of data blocks in superblocks
>>> prior
>>> >> >> >>> to
>>> >> >> >>> SB_k
>>> >> >> >>> should be:
>>> >> >> >>>
>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>> >> >> >>>
>>> >> >> >>> This, of course, seems to work out much better in my example
>>> above,
>>> >> >> >>> giving the correct answer to my interpretation of their data
>>> >> >> >>> structure, but
>>> >> >> >>> I have a hard time believing that this is their mistake rather
>>> than
>>> >> >> >>> my
>>> >> >> >>> misinterpretation.
>>> >> >> >>>
>>> >> >> >>> Thoughts?
>>> >> >> >>>
>>> >> >> >>> Kevin
>>> >> >> >>>
>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>> >> >> >>> <martinrb at google.com>
>>> >> >> >>> wrote:
>>> >> >> >>>>
>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>> >> >> >>>> wrote:
>>> >> >> >>>> > Hi Martin,
>>> >> >> >>>> >
>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>> comes to
>>> >> >> >>>> > mind
>>> >> >> >>>> > to
>>> >> >> >>>> > implement O(1) time front as well as rear insertion is to
>>> create
>>> >> >> >>>> > a
>>> >> >> >>>> > cyclic
>>> >> >> >>>> > list structure with a front/rear pointer - to insert at the
>>> >> >> >>>> > front
>>> >> >> >>>> > requires
>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>> insert
>>> >> >> >>>> > at
>>> >> >> >>>> > the
>>> >> >> >>>> > rear
>>> >> >> >>>> > requires incrementing the rear pointer (modulo the size).
>>> We
>>> >> >> >>>> > need
>>> >> >> >>>> > to
>>> >> >> >>>> > resize
>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>> explain
>>> >> >> >>>> > more
>>> >> >> >>>> > about
>>> >> >> >>>> > your suggestion of introducing an arraylet that is shared by
>>> the
>>> >> >> >>>> > front
>>> >> >> >>>> > and
>>> >> >> >>>> > the rear?
>>> >> >> >>>>
>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way to
>>> turn
>>> >> >> >>>> it
>>> >> >> >>>> into
>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>> >> >> >>>> implementation,
>>> >> >> >>>> where all the elements live in a single array.
>>> >> >> >>>>
>>> >> >> >>>> >  It's not clear to me how that would help and/or be a better
>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
>>> >> >> >>>> > reference,
>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a deque
>>> so
>>> >> >> >>>> > if
>>> >> >> >>>> > we
>>> >> >> >>>> > take
>>> >> >> >>>> > that approach then the deque is specified.
>>> >> >> >>>>
>>> >> >> >>>> Technically, ArrayList also supports the Deque operations -
>>> >> >> >>>> just not efficiently.
>>> >> >> >>>
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100413/315fea22/attachment.html>

From martinrb at google.com  Tue Apr 13 19:26:40 2010
From: martinrb at google.com (Martin Buchholz)
Date: Tue, 13 Apr 2010 12:26:40 -0700
Subject: Disambiguating empty catch blocks
In-Reply-To: <4BC428D3.1060004@paradise.net.nz>
References: <4BC428D3.1060004@paradise.net.nz>
Message-ID: <s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>

Hi Bruce,

I don't think you're going to find enough support for adding
rarely used new methods to existing exception classes.

There's already a clear way to indicate to human readers
that an exception should be ignored or is expected

catch (SomeException ignored) { }

catch (SomeException expected) { }

All we need for true happiness is for all linty tools to
buy into this convention and to advertise it as the
One True Way to ignore exceptions.

One thing we definitely can do is to add a new constructor
to AssertionError, that takes a cause as an argument,
consistent with other exceptions.

There is a bit of a religious argument as to whether
AssertionError should only be thrown by an assert statement.
Is there a better Error to throw?
Hmmmm...checking the JDK sources, it seems that
AssertionError is the defacto standard.
I will sponsor a patch to add the new constructor to AssertionError,
which will at least make the boilerplate for throwing an
impossible error smaller.

Martin

On Tue, Apr 13, 2010 at 01:18, Bruce Chapman
<brucechapman at paradise.net.nz> wrote:
> Empty catch blocks are ambiguous and can mean one of
>
> - I want to ignore this exception - the code works correctly when the catch
> block executes.
> - I know that this exception will never actually occur in this particular
> case - the catch block never executes.
> - The catch block is incomplete - a bug.
>
>
>
> To enable programmers to disambiguate these situations I propose adding the
> following methods to Exception:
>
> /**
> This method does absolutely nothing. Calling it in a catch clause can be
> used to explicitly indicate that the exception itself is being deliberately
> ignored.
> */
> public void ignore() {
> }
>
>
>
> and
>
>
> ? ?/**
> ? ?This method throws an AssertionError with this Exception as its
> ? ?cause.
> ? ?@throws an AssertionError
> ? ?*/
> ? ?public void impossible() {
> ? ? ? ?AssertionError aPaddy =
> ? ? ? ? ? ?new AssertionError("The impossible happened");
> ? ? ? ?aPaddy.initCause(this);
> ? ? ? ?throw aPaddy;
> ? ?}
>
>
>
> example:
>
> ? ?byte[] rawMessage= ...;
> ? ?String msg = null;
> ? ?try {
> ? ? ? ?msg = new String(rawMessage,"UTF-8");
> ? ?} catch (UnsupportedEncodingException e) {
> ? ? ? ?e.impossible();
> ? ?}
>
> These two methods provide a means to make the programmer's intent explicit
> for the first two meanings of an empty catch block and satisfies tools (IDEs
> and static anaylsis) that the catch block isn't just declaring an unused
> variable, thus enabling a more robust interpretation of an empty catch block
> (or unused exception variable in a catch block) ?as a problem to be
> rectified.
>
>
>
> Some questions:
>
> Is Exception the right home for this, or is there any value in hoisting it
> into Throwable?
>
> Any problems which I have overlooked?
>
> Comments?
>
> With a little encouragement (like someone offering to mentor this), I will
> file an RFE for this, and start work on a changeset against openJDK7 to
> implement it. Yes I realise that the javadoc needs a little more
> development.
>
> regards
>
> Bruce Chapman
>
>
>
>


From develop4lasu at gmail.com  Wed Apr 14 07:26:43 2010
From: develop4lasu at gmail.com (=?UTF-8?Q?Marek_Kozie=C5=82?=)
Date: Wed, 14 Apr 2010 09:26:43 +0200
Subject: Disambiguating empty catch blocks
In-Reply-To: <4BC428D3.1060004@paradise.net.nz>
References: <4BC428D3.1060004@paradise.net.nz>
Message-ID: <v2q28bca0ff1004140026s96c8b2cct4436a3742fa500f1@mail.gmail.com>

2010/4/13 Bruce Chapman <brucechapman at paradise.net.nz>:
> Empty catch blocks are ambiguous and can mean one of
>
> - I want to ignore this exception - the code works correctly when the catch
> block executes.
> - I know that this exception will never actually occur in this particular
> case - the catch block never executes.

Hello,
Some time ago I was thinking same about it, but
You can do ignore with static import which can be added do base
library, while adding method to exception is not possible.

Exceptions that will never actually occur mean that Java is in lack of smf.
Maybe you will find my way of thinking as interesting:
http://lasu2string.blogspot.com/2010/02/forwarding-as-language.html
but I doubt that any will be don about it. ^___^

-- 
Pozdrowionka. / Regards.
Lasu aka Marek Kozie?

> - The catch block is incomplete - a bug.
>
>
>
> To enable programmers to disambiguate these situations I propose adding the
> following methods to Exception:
>
> /**
> This method does absolutely nothing. Calling it in a catch clause can be
> used to explicitly indicate that the exception itself is being deliberately
> ignored.
> */
> public void ignore() {
> }
>
>
>
> and
>
>
> ? ?/**
> ? ?This method throws an AssertionError with this Exception as its
> ? ?cause.
> ? ?@throws an AssertionError
> ? ?*/
> ? ?public void impossible() {
> ? ? ? ?AssertionError aPaddy =
> ? ? ? ? ? ?new AssertionError("The impossible happened");
> ? ? ? ?aPaddy.initCause(this);
> ? ? ? ?throw aPaddy;
> ? ?}
>
>
>
> example:
>
> ? ?byte[] rawMessage= ...;
> ? ?String msg = null;
> ? ?try {
> ? ? ? ?msg = new String(rawMessage,"UTF-8");
> ? ?} catch (UnsupportedEncodingException e) {
> ? ? ? ?e.impossible();
> ? ?}
>
> These two methods provide a means to make the programmer's intent explicit
> for the first two meanings of an empty catch block and satisfies tools (IDEs
> and static anaylsis) that the catch block isn't just declaring an unused
> variable, thus enabling a more robust interpretation of an empty catch block
> (or unused exception variable in a catch block) ?as a problem to be
> rectified.
>
>
>
> Some questions:
>
> Is Exception the right home for this, or is there any value in hoisting it
> into Throwable?
>
> Any problems which I have overlooked?
>
> Comments?
>
> With a little encouragement (like someone offering to mentor this), I will
> file an RFE for this, and start work on a changeset against openJDK7 to
> implement it. Yes I realise that the javadoc needs a little more
> development.
>
> regards
>
> Bruce Chapman
>
>
>
>


From brucechapman at paradise.net.nz  Wed Apr 14 09:42:48 2010
From: brucechapman at paradise.net.nz (Bruce Chapman)
Date: Wed, 14 Apr 2010 21:42:48 +1200
Subject: Disambiguating empty catch blocks
In-Reply-To: <l2g7d7138c11004131046lfc95c4b5j8c4bc26091fb575c@mail.gmail.com>
References: <4BC428D3.1060004@paradise.net.nz>
	<l2g7d7138c11004131046lfc95c4b5j8c4bc26091fb575c@mail.gmail.com>
Message-ID: <4BC58E18.9070508@paradise.net.nz>

Dimitris Andreou wrote:
> Usual idioms:
>
> 1) catch (Exception ignored) { }
>
> 2) catch (Exception e) { throw new AssertionError(e); }
That constructor doesn't exist (yet)
>
> Also, your impossible() method's signature is not quite right. It 
> should be returning some unchecked exception, so the calling code 
> could write "throw e.impossible();" - even if this statement would 
> never be called, and the exception is thrown from the impossible() 
> method, still this is needed in order not to confuse the compiler 
> (otherwise it would think the subsequent code could be reachable).
>
Good point.

Thanks
> Regards,
> Dimitris



From brucechapman at paradise.net.nz  Wed Apr 14 10:49:54 2010
From: brucechapman at paradise.net.nz (Bruce Chapman)
Date: Wed, 14 Apr 2010 22:49:54 +1200
Subject: Disambiguating empty catch blocks
In-Reply-To: <s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
References: <4BC428D3.1060004@paradise.net.nz>
	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
Message-ID: <4BC59DD2.4060900@paradise.net.nz>

Thanks Martin,

comment inline.

Bruce


Martin Buchholz wrote:
> Hi Bruce,
>
> I don't think you're going to find enough support for adding
> rarely used new methods to existing exception classes.
>   
Of course it is rare at present - it doesn't yet exist. So "rarely used" 
is at this point subjective and hypothetical. My feeling is that it 
would be useful, if you think not, then that is another data point. 
Maybe a weekly survey question for java.net?  - I am sure they could 
find a way to turn it into 5 choices.  :)
> There's already a clear way to indicate to human readers
> that an exception should be ignored or is expected
>
> catch (SomeException ignored) { }
>
> catch (SomeException expected) { }
>   

Oh dear, I thought we had finally got away from "magic name" solutions. 
Of course in JDK7 we'll be able to

catch (@Ignored SomeException ex) { }

catch (@Impossible SomeException ex) { }


and maybe that is an OK option. However getting those annotations added 
to the JDK is probably less likely than a couple of methods on Exception.
Also the @Impossible (or a magic variable name) can't throw an 
AssertionError without a language change and is therefore significantly 
inferior to an impossible() method.
> All we need for true happiness is for all linty tools to
> buy into this convention and to advertise it as the
> One True Way to ignore exceptions.
>   
ahhh Nirvana, but we can't MAKE that happen - but we can add methods (or 
annotations) to the JDK.
> One thing we definitely can do is to add a new constructor
> to AssertionError, that takes a cause as an argument,
> consistent with other exceptions.
>   
Yeah, that was a minor niggle - I'll check  for an RFE and get the ball 
started on that.
> There is a bit of a religious argument as to whether
> AssertionError should only be thrown by an assert statement.
> Is there a better Error to throw? 
> Hmmmm...checking the JDK sources, it seems that
> AssertionError is the defacto standard.
> I will sponsor a patch to add the new constructor to AssertionError,
> which will at least make the boilerplate for throwing an
> impossible error smaller.
>   
I guess if it had been the intention to only throw those from asserts, 
then the constructor could have been private to stop other uses, and let 
the language and compiler do some magic to let it be so, but that didn't 
happen.

Thanks

Bruce



From kevin.l.stern at gmail.com  Wed Apr 14 11:04:53 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Wed, 14 Apr 2010 06:04:53 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
Message-ID: <u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>

Hi Joe,

I was referring to the ChunkedArrayList when I stated that add does not
amortize to constant time when the data structure employs the circular list
trick to achieve deque behavior; ChunkedArrayList potentially resizes every
n^(1/2) operations.

Regarding your CircularArrayList, does it differ from Java's ArrayDeque?  I
took only a cursory look at it, so please understand if I have missed your
reason for creating CircularArrayList altogether.

Regards,

Kevin

On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney
<joe.j.kearney at googlemail.com>wrote:

> Hi Kevin, Martin,
>
> To add another discussion point, I've been writing a draft/proof-of-concept
> of retrofitting the List interface onto ArrayDeque. This works over the raw
> array, it doesn't use the fancier structures being discussed elsewhere on
> this list that deal with splitting huge arrays into arraylets, or that
> provide for O(1) insert in the middle.
>
>
> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>
> I'd be interested if you have any comments in the context of this
> discussion. The code is not entirely ready yet, a couple of tests fail
> (6/789) because of a corner case I haven't nailed yet, but the idea is there
> at least. I'd like to add array shrinking later, when the size dips below
> capacity*0.4 perhaps, to avoid flickering up and down around...
>
> Tests show performance to be close to ArrayList for the O(1)
> operations. Timings for indexed reads and writes showed
> no discernible difference between implementations last time I ran the
> tests. I don't understand at the moment why the iterator add at index
> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
> operations that become O(1) in a circular implementation (that are
> implemented and tested here) are faster than in ArrayList. Insert/remove in
> the middle are somewhat faster than ArrayList because we only have to copy
> at most half of the elements, except when resizing the array.
>
> Kevin, I don't fully understand your point about not amortizing to O(1).
> Certainly that's true for insert not at head or tail. Otherwise this
> implementation only moves array elements to the front on an array resize
> operation which happens every O(ln n) operations at most, if we do lots of
> adds, maybe a little more if we add array shrinking too.  This is the same
> as ArrayList. Are you just referring to the add-in-the-middle case?
>
> Some performance results below, code for these is in the repository above
> too. This was the second run, after a warmup.
>
> Thanks,
> Joe
>
> ------------------------------------------------ CircularArrayList
> ------------------------------------------------
>   size         add         get         set   iterAdd/3   iterAdd/2
> insert(5)   removeRnd   removeMid   remove(0)
>      10          20          67          70         125         102
>    90         240         191         138
>    100          19          67          70         166         138
>  94         230         194         118
>   1000          28          64          67         681         538
>  91         324         382         119
>   10000          30          65          67        5884        4425
>    94        1296        2330         124
> ---------------------------------------------------- ArrayList
> ----------------------------------------------------
>   size         add         get         set   iterAdd/3   iterAdd/2
> insert(5)   removeRnd   removeMid   remove(0)
>      10          23          68          70         100          69
> 32913         162         130         105
>    100          20          67          70         129         104
> 21944         169         134         135
>   1000          29          63          67         651         506
>  9602         364         333         526
>   10000          30          63          66        5878        4414
>  9947        2312        2280        4437
>
> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>
> Hi Martin,
>>
>> I had intended to address your request for absolute O(1) operations in the
>> previous email.  The approach to achieving this suggested in
>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>> a constant number of entries from the main array each time add is called.
>> Although this distributes the work done during a resize across the n
>> operations required to enter a resize-required state, it is at the expense
>> of additional memory usage and slower add operations.  My thought is that
>> this would be a fine approach for a real-time application that requires hard
>> guarantees on performance but would be a liability in so many Java
>> applications that do not require these hard guarantees.  I look forward to
>> hearing your thoughts on the matter, though.
>>
>> Kevin
>>
>>
>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>
>>> Hi Martin,
>>>
>>> It's interesting to note that the old circular list trick will not
>>> suffice to turn this data structure into a deque since we might be copying
>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>> beginning or the end; this is the approach that I will take to create a
>>> deque.
>>>
>>> The FAQ for the Sun Contributor Agreement Q3 (
>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>> indicates that one should check with the project to determine where the SCA
>>> should be sent.  Do you know where I would find this information?
>>>
>>> Kevin
>>>
>>> @MISC{Brodnik99resizablearrays,
>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine and
>>> J. Ian Munro and Robert Sedgewick},
>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>     year = {1999}
>>>
>>> }
>>>
>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> Thanks for your continuing work on this.
>>>>
>>>> I like the test results, and agree with your analysis.
>>>> I'm especially happy that you're beating
>>>> ArrayList at some operations.
>>>>
>>>> I'd like to see O(1) addition at the beginning,
>>>> implement both List and Deque (I regret
>>>> our not having done this with ArrayDeque).
>>>>
>>>> An additional property that would be nice to
>>>> have (but don't try too hard)
>>>> is to provide some kind of real-time
>>>> guarantees on the cost of an individual operation,
>>>> not just amortized time.  E.g. ArrayList.add
>>>> is worst-case O(n), making it unsuitable for use
>>>> in some real-time applications.
>>>>
>>>> I will help get your changes into the obvious
>>>> software distributions.  I assume you're happy
>>>> with having this class included in any of
>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>> You should sign a Sun contributor agreement,
>>>> or whatever the Oracle equivalent is,
>>>> if you have not done so yet.
>>>>
>>>> Doug Lea likes public domain,
>>>> guava-libraries likes the Apache license.
>>>>
>>>> We should get various people a chance to give
>>>> a thumbs up on the design of this class -
>>>> Doug Lea, Josh Bloch.
>>>>
>>>> Martin
>>>>
>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>> wrote:
>>>> > Hello Martin,
>>>> >
>>>> > I spent some time this weekend trying to bring out bugs in the
>>>> > implementation; I believe the latest version to be in decent shape.  I
>>>> have
>>>> > also gathered some data on the performance of ChunkedArrayList over
>>>> > ArrayList using the latest 1.6 JDK, which I've included below (note
>>>> that the
>>>> > numbers represent the time spent performing the specified operation
>>>> with
>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00 indicates
>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is less
>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>> noticed
>>>> > relatively significant variability in a few of the numbers when I
>>>> switch
>>>> > hardware; though, these data do seem to represent rough performance
>>>> > expectations.  For my test I generated x elements and then timed the
>>>> process
>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
>>>> > operation on each for indices 0 through x-1 and finally I used the
>>>> iterator
>>>> > mechanism to retrieve the first through xth element (of course, I
>>>> performed
>>>> > each of these operations multiple times throwing away the timing for
>>>> the
>>>> > first few iterations to warm up the JVM).
>>>> >
>>>> > Regarding the question of whether or not this belongs in java.util, I
>>>> would
>>>> > suggest that if it is desirable from a GC point of view to eliminate
>>>> the
>>>> > large backing array from ArrayList then your suggestion of achieving
>>>> this by
>>>> > way of a data structure that is both time and space optimal is a
>>>> > particularly elegant solution as it not only guarantees that no
>>>> backing
>>>> > array will be larger than sqrt(n) elements but it also provides
>>>> dynamic
>>>> > shrinking behavior, has less maximum memory overhead than ArrayList,
>>>> and
>>>> > copies (asymptotically) fewer elements during a resize than
>>>> ArrayList.  Of
>>>> > course, this data structure does not do everything better than
>>>> ArrayList; in
>>>> > particular, indexed access is more costly, due to the required
>>>> decomposition
>>>> > of the index into backing array index and offset and the additional
>>>> memory
>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>> multiple
>>>> > array copies necessary to complete the shift.  That being said, I
>>>> think that
>>>> > the additional cost of indexed access is partially mitigated by the
>>>> > availability of iterator and listIterator, whose implementations do
>>>> not use
>>>> > the index decomposition procedure, and the additional cost of
>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>> > insertion-at-an-index is already an undesirable operation on ArrayList
>>>> due
>>>> > to its linear time complexity.
>>>> >
>>>> > Kevin
>>>> >
>>>> > 1000000 elements:
>>>> > Client JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>> >
>>>> > Server JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>> >
>>>> > 100000 elements:
>>>> > Client JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>> >
>>>> > Server JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>> >
>>>> > 10000 elements:
>>>> > Client JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>> >
>>>> > Server JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>> >
>>>> > 1000 elements:
>>>> > Client JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>> >
>>>> > Server JVM:
>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>> >
>>>> >
>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com>
>>>> wrote:
>>>> >>
>>>> >> My feeling on whether to support O(1) at both ends
>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>> >> should really do this.  My idea is that we can
>>>> >> wholeheartedly recommend this collection class
>>>> >> for overall good behavior without any of the surprising
>>>> >> performance traps of existing collection classes.
>>>> >>
>>>> >> But for the preliminary version, it makes sense to
>>>> >> support only O(1) at one end, if it simplifies the
>>>> >> implementation.  Random access will of course
>>>> >> be worse than ArrayList, but by how much?
>>>> >> We can do some benchmarking and look for
>>>> >> micro-optimizations now.
>>>> >>
>>>> >> Kevin, what is you own personal feeling?
>>>> >> Is the algorithm correct, and efficient enough?
>>>> >> Do you think your new collection belongs in java.util?
>>>> >>
>>>> >> Martin
>>>> >>
>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>> kevin.l.stern at gmail.com>
>>>> >> wrote:
>>>> >> > The data structure is available at the second link that I
>>>> originally
>>>> >> > provided (once again, it is
>>>> >> >
>>>> >> >
>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>> ).
>>>> >> > This does not have O(1) time insertion at the front as yet as it
>>>> was
>>>> >> > unclear
>>>> >> > to me whether or not it was agreed upon:
>>>> >> > _________________
>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>> >> > Subject: Re: A List implementation backed by multiple small arrays
>>>> >> > rather
>>>> >> > than the traditional single large array.
>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>> >> > core-libs-dev at openjdk.java.net
>>>> >> >
>>>> >> > Initially, it would be good enough to replace only
>>>> java.util.ArrayList
>>>> >> > with
>>>> >> > minimal overhead. ArrayList does not support efficient add-at-front
>>>> or
>>>> >> > other
>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>> important
>>>> >> > and
>>>> >> > popular collection, it's the primary "straight replacement for
>>>> primitive
>>>> >> > arrrays" and I guess it should continue with that role.
>>>> >> > _________________
>>>> >> >
>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be updating
>>>> the
>>>> >> > document at the provided link as I find improvements.
>>>> >> >
>>>> >> > Thoughts?
>>>> >> >
>>>> >> > Thanks,
>>>> >> >
>>>> >> > Kevin
>>>> >> >
>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>> martinrb at google.com>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Hi Kevin,
>>>> >> >>
>>>> >> >> You're probably the only one on this list who has
>>>> >> >> seriously read the paper.  It is not surprising that
>>>> >> >> taking a research paper into production would
>>>> >> >> discover bugs - the research never had to undergo
>>>> >> >> rigorous testing.  (I like the Java culture of
>>>> >> >> combining spec + implementation + test suite)
>>>> >> >>
>>>> >> >> I suggest you ask the authors directly about the bug.
>>>> >> >> They would probably also be interested to hear
>>>> >> >> about your implementation.
>>>> >> >>
>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>> >> >>
>>>> >> >> Martin
>>>> >> >>
>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>> kevin.l.stern at gmail.com>
>>>> >> >> wrote:
>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The code
>>>> below
>>>> >> >> > gives
>>>> >> >> > me the appropriate index into the index array and the offset
>>>> into the
>>>> >> >> > data
>>>> >> >> > block.  That being said, remember when I mentioned that this
>>>> will
>>>> >> >> > include a
>>>> >> >> > bit more work to access an element than a simple bit shift and a
>>>> bit
>>>> >> >> > mask?
>>>> >> >> > Well this is more than a bit more - we'll be doing this each
>>>> time an
>>>> >> >> > index
>>>> >> >> > is requested.  I'll spend some time trying to twiddle the bits
>>>> to see
>>>> >> >> > if
>>>> >> >> > I
>>>> >> >> > can eliminate/combine some of the operations.
>>>> >> >> >
>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>> >> >> >             int k = lg(r);
>>>> >> >> >             int floorKO2 = k >> 1;
>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>> >> >> >             int ceilKO2;
>>>> >> >> >             if ((k & 1) == 1) {
>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>> >> >> >                 p += powFloorKO2;
>>>> >> >> >             } else {
>>>> >> >> >                 ceilKO2 = floorKO2;
>>>> >> >> >             }
>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>> >> >> >
>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " +
>>>> e);
>>>> >> >> >         }
>>>> >> >> >
>>>> >> >> > Kevin
>>>> >> >> >
>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>> >> >> > <kevin.l.stern at gmail.com>
>>>> >> >> > wrote:
>>>> >> >> >>
>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>>>> >> >> >> superblocks,
>>>> >> >> >> the odd numbered superblocks need an additional term added (the
>>>> >> >> >> number
>>>> >> >> >> of
>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>> anyhow, I
>>>> >> >> >> also
>>>> >> >> >> came
>>>> >> >> >> across an alternative characterization of superblock in the
>>>> paper
>>>> >> >> >> which
>>>> >> >> >> states that data blocks are grouped within a superblock when
>>>> they
>>>> >> >> >> are
>>>> >> >> >> the
>>>> >> >> >> same size - to me, though, that implies that my example
>>>> structure
>>>> >> >> >> below
>>>> >> >> >> would be
>>>> >> >> >>
>>>> >> >> >> SB_0: [1]
>>>> >> >> >> SB_1: [2][2][2]
>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>> >> >> >>
>>>> >> >> >> which seems to contradict my understanding of (1) below.  I
>>>> must be
>>>> >> >> >> reading this upside down.
>>>> >> >> >>
>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>> >> >> >> wrote:
>>>> >> >> >>>
>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal time
>>>> and
>>>> >> >> >>> space"
>>>> >> >> >>> the authors define their data structure with the following
>>>> >> >> >>> property:
>>>> >> >> >>>
>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>> >> >> >>>
>>>> >> >> >>> Since the superblock is zero-based indexed this implies the
>>>> >> >> >>> following
>>>> >> >> >>> structure:
>>>> >> >> >>>
>>>> >> >> >>> SB_0: [1]
>>>> >> >> >>> SB_1: [2]
>>>> >> >> >>> SB_2: [2][2]
>>>> >> >> >>> SB_3: [4][4]
>>>> >> >> >>> SB_4: [4][4][4][4]
>>>> >> >> >>> [...]
>>>> >> >> >>>
>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>> >> >> >>>
>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>> >> >> >>> k = |r| - 1 = 2
>>>> >> >> >>> p = 2^k - 1 = 3
>>>> >> >> >>>
>>>> >> >> >>> What concerns me is their statement that p represents "the
>>>> number
>>>> >> >> >>> of
>>>> >> >> >>> data
>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two data
>>>> >> >> >>> blocks
>>>> >> >> >>> in
>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above, unless
>>>> I'm
>>>> >> >> >>> misinterpreting it, the number of data blocks in superblocks
>>>> prior
>>>> >> >> >>> to
>>>> >> >> >>> SB_k
>>>> >> >> >>> should be:
>>>> >> >> >>>
>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>> >> >> >>>
>>>> >> >> >>> This, of course, seems to work out much better in my example
>>>> above,
>>>> >> >> >>> giving the correct answer to my interpretation of their data
>>>> >> >> >>> structure, but
>>>> >> >> >>> I have a hard time believing that this is their mistake rather
>>>> than
>>>> >> >> >>> my
>>>> >> >> >>> misinterpretation.
>>>> >> >> >>>
>>>> >> >> >>> Thoughts?
>>>> >> >> >>>
>>>> >> >> >>> Kevin
>>>> >> >> >>>
>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>> >> >> >>> <martinrb at google.com>
>>>> >> >> >>> wrote:
>>>> >> >> >>>>
>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>> >> >> >>>> wrote:
>>>> >> >> >>>> > Hi Martin,
>>>> >> >> >>>> >
>>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>>> comes to
>>>> >> >> >>>> > mind
>>>> >> >> >>>> > to
>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is to
>>>> create
>>>> >> >> >>>> > a
>>>> >> >> >>>> > cyclic
>>>> >> >> >>>> > list structure with a front/rear pointer - to insert at the
>>>> >> >> >>>> > front
>>>> >> >> >>>> > requires
>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>>> insert
>>>> >> >> >>>> > at
>>>> >> >> >>>> > the
>>>> >> >> >>>> > rear
>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the size).
>>>> We
>>>> >> >> >>>> > need
>>>> >> >> >>>> > to
>>>> >> >> >>>> > resize
>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>> explain
>>>> >> >> >>>> > more
>>>> >> >> >>>> > about
>>>> >> >> >>>> > your suggestion of introducing an arraylet that is shared
>>>> by the
>>>> >> >> >>>> > front
>>>> >> >> >>>> > and
>>>> >> >> >>>> > the rear?
>>>> >> >> >>>>
>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way to
>>>> turn
>>>> >> >> >>>> it
>>>> >> >> >>>> into
>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>> >> >> >>>> implementation,
>>>> >> >> >>>> where all the elements live in a single array.
>>>> >> >> >>>>
>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>> better
>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
>>>> >> >> >>>> > reference,
>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a deque
>>>> so
>>>> >> >> >>>> > if
>>>> >> >> >>>> > we
>>>> >> >> >>>> > take
>>>> >> >> >>>> > that approach then the deque is specified.
>>>> >> >> >>>>
>>>> >> >> >>>> Technically, ArrayList also supports the Deque operations -
>>>> >> >> >>>> just not efficiently.
>>>> >> >> >>>
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100414/4df9f9a1/attachment.html>

From joe.j.kearney at googlemail.com  Wed Apr 14 11:25:43 2010
From: joe.j.kearney at googlemail.com (Joe Kearney)
Date: Wed, 14 Apr 2010 12:25:43 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com> 
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com> 
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com> 
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com> 
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com> 
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com> 
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com> 
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com> 
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com> 
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
Message-ID: <x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>

Hi Kevin,

It implements List, as well as Deque. It is indeed based on ArrayDeque, with
the added operations to implement list. It does so reasonably efficiently,
moving the fewest elements possible on each operation, that is zero for the
queue operations, at most n/2 for the rest and all of them for a backing
array resize.

The idea is to get a replacement for arraylist that performs like arraydeque
on remove(0). As a side effect, we should be able to get better performance
on other operations by requiring fewer elements to be moved.

Thanks,
Joe

2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>

> Hi Joe,
>
> I was referring to the ChunkedArrayList when I stated that add does not
> amortize to constant time when the data structure employs the circular list
> trick to achieve deque behavior; ChunkedArrayList potentially resizes every
> n^(1/2) operations.
>
> Regarding your CircularArrayList, does it differ from Java's ArrayDeque?  I
> took only a cursory look at it, so please understand if I have missed your
> reason for creating CircularArrayList altogether.
>
> Regards,
>
> Kevin
>
>
> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <joe.j.kearney at googlemail.com
> > wrote:
>
>> Hi Kevin, Martin,
>>
>> To add another discussion point, I've been writing a
>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>> This works over the raw array, it doesn't use the fancier structures being
>> discussed elsewhere on this list that deal with splitting huge arrays into
>> arraylets, or that provide for O(1) insert in the middle.
>>
>>
>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>
>> I'd be interested if you have any comments in the context of this
>> discussion. The code is not entirely ready yet, a couple of tests fail
>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>> at least. I'd like to add array shrinking later, when the size dips below
>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>
>> Tests show performance to be close to ArrayList for the O(1)
>> operations. Timings for indexed reads and writes showed
>> no discernible difference between implementations last time I ran the
>> tests. I don't understand at the moment why the iterator add at index
>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>> operations that become O(1) in a circular implementation (that are
>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>> the middle are somewhat faster than ArrayList because we only have to copy
>> at most half of the elements, except when resizing the array.
>>
>> Kevin, I don't fully understand your point about not amortizing to O(1).
>> Certainly that's true for insert not at head or tail. Otherwise this
>> implementation only moves array elements to the front on an array resize
>> operation which happens every O(ln n) operations at most, if we do lots of
>> adds, maybe a little more if we add array shrinking too.  This is the same
>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>
>> Some performance results below, code for these is in the repository above
>> too. This was the second run, after a warmup.
>>
>> Thanks,
>> Joe
>>
>> ------------------------------------------------ CircularArrayList
>> ------------------------------------------------
>>   size         add         get         set   iterAdd/3   iterAdd/2
>> insert(5)   removeRnd   removeMid   remove(0)
>>      10          20          67          70         125         102
>>    90         240         191         138
>>    100          19          67          70         166         138
>>  94         230         194         118
>>   1000          28          64          67         681         538
>>  91         324         382         119
>>   10000          30          65          67        5884        4425
>>    94        1296        2330         124
>> ---------------------------------------------------- ArrayList
>> ----------------------------------------------------
>>   size         add         get         set   iterAdd/3   iterAdd/2
>> insert(5)   removeRnd   removeMid   remove(0)
>>      10          23          68          70         100          69
>> 32913         162         130         105
>>    100          20          67          70         129         104
>> 21944         169         134         135
>>   1000          29          63          67         651         506
>>  9602         364         333         526
>>   10000          30          63          66        5878        4414
>>  9947        2312        2280        4437
>>
>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>
>> Hi Martin,
>>>
>>> I had intended to address your request for absolute O(1) operations in
>>> the previous email.  The approach to achieving this suggested in
>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>> a constant number of entries from the main array each time add is called.
>>> Although this distributes the work done during a resize across the n
>>> operations required to enter a resize-required state, it is at the expense
>>> of additional memory usage and slower add operations.  My thought is that
>>> this would be a fine approach for a real-time application that requires hard
>>> guarantees on performance but would be a liability in so many Java
>>> applications that do not require these hard guarantees.  I look forward to
>>> hearing your thoughts on the matter, though.
>>>
>>> Kevin
>>>
>>>
>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <kevin.l.stern at gmail.com
>>> > wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> It's interesting to note that the old circular list trick will not
>>>> suffice to turn this data structure into a deque since we might be copying
>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>> beginning or the end; this is the approach that I will take to create a
>>>> deque.
>>>>
>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>> indicates that one should check with the project to determine where the SCA
>>>> should be sent.  Do you know where I would find this information?
>>>>
>>>> Kevin
>>>>
>>>> @MISC{Brodnik99resizablearrays,
>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine and
>>>> J. Ian Munro and Robert Sedgewick},
>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>     year = {1999}
>>>>
>>>> }
>>>>
>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> Thanks for your continuing work on this.
>>>>>
>>>>> I like the test results, and agree with your analysis.
>>>>> I'm especially happy that you're beating
>>>>> ArrayList at some operations.
>>>>>
>>>>> I'd like to see O(1) addition at the beginning,
>>>>> implement both List and Deque (I regret
>>>>> our not having done this with ArrayDeque).
>>>>>
>>>>> An additional property that would be nice to
>>>>> have (but don't try too hard)
>>>>> is to provide some kind of real-time
>>>>> guarantees on the cost of an individual operation,
>>>>> not just amortized time.  E.g. ArrayList.add
>>>>> is worst-case O(n), making it unsuitable for use
>>>>> in some real-time applications.
>>>>>
>>>>> I will help get your changes into the obvious
>>>>> software distributions.  I assume you're happy
>>>>> with having this class included in any of
>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>> You should sign a Sun contributor agreement,
>>>>> or whatever the Oracle equivalent is,
>>>>> if you have not done so yet.
>>>>>
>>>>> Doug Lea likes public domain,
>>>>> guava-libraries likes the Apache license.
>>>>>
>>>>> We should get various people a chance to give
>>>>> a thumbs up on the design of this class -
>>>>> Doug Lea, Josh Bloch.
>>>>>
>>>>> Martin
>>>>>
>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>> wrote:
>>>>> > Hello Martin,
>>>>> >
>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>> > implementation; I believe the latest version to be in decent shape.
>>>>> I have
>>>>> > also gathered some data on the performance of ChunkedArrayList over
>>>>> > ArrayList using the latest 1.6 JDK, which I've included below (note
>>>>> that the
>>>>> > numbers represent the time spent performing the specified operation
>>>>> with
>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>> indicates
>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is
>>>>> less
>>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>>> noticed
>>>>> > relatively significant variability in a few of the numbers when I
>>>>> switch
>>>>> > hardware; though, these data do seem to represent rough performance
>>>>> > expectations.  For my test I generated x elements and then timed the
>>>>> process
>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
>>>>> > operation on each for indices 0 through x-1 and finally I used the
>>>>> iterator
>>>>> > mechanism to retrieve the first through xth element (of course, I
>>>>> performed
>>>>> > each of these operations multiple times throwing away the timing for
>>>>> the
>>>>> > first few iterations to warm up the JVM).
>>>>> >
>>>>> > Regarding the question of whether or not this belongs in java.util, I
>>>>> would
>>>>> > suggest that if it is desirable from a GC point of view to eliminate
>>>>> the
>>>>> > large backing array from ArrayList then your suggestion of achieving
>>>>> this by
>>>>> > way of a data structure that is both time and space optimal is a
>>>>> > particularly elegant solution as it not only guarantees that no
>>>>> backing
>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>> dynamic
>>>>> > shrinking behavior, has less maximum memory overhead than ArrayList,
>>>>> and
>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>> ArrayList.  Of
>>>>> > course, this data structure does not do everything better than
>>>>> ArrayList; in
>>>>> > particular, indexed access is more costly, due to the required
>>>>> decomposition
>>>>> > of the index into backing array index and offset and the additional
>>>>> memory
>>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>>> multiple
>>>>> > array copies necessary to complete the shift.  That being said, I
>>>>> think that
>>>>> > the additional cost of indexed access is partially mitigated by the
>>>>> > availability of iterator and listIterator, whose implementations do
>>>>> not use
>>>>> > the index decomposition procedure, and the additional cost of
>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>> ArrayList due
>>>>> > to its linear time complexity.
>>>>> >
>>>>> > Kevin
>>>>> >
>>>>> > 1000000 elements:
>>>>> > Client JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>> >
>>>>> > Server JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>> >
>>>>> > 100000 elements:
>>>>> > Client JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>> >
>>>>> > Server JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>> >
>>>>> > 10000 elements:
>>>>> > Client JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>> >
>>>>> > Server JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>> >
>>>>> > 1000 elements:
>>>>> > Client JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>> >
>>>>> > Server JVM:
>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>> >
>>>>> >
>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <martinrb at google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> My feeling on whether to support O(1) at both ends
>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>> >> should really do this.  My idea is that we can
>>>>> >> wholeheartedly recommend this collection class
>>>>> >> for overall good behavior without any of the surprising
>>>>> >> performance traps of existing collection classes.
>>>>> >>
>>>>> >> But for the preliminary version, it makes sense to
>>>>> >> support only O(1) at one end, if it simplifies the
>>>>> >> implementation.  Random access will of course
>>>>> >> be worse than ArrayList, but by how much?
>>>>> >> We can do some benchmarking and look for
>>>>> >> micro-optimizations now.
>>>>> >>
>>>>> >> Kevin, what is you own personal feeling?
>>>>> >> Is the algorithm correct, and efficient enough?
>>>>> >> Do you think your new collection belongs in java.util?
>>>>> >>
>>>>> >> Martin
>>>>> >>
>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>> kevin.l.stern at gmail.com>
>>>>> >> wrote:
>>>>> >> > The data structure is available at the second link that I
>>>>> originally
>>>>> >> > provided (once again, it is
>>>>> >> >
>>>>> >> >
>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>> ).
>>>>> >> > This does not have O(1) time insertion at the front as yet as it
>>>>> was
>>>>> >> > unclear
>>>>> >> > to me whether or not it was agreed upon:
>>>>> >> > _________________
>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>> >> > Subject: Re: A List implementation backed by multiple small arrays
>>>>> >> > rather
>>>>> >> > than the traditional single large array.
>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>> >> > core-libs-dev at openjdk.java.net
>>>>> >> >
>>>>> >> > Initially, it would be good enough to replace only
>>>>> java.util.ArrayList
>>>>> >> > with
>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>> add-at-front or
>>>>> >> > other
>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>>> important
>>>>> >> > and
>>>>> >> > popular collection, it's the primary "straight replacement for
>>>>> primitive
>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>> >> > _________________
>>>>> >> >
>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be updating
>>>>> the
>>>>> >> > document at the provided link as I find improvements.
>>>>> >> >
>>>>> >> > Thoughts?
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> >
>>>>> >> > Kevin
>>>>> >> >
>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>> martinrb at google.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> Hi Kevin,
>>>>> >> >>
>>>>> >> >> You're probably the only one on this list who has
>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>> >> >> taking a research paper into production would
>>>>> >> >> discover bugs - the research never had to undergo
>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>> >> >> combining spec + implementation + test suite)
>>>>> >> >>
>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>> >> >> They would probably also be interested to hear
>>>>> >> >> about your implementation.
>>>>> >> >>
>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>> >> >>
>>>>> >> >>
>>>>> >> >>
>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>> >> >>
>>>>> >> >> Martin
>>>>> >> >>
>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>> kevin.l.stern at gmail.com>
>>>>> >> >> wrote:
>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The code
>>>>> below
>>>>> >> >> > gives
>>>>> >> >> > me the appropriate index into the index array and the offset
>>>>> into the
>>>>> >> >> > data
>>>>> >> >> > block.  That being said, remember when I mentioned that this
>>>>> will
>>>>> >> >> > include a
>>>>> >> >> > bit more work to access an element than a simple bit shift and
>>>>> a bit
>>>>> >> >> > mask?
>>>>> >> >> > Well this is more than a bit more - we'll be doing this each
>>>>> time an
>>>>> >> >> > index
>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the bits
>>>>> to see
>>>>> >> >> > if
>>>>> >> >> > I
>>>>> >> >> > can eliminate/combine some of the operations.
>>>>> >> >> >
>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>> >> >> >             int k = lg(r);
>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>> >> >> >             int ceilKO2;
>>>>> >> >> >             if ((k & 1) == 1) {
>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>> >> >> >                 p += powFloorKO2;
>>>>> >> >> >             } else {
>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>> >> >> >             }
>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>> >> >> >
>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " +
>>>>> e);
>>>>> >> >> >         }
>>>>> >> >> >
>>>>> >> >> > Kevin
>>>>> >> >> >
>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>> >> >> > wrote:
>>>>> >> >> >>
>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>>>>> >> >> >> superblocks,
>>>>> >> >> >> the odd numbered superblocks need an additional term added
>>>>> (the
>>>>> >> >> >> number
>>>>> >> >> >> of
>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>> anyhow, I
>>>>> >> >> >> also
>>>>> >> >> >> came
>>>>> >> >> >> across an alternative characterization of superblock in the
>>>>> paper
>>>>> >> >> >> which
>>>>> >> >> >> states that data blocks are grouped within a superblock when
>>>>> they
>>>>> >> >> >> are
>>>>> >> >> >> the
>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>> structure
>>>>> >> >> >> below
>>>>> >> >> >> would be
>>>>> >> >> >>
>>>>> >> >> >> SB_0: [1]
>>>>> >> >> >> SB_1: [2][2][2]
>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>> >> >> >>
>>>>> >> >> >> which seems to contradict my understanding of (1) below.  I
>>>>> must be
>>>>> >> >> >> reading this upside down.
>>>>> >> >> >>
>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>> >> >> >> wrote:
>>>>> >> >> >>>
>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal time
>>>>> and
>>>>> >> >> >>> space"
>>>>> >> >> >>> the authors define their data structure with the following
>>>>> >> >> >>> property:
>>>>> >> >> >>>
>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists of
>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>> >> >> >>>
>>>>> >> >> >>> Since the superblock is zero-based indexed this implies the
>>>>> >> >> >>> following
>>>>> >> >> >>> structure:
>>>>> >> >> >>>
>>>>> >> >> >>> SB_0: [1]
>>>>> >> >> >>> SB_1: [2]
>>>>> >> >> >>> SB_2: [2][2]
>>>>> >> >> >>> SB_3: [4][4]
>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>> >> >> >>> [...]
>>>>> >> >> >>>
>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>> >> >> >>>
>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>> >> >> >>> k = |r| - 1 = 2
>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>> >> >> >>>
>>>>> >> >> >>> What concerns me is their statement that p represents "the
>>>>> number
>>>>> >> >> >>> of
>>>>> >> >> >>> data
>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two
>>>>> data
>>>>> >> >> >>> blocks
>>>>> >> >> >>> in
>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>> unless I'm
>>>>> >> >> >>> misinterpreting it, the number of data blocks in superblocks
>>>>> prior
>>>>> >> >> >>> to
>>>>> >> >> >>> SB_k
>>>>> >> >> >>> should be:
>>>>> >> >> >>>
>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>> >> >> >>>
>>>>> >> >> >>> This, of course, seems to work out much better in my example
>>>>> above,
>>>>> >> >> >>> giving the correct answer to my interpretation of their data
>>>>> >> >> >>> structure, but
>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>> rather than
>>>>> >> >> >>> my
>>>>> >> >> >>> misinterpretation.
>>>>> >> >> >>>
>>>>> >> >> >>> Thoughts?
>>>>> >> >> >>>
>>>>> >> >> >>> Kevin
>>>>> >> >> >>>
>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>> >> >> >>> <martinrb at google.com>
>>>>> >> >> >>> wrote:
>>>>> >> >> >>>>
>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>> >> >> >>>> wrote:
>>>>> >> >> >>>> > Hi Martin,
>>>>> >> >> >>>> >
>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>>>> comes to
>>>>> >> >> >>>> > mind
>>>>> >> >> >>>> > to
>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is to
>>>>> create
>>>>> >> >> >>>> > a
>>>>> >> >> >>>> > cyclic
>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert at
>>>>> the
>>>>> >> >> >>>> > front
>>>>> >> >> >>>> > requires
>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>>>> insert
>>>>> >> >> >>>> > at
>>>>> >> >> >>>> > the
>>>>> >> >> >>>> > rear
>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the size).
>>>>> We
>>>>> >> >> >>>> > need
>>>>> >> >> >>>> > to
>>>>> >> >> >>>> > resize
>>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>>> explain
>>>>> >> >> >>>> > more
>>>>> >> >> >>>> > about
>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is shared
>>>>> by the
>>>>> >> >> >>>> > front
>>>>> >> >> >>>> > and
>>>>> >> >> >>>> > the rear?
>>>>> >> >> >>>>
>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way to
>>>>> turn
>>>>> >> >> >>>> it
>>>>> >> >> >>>> into
>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>> >> >> >>>> implementation,
>>>>> >> >> >>>> where all the elements live in a single array.
>>>>> >> >> >>>>
>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>> better
>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that you
>>>>> >> >> >>>> > reference,
>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>> deque so
>>>>> >> >> >>>> > if
>>>>> >> >> >>>> > we
>>>>> >> >> >>>> > take
>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>> >> >> >>>>
>>>>> >> >> >>>> Technically, ArrayList also supports the Deque operations -
>>>>> >> >> >>>> just not efficiently.
>>>>> >> >> >>>
>>>>> >> >> >>
>>>>> >> >> >
>>>>> >> >> >
>>>>> >> >
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100414/87ef210d/attachment.html>

From mario.torre at aicas.com  Wed Apr 14 11:26:38 2010
From: mario.torre at aicas.com (Mario Torre)
Date: Wed, 14 Apr 2010 13:26:38 +0200
Subject: Disambiguating empty catch blocks
In-Reply-To: <4BC59DD2.4060900@paradise.net.nz>
References: <4BC428D3.1060004@paradise.net.nz>
	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
	<4BC59DD2.4060900@paradise.net.nz>
Message-ID: <1271244398.2416.52.camel@localhost>

Il giorno mer, 14/04/2010 alle 22.49 +1200, Bruce Chapman ha scritto:
> Thanks Martin,
> 
> comment inline.
> 
> Bruce
> 
> 
> Martin Buchholz wrote:
> > Hi Bruce,
> >
> > I don't think you're going to find enough support for adding
> > rarely used new methods to existing exception classes.
> >   
> Of course it is rare at present - it doesn't yet exist. So "rarely used" 
> is at this point subjective and hypothetical. My feeling is that it 
> would be useful, if you think not, then that is another data point. 
> Maybe a weekly survey question for java.net?  - I am sure they could 
> find a way to turn it into 5 choices.  :)
> > There's already a clear way to indicate to human readers
> > that an exception should be ignored or is expected
> >
> > catch (SomeException ignored) { }
> >
> > catch (SomeException expected) { }
> >   
> 
> Oh dear, I thought we had finally got away from "magic name" solutions. 
> Of course in JDK7 we'll be able to
> 
> catch (@Ignored SomeException ex) { }
> 
> catch (@Impossible SomeException ex) { }

It think I like this one.

I also used

catch (SomeException _) { /* ignored */ }

The _ is a clear sign. If you don't name it, it means you're not
interested in it, and the comment is purely optional then.

Cheers,
Mario
-- 
Mario Torre, Software Developer, http://www.jroller.com/neugens/
aicas GmbH, Haid-und-Neu-Stra?e 18 * D-76131 Karlsruhe * Germany
http://www.aicas.com   * Tel: +49-721-663 968-0
USt-Id: DE216375633, Handelsregister HRB 109481, AG Mannheim
Gesch?ftsf?hrer: Dr. James J. Hunt

pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Please, support open standards: http://endsoftpatents.org/



From David.Holmes at oracle.com  Wed Apr 14 11:33:21 2010
From: David.Holmes at oracle.com (David Holmes)
Date: Wed, 14 Apr 2010 21:33:21 +1000
Subject: Disambiguating empty catch blocks
In-Reply-To: <4BC59DD2.4060900@paradise.net.nz>
References: <4BC428D3.1060004@paradise.net.nz>	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
	<4BC59DD2.4060900@paradise.net.nz>
Message-ID: <4BC5A801.4000201@oracle.com>

Bruce Chapman said the following on 04/14/10 20:49:
> Martin Buchholz wrote:
>> I don't think you're going to find enough support for adding
>> rarely used new methods to existing exception classes.
>>   
> Of course it is rare at present - it doesn't yet exist. So "rarely used" 
> is at this point subjective and hypothetical. My feeling is that it 
> would be useful, if you think not, then that is another data point. 
> Maybe a weekly survey question for java.net?  - I am sure they could 
> find a way to turn it into 5 choices.  :)

I agree with Martin these methods are unnecessary and would be rarely 
used if they existed. That's subjective of course but all library design 
involves some considered subjectivity about likelihood of use.

>> There's already a clear way to indicate to human readers
>> that an exception should be ignored or is expected
>>
>> catch (SomeException ignored) { }
>>
>> catch (SomeException expected) { }
>>   
> 
> Oh dear, I thought we had finally got away from "magic name" solutions. 

I don't see anything wrong with this. On the other hand I find it 
completely <insert strong adjective> to signify you are ignoring an 
exception by calling an ignore() method that does nothing. This isn't a 
coding issue it is a documentation issue - what is the intent? Why is 
the occurrence of the exception of no consequence? Using:

catch (SomeException ignore) { }

is fine though perhaps a little obscure; using

catch (SomeException ignore { /* reason we can ignore*/ }

is better. Using:

catch (SomeException e) { e.ignore(); }

is just pointless and just as obscure.

> Of course in JDK7 we'll be able to
> 
> catch (@Ignored SomeException ex) { }
> 
> catch (@Impossible SomeException ex) { }

Annotations are overkill for this.

As for "impossible":

throw new Error("can't happen");

Seems fine to me.

My 2c.

David Holmes

> and maybe that is an OK option. However getting those annotations added 
> to the JDK is probably less likely than a couple of methods on Exception.
> Also the @Impossible (or a magic variable name) can't throw an 
> AssertionError without a language change and is therefore significantly 
> inferior to an impossible() method.
>> All we need for true happiness is for all linty tools to
>> buy into this convention and to advertise it as the
>> One True Way to ignore exceptions.
>>   
> ahhh Nirvana, but we can't MAKE that happen - but we can add methods (or 
> annotations) to the JDK.
>> One thing we definitely can do is to add a new constructor
>> to AssertionError, that takes a cause as an argument,
>> consistent with other exceptions.
>>   
> Yeah, that was a minor niggle - I'll check  for an RFE and get the ball 
> started on that.
>> There is a bit of a religious argument as to whether
>> AssertionError should only be thrown by an assert statement.
>> Is there a better Error to throw? Hmmmm...checking the JDK sources, it 
>> seems that
>> AssertionError is the defacto standard.
>> I will sponsor a patch to add the new constructor to AssertionError,
>> which will at least make the boilerplate for throwing an
>> impossible error smaller.
>>   
> I guess if it had been the intention to only throw those from asserts, 
> then the constructor could have been private to stop other uses, and let 
> the language and compiler do some magic to let it be so, but that didn't 
> happen.
> 
> Thanks
> 
> Bruce
> 


From maurizio.cimadamore at sun.com  Wed Apr 14 11:41:19 2010
From: maurizio.cimadamore at sun.com (maurizio.cimadamore at sun.com)
Date: Wed, 14 Apr 2010 11:41:19 +0000
Subject: hg: jdk7/tl/langtools: 2 new changesets
Message-ID: <20100414114128.E2D2A4454C@hg.openjdk.java.net>

Changeset: 396b117c1743
Author:    mcimadamore
Date:      2010-04-14 12:23 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/396b117c1743

6939618: Revert 'simple' diamond implementation
Summary: backout changeset for 6840638
Reviewed-by: jjg

! src/share/classes/com/sun/tools/javac/code/Source.java
! src/share/classes/com/sun/tools/javac/comp/Attr.java
! src/share/classes/com/sun/tools/javac/comp/Check.java
! src/share/classes/com/sun/tools/javac/parser/JavacParser.java
! src/share/classes/com/sun/tools/javac/resources/compiler.properties
! src/share/classes/com/sun/tools/javac/tree/TreeInfo.java
- test/tools/javac/generics/diamond/neg/Neg01.java
- test/tools/javac/generics/diamond/neg/Neg01.out
- test/tools/javac/generics/diamond/neg/Neg02.java
- test/tools/javac/generics/diamond/neg/Neg02.out
- test/tools/javac/generics/diamond/neg/Neg03.java
- test/tools/javac/generics/diamond/neg/Neg03.out
- test/tools/javac/generics/diamond/neg/Neg04.java
- test/tools/javac/generics/diamond/neg/Neg04.out
- test/tools/javac/generics/diamond/neg/Neg05.java
- test/tools/javac/generics/diamond/neg/Neg05.out
- test/tools/javac/generics/diamond/pos/Pos01.java
- test/tools/javac/generics/diamond/pos/Pos02.java
- test/tools/javac/generics/diamond/pos/Pos03.java
- test/tools/javac/generics/diamond/pos/Pos04.java
! test/tools/javac/processing/environment/round/TestElementsAnnotatedWith.java
! test/tools/javac/processing/model/element/TestAnonClassNames.java

Changeset: 9d9d08922405
Author:    mcimadamore
Date:      2010-04-14 12:31 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/9d9d08922405

6939620: Switch to 'complex' diamond inference scheme
Summary: Implement new inference scheme for diamond operator that takes into account type of actual arguments supplied to constructor
Reviewed-by: jjg, darcy

! src/share/classes/com/sun/tools/javac/code/Source.java
! src/share/classes/com/sun/tools/javac/code/Symbol.java
! src/share/classes/com/sun/tools/javac/comp/Attr.java
! src/share/classes/com/sun/tools/javac/comp/Check.java
! src/share/classes/com/sun/tools/javac/comp/Resolve.java
! src/share/classes/com/sun/tools/javac/parser/JavacParser.java
! src/share/classes/com/sun/tools/javac/resources/compiler.properties
! src/share/classes/com/sun/tools/javac/tree/TreeInfo.java
! test/tools/javac/6840059/T6840059.out
+ test/tools/javac/generics/diamond/neg/Neg01.java
+ test/tools/javac/generics/diamond/neg/Neg01.out
+ test/tools/javac/generics/diamond/neg/Neg02.java
+ test/tools/javac/generics/diamond/neg/Neg02.out
+ test/tools/javac/generics/diamond/neg/Neg03.java
+ test/tools/javac/generics/diamond/neg/Neg03.out
+ test/tools/javac/generics/diamond/neg/Neg04.java
+ test/tools/javac/generics/diamond/neg/Neg04.out
+ test/tools/javac/generics/diamond/neg/Neg05.java
+ test/tools/javac/generics/diamond/neg/Neg05.out
+ test/tools/javac/generics/diamond/neg/Neg06.java
+ test/tools/javac/generics/diamond/neg/Neg06.out
+ test/tools/javac/generics/diamond/neg/Neg07.java
+ test/tools/javac/generics/diamond/neg/Neg07.out
+ test/tools/javac/generics/diamond/neg/Neg08.java
+ test/tools/javac/generics/diamond/neg/Neg08.out
+ test/tools/javac/generics/diamond/neg/Neg09.java
+ test/tools/javac/generics/diamond/neg/Neg09.out
+ test/tools/javac/generics/diamond/neg/Neg10.java
+ test/tools/javac/generics/diamond/neg/Neg10.out
+ test/tools/javac/generics/diamond/neg/Neg11.java
+ test/tools/javac/generics/diamond/neg/Neg11.out
+ test/tools/javac/generics/diamond/pos/Pos01.java
+ test/tools/javac/generics/diamond/pos/Pos02.java
+ test/tools/javac/generics/diamond/pos/Pos03.java
+ test/tools/javac/generics/diamond/pos/Pos04.java
+ test/tools/javac/generics/diamond/pos/Pos05.java



From lists at laerad.com  Wed Apr 14 21:42:31 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Wed, 14 Apr 2010 22:42:31 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
Message-ID: <o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>

Hi,

I hope you don't consider it rude to involve myself in this conversation
towards the end - I joined the mailing list only recently.

I'm not sure if this offers a huge amount to the discussion, but I have
tinkered with a "chunked" array list which seems to offer better time
performance in general at the cost of greater (worst case) memory
utilisation. It is easier to understand IMHO as well, although this is not
necessarily a great benefit here. It turns out the idea is very similar to
the one implemented already by Kevin, though; but perhaps simpler. The idea
is to simply double the new array size each time a new array needs to be
allocated, or in effect allocate an array that is the size of all existing
arrays put together. With this scheme the calculation for array and offset
are really very straight forward ( floor(log(i)) and 1 + i -
2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
obviously inserts at the end are much quicker.

I have prototyped the data structure this evening and benchmarked additions
at the end of the list, for which the performance is pretty impressive.

Some random statistics for addition only on the client JVM (I have quickly
dubbed my implementation ExpArrayList)
All statistics were run in two rounds with ~1000 runs per round per
statistic per list, and the second round results were used.

10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
10 items Chunked / ExpArray = 1.12
100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
100 items Chunked / ExpArray = 1.45
1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
1000 items Chunked / ExpArray = 2.02
10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
10000 items Chunked / ExpArray = 1.79
100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
100000 items Chunked / ExpArray = 1.64

and server JVM:
10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
10 items Chunked / ExpArray = 0.86
100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
100 items Chunked / ExpArray = 1.34
1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
1000 items Chunked / ExpArray = 1.27
10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
10000 items Chunked / ExpArray = 1.12
100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
100000 items Chunked / ExpArray = 1.10

Interestingly insertion at the beginning of the list appears to be quicker
with ExpArrayList, at least on the server JVM, whereas I would have expected
them to be fairly close.
Amazingly ExpArrayList is faster even than ArrayList for insertion at the
beginning of large lists, which I haven't yet tried to understand. Insertion
in the middle is similar.

10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
10 items Chunked / ExpArray = 2.59
100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
100 items Chunked / ExpArray = 2.14
1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
1000 items Chunked / ExpArray = 2.59
10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
10000 items Chunked / ExpArray = 2.16

Finally, there are promising results for get() from the ExpArrayList as well
(server JVM), again somehow beating ArrayList for larger lists:
10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
10 items get Chunked / ExpArray = 1.10
100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
100 items get Chunked / ExpArray = 1.25
1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
1000 items get Chunked / ExpArray = 1.33
10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
10000 items get Chunked / ExpArray = 1.24
100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
100000 items get Chunked / ExpArray = 1.22


I'm willing to explore this further but I'm not sure how desirable that is,
given that Kevin's data structure appears to perform pretty well already wrt
to CPU time, and better wrt to memory utilisation, and in effect this mostly
changes only the function to determine which array to use, not the body of
the implementation. Let me know if you would like a copy of the source code
and I will find somewhere to upload it.

Also, with regard to a Deque implementation, it seems that the simplest
solution would be to simply have two lists, with one accepting inserts for
near the beginning and being ordered in reverse whilst the other accepted
inserts for near to the end. The only trick would be having the list at the
beginning support iteration in reverse order cheaply, but this could easily
be achieved by creating an extension of List with a reverseIterator()
method.


Anyway, not sure if this helped at all but fancied joining in...




On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com> wrote:

> Hi Kevin,
>
> It implements List, as well as Deque. It is indeed based on ArrayDeque,
> with the added operations to implement list. It does so reasonably
> efficiently, moving the fewest elements possible on each operation, that is
> zero for the queue operations, at most n/2 for the rest and all of them for
> a backing array resize.
>
> The idea is to get a replacement for arraylist that performs like
> arraydeque on remove(0). As a side effect, we should be able to get better
> performance on other operations by requiring fewer elements to be moved.
>
> Thanks,
> Joe
>
> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>
> Hi Joe,
>>
>> I was referring to the ChunkedArrayList when I stated that add does not
>> amortize to constant time when the data structure employs the circular list
>> trick to achieve deque behavior; ChunkedArrayList potentially resizes every
>> n^(1/2) operations.
>>
>> Regarding your CircularArrayList, does it differ from Java's ArrayDeque?
>> I took only a cursory look at it, so please understand if I have missed your
>> reason for creating CircularArrayList altogether.
>>
>> Regards,
>>
>> Kevin
>>
>>
>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>> joe.j.kearney at googlemail.com> wrote:
>>
>>> Hi Kevin, Martin,
>>>
>>> To add another discussion point, I've been writing a
>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>> This works over the raw array, it doesn't use the fancier structures being
>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>> arraylets, or that provide for O(1) insert in the middle.
>>>
>>>
>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>
>>> I'd be interested if you have any comments in the context of this
>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>> at least. I'd like to add array shrinking later, when the size dips below
>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>
>>> Tests show performance to be close to ArrayList for the O(1)
>>> operations. Timings for indexed reads and writes showed
>>> no discernible difference between implementations last time I ran the
>>> tests. I don't understand at the moment why the iterator add at index
>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>> operations that become O(1) in a circular implementation (that are
>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>> the middle are somewhat faster than ArrayList because we only have to copy
>>> at most half of the elements, except when resizing the array.
>>>
>>> Kevin, I don't fully understand your point about not amortizing to O(1).
>>> Certainly that's true for insert not at head or tail. Otherwise this
>>> implementation only moves array elements to the front on an array resize
>>> operation which happens every O(ln n) operations at most, if we do lots of
>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>
>>> Some performance results below, code for these is in the repository above
>>> too. This was the second run, after a warmup.
>>>
>>> Thanks,
>>> Joe
>>>
>>> ------------------------------------------------ CircularArrayList
>>> ------------------------------------------------
>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>> insert(5)   removeRnd   removeMid   remove(0)
>>>      10          20          67          70         125         102
>>>      90         240         191         138
>>>    100          19          67          70         166         138
>>>    94         230         194         118
>>>   1000          28          64          67         681         538
>>>    91         324         382         119
>>>   10000          30          65          67        5884        4425
>>>      94        1296        2330         124
>>> ---------------------------------------------------- ArrayList
>>> ----------------------------------------------------
>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>> insert(5)   removeRnd   removeMid   remove(0)
>>>      10          23          68          70         100          69
>>>   32913         162         130         105
>>>    100          20          67          70         129         104
>>> 21944         169         134         135
>>>   1000          29          63          67         651         506
>>>  9602         364         333         526
>>>   10000          30          63          66        5878        4414
>>>    9947        2312        2280        4437
>>>
>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>
>>> Hi Martin,
>>>>
>>>> I had intended to address your request for absolute O(1) operations in
>>>> the previous email.  The approach to achieving this suggested in
>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>> a constant number of entries from the main array each time add is called.
>>>> Although this distributes the work done during a resize across the n
>>>> operations required to enter a resize-required state, it is at the expense
>>>> of additional memory usage and slower add operations.  My thought is that
>>>> this would be a fine approach for a real-time application that requires hard
>>>> guarantees on performance but would be a liability in so many Java
>>>> applications that do not require these hard guarantees.  I look forward to
>>>> hearing your thoughts on the matter, though.
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>> kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Hi Martin,
>>>>>
>>>>> It's interesting to note that the old circular list trick will not
>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>> beginning or the end; this is the approach that I will take to create a
>>>>> deque.
>>>>>
>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>> indicates that one should check with the project to determine where the SCA
>>>>> should be sent.  Do you know where I would find this information?
>>>>>
>>>>> Kevin
>>>>>
>>>>> @MISC{Brodnik99resizablearrays,
>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine
>>>>> and J. Ian Munro and Robert Sedgewick},
>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>     year = {1999}
>>>>>
>>>>> }
>>>>>
>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com>wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> Thanks for your continuing work on this.
>>>>>>
>>>>>> I like the test results, and agree with your analysis.
>>>>>> I'm especially happy that you're beating
>>>>>> ArrayList at some operations.
>>>>>>
>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>> implement both List and Deque (I regret
>>>>>> our not having done this with ArrayDeque).
>>>>>>
>>>>>> An additional property that would be nice to
>>>>>> have (but don't try too hard)
>>>>>> is to provide some kind of real-time
>>>>>> guarantees on the cost of an individual operation,
>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>> in some real-time applications.
>>>>>>
>>>>>> I will help get your changes into the obvious
>>>>>> software distributions.  I assume you're happy
>>>>>> with having this class included in any of
>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>> You should sign a Sun contributor agreement,
>>>>>> or whatever the Oracle equivalent is,
>>>>>> if you have not done so yet.
>>>>>>
>>>>>> Doug Lea likes public domain,
>>>>>> guava-libraries likes the Apache license.
>>>>>>
>>>>>> We should get various people a chance to give
>>>>>> a thumbs up on the design of this class -
>>>>>> Doug Lea, Josh Bloch.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>> > Hello Martin,
>>>>>> >
>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>> > implementation; I believe the latest version to be in decent shape.
>>>>>> I have
>>>>>> > also gathered some data on the performance of ChunkedArrayList over
>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below (note
>>>>>> that the
>>>>>> > numbers represent the time spent performing the specified operation
>>>>>> with
>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>> indicates
>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is
>>>>>> less
>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>>>> noticed
>>>>>> > relatively significant variability in a few of the numbers when I
>>>>>> switch
>>>>>> > hardware; though, these data do seem to represent rough performance
>>>>>> > expectations.  For my test I generated x elements and then timed the
>>>>>> process
>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a get
>>>>>> > operation on each for indices 0 through x-1 and finally I used the
>>>>>> iterator
>>>>>> > mechanism to retrieve the first through xth element (of course, I
>>>>>> performed
>>>>>> > each of these operations multiple times throwing away the timing for
>>>>>> the
>>>>>> > first few iterations to warm up the JVM).
>>>>>> >
>>>>>> > Regarding the question of whether or not this belongs in java.util,
>>>>>> I would
>>>>>> > suggest that if it is desirable from a GC point of view to eliminate
>>>>>> the
>>>>>> > large backing array from ArrayList then your suggestion of achieving
>>>>>> this by
>>>>>> > way of a data structure that is both time and space optimal is a
>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>> backing
>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>> dynamic
>>>>>> > shrinking behavior, has less maximum memory overhead than ArrayList,
>>>>>> and
>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>> ArrayList.  Of
>>>>>> > course, this data structure does not do everything better than
>>>>>> ArrayList; in
>>>>>> > particular, indexed access is more costly, due to the required
>>>>>> decomposition
>>>>>> > of the index into backing array index and offset and the additional
>>>>>> memory
>>>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>>>> multiple
>>>>>> > array copies necessary to complete the shift.  That being said, I
>>>>>> think that
>>>>>> > the additional cost of indexed access is partially mitigated by the
>>>>>> > availability of iterator and listIterator, whose implementations do
>>>>>> not use
>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>> ArrayList due
>>>>>> > to its linear time complexity.
>>>>>> >
>>>>>> > Kevin
>>>>>> >
>>>>>> > 1000000 elements:
>>>>>> > Client JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>> >
>>>>>> > Server JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>> >
>>>>>> > 100000 elements:
>>>>>> > Client JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>> >
>>>>>> > Server JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>> >
>>>>>> > 10000 elements:
>>>>>> > Client JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>> >
>>>>>> > Server JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>> >
>>>>>> > 1000 elements:
>>>>>> > Client JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>> >
>>>>>> > Server JVM:
>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>> martinrb at google.com> wrote:
>>>>>> >>
>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>> >> should really do this.  My idea is that we can
>>>>>> >> wholeheartedly recommend this collection class
>>>>>> >> for overall good behavior without any of the surprising
>>>>>> >> performance traps of existing collection classes.
>>>>>> >>
>>>>>> >> But for the preliminary version, it makes sense to
>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>> >> implementation.  Random access will of course
>>>>>> >> be worse than ArrayList, but by how much?
>>>>>> >> We can do some benchmarking and look for
>>>>>> >> micro-optimizations now.
>>>>>> >>
>>>>>> >> Kevin, what is you own personal feeling?
>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>> >>
>>>>>> >> Martin
>>>>>> >>
>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>> kevin.l.stern at gmail.com>
>>>>>> >> wrote:
>>>>>> >> > The data structure is available at the second link that I
>>>>>> originally
>>>>>> >> > provided (once again, it is
>>>>>> >> >
>>>>>> >> >
>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>> ).
>>>>>> >> > This does not have O(1) time insertion at the front as yet as it
>>>>>> was
>>>>>> >> > unclear
>>>>>> >> > to me whether or not it was agreed upon:
>>>>>> >> > _________________
>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>> arrays
>>>>>> >> > rather
>>>>>> >> > than the traditional single large array.
>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>> >> >
>>>>>> >> > Initially, it would be good enough to replace only
>>>>>> java.util.ArrayList
>>>>>> >> > with
>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>> add-at-front or
>>>>>> >> > other
>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>>>> important
>>>>>> >> > and
>>>>>> >> > popular collection, it's the primary "straight replacement for
>>>>>> primitive
>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>> >> > _________________
>>>>>> >> >
>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>> updating the
>>>>>> >> > document at the provided link as I find improvements.
>>>>>> >> >
>>>>>> >> > Thoughts?
>>>>>> >> >
>>>>>> >> > Thanks,
>>>>>> >> >
>>>>>> >> > Kevin
>>>>>> >> >
>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>> martinrb at google.com>
>>>>>> >> > wrote:
>>>>>> >> >>
>>>>>> >> >> Hi Kevin,
>>>>>> >> >>
>>>>>> >> >> You're probably the only one on this list who has
>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>> >> >> taking a research paper into production would
>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>> >> >> combining spec + implementation + test suite)
>>>>>> >> >>
>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>> >> >> They would probably also be interested to hear
>>>>>> >> >> about your implementation.
>>>>>> >> >>
>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>> >> >>
>>>>>> >> >>
>>>>>> >> >>
>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>> >> >>
>>>>>> >> >> Martin
>>>>>> >> >>
>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>> kevin.l.stern at gmail.com>
>>>>>> >> >> wrote:
>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The
>>>>>> code below
>>>>>> >> >> > gives
>>>>>> >> >> > me the appropriate index into the index array and the offset
>>>>>> into the
>>>>>> >> >> > data
>>>>>> >> >> > block.  That being said, remember when I mentioned that this
>>>>>> will
>>>>>> >> >> > include a
>>>>>> >> >> > bit more work to access an element than a simple bit shift and
>>>>>> a bit
>>>>>> >> >> > mask?
>>>>>> >> >> > Well this is more than a bit more - we'll be doing this each
>>>>>> time an
>>>>>> >> >> > index
>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the bits
>>>>>> to see
>>>>>> >> >> > if
>>>>>> >> >> > I
>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>> >> >> >
>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>> >> >> >             int k = lg(r);
>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>> >> >> >             int ceilKO2;
>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>> >> >> >                 p += powFloorKO2;
>>>>>> >> >> >             } else {
>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>> >> >> >             }
>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>> >> >> >
>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " " +
>>>>>> e);
>>>>>> >> >> >         }
>>>>>> >> >> >
>>>>>> >> >> > Kevin
>>>>>> >> >> >
>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>> >> >> > wrote:
>>>>>> >> >> >>
>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even numbered
>>>>>> >> >> >> superblocks,
>>>>>> >> >> >> the odd numbered superblocks need an additional term added
>>>>>> (the
>>>>>> >> >> >> number
>>>>>> >> >> >> of
>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>> anyhow, I
>>>>>> >> >> >> also
>>>>>> >> >> >> came
>>>>>> >> >> >> across an alternative characterization of superblock in the
>>>>>> paper
>>>>>> >> >> >> which
>>>>>> >> >> >> states that data blocks are grouped within a superblock when
>>>>>> they
>>>>>> >> >> >> are
>>>>>> >> >> >> the
>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>> structure
>>>>>> >> >> >> below
>>>>>> >> >> >> would be
>>>>>> >> >> >>
>>>>>> >> >> >> SB_0: [1]
>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>> >> >> >>
>>>>>> >> >> >> which seems to contradict my understanding of (1) below.  I
>>>>>> must be
>>>>>> >> >> >> reading this upside down.
>>>>>> >> >> >>
>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>> >> >> >> wrote:
>>>>>> >> >> >>>
>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal
>>>>>> time and
>>>>>> >> >> >>> space"
>>>>>> >> >> >>> the authors define their data structure with the following
>>>>>> >> >> >>> property:
>>>>>> >> >> >>>
>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists
>>>>>> of
>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>>> >> >> >>>
>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies the
>>>>>> >> >> >>> following
>>>>>> >> >> >>> structure:
>>>>>> >> >> >>>
>>>>>> >> >> >>> SB_0: [1]
>>>>>> >> >> >>> SB_1: [2]
>>>>>> >> >> >>> SB_2: [2][2]
>>>>>> >> >> >>> SB_3: [4][4]
>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>> >> >> >>> [...]
>>>>>> >> >> >>>
>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>>> >> >> >>>
>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>> >> >> >>>
>>>>>> >> >> >>> What concerns me is their statement that p represents "the
>>>>>> number
>>>>>> >> >> >>> of
>>>>>> >> >> >>> data
>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two
>>>>>> data
>>>>>> >> >> >>> blocks
>>>>>> >> >> >>> in
>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>> unless I'm
>>>>>> >> >> >>> misinterpreting it, the number of data blocks in superblocks
>>>>>> prior
>>>>>> >> >> >>> to
>>>>>> >> >> >>> SB_k
>>>>>> >> >> >>> should be:
>>>>>> >> >> >>>
>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>> >> >> >>>
>>>>>> >> >> >>> This, of course, seems to work out much better in my example
>>>>>> above,
>>>>>> >> >> >>> giving the correct answer to my interpretation of their data
>>>>>> >> >> >>> structure, but
>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>> rather than
>>>>>> >> >> >>> my
>>>>>> >> >> >>> misinterpretation.
>>>>>> >> >> >>>
>>>>>> >> >> >>> Thoughts?
>>>>>> >> >> >>>
>>>>>> >> >> >>> Kevin
>>>>>> >> >> >>>
>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>> >> >> >>> <martinrb at google.com>
>>>>>> >> >> >>> wrote:
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>> >> >> >>>> wrote:
>>>>>> >> >> >>>> > Hi Martin,
>>>>>> >> >> >>>> >
>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>>>>> comes to
>>>>>> >> >> >>>> > mind
>>>>>> >> >> >>>> > to
>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is to
>>>>>> create
>>>>>> >> >> >>>> > a
>>>>>> >> >> >>>> > cyclic
>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert at
>>>>>> the
>>>>>> >> >> >>>> > front
>>>>>> >> >> >>>> > requires
>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>>>>> insert
>>>>>> >> >> >>>> > at
>>>>>> >> >> >>>> > the
>>>>>> >> >> >>>> > rear
>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>> size).  We
>>>>>> >> >> >>>> > need
>>>>>> >> >> >>>> > to
>>>>>> >> >> >>>> > resize
>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>>>> explain
>>>>>> >> >> >>>> > more
>>>>>> >> >> >>>> > about
>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is shared
>>>>>> by the
>>>>>> >> >> >>>> > front
>>>>>> >> >> >>>> > and
>>>>>> >> >> >>>> > the rear?
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way to
>>>>>> turn
>>>>>> >> >> >>>> it
>>>>>> >> >> >>>> into
>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>> >> >> >>>> implementation,
>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>>> better
>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that
>>>>>> you
>>>>>> >> >> >>>> > reference,
>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>>> deque so
>>>>>> >> >> >>>> > if
>>>>>> >> >> >>>> > we
>>>>>> >> >> >>>> > take
>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>> >> >> >>>>
>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque operations -
>>>>>> >> >> >>>> just not efficiently.
>>>>>> >> >> >>>
>>>>>> >> >> >>
>>>>>> >> >> >
>>>>>> >> >> >
>>>>>> >> >
>>>>>> >> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100414/f0e73a47/attachment.html>

From kelly.ohair at oracle.com  Wed Apr 14 23:58:46 2010
From: kelly.ohair at oracle.com (Kelly O'Hair)
Date: Wed, 14 Apr 2010 16:58:46 -0700
Subject: Need reviewer - Change to jdk/test/makefile for shared library
	execute permissions
Message-ID: <95C9525A-2AEC-48D3-BBAE-50CCB734182A@oracle.com>

Need reviewer - Change to jdk/test/makefile for shared library execute  
permissions

We been having some spurious test failures on windows and finally  
tracked it down to this
issue. Fixing it in the jdk/test/Makefile seemed like the best place,  
even though none of us
really want shared libraries in the repositories, this seems to be the  
best fix.

This allows more tests to reliably run, although the shared libraries  
do need to exist
for the test to pass, so anyone removing these binary files should of  
course not run
the associated tests.

6943915: Adjust jdk/test/Makefile to deal with .dll and .so libraries  
needing execute permissions
http://cr.openjdk.java.net/~ohair/openjdk7/testMakefile-6943915/webrev/

I'll be testing this on all platforms tonight.

-kto



From kevin.l.stern at gmail.com  Thu Apr 15 01:44:24 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Wed, 14 Apr 2010 20:44:24 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
Message-ID: <i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>

Hi Benedict,

Like you, I am relatively new to this mailing list; I am also trying to
tread lightly so as not to step on any toes.  That being said, I think that
I can offer a response to your inquiry.

Regarding:  "The idea is to simply double the new array size each time a new
array needs to be allocated"

It seems this would not address the desire of offering an alternative to the
allocation of a large backing array for ArrayList (your largest backing
array could still reach a size of 1/2 * Integer.MAX_VALUE) and would not
address the desire of wasting the (asymptotically) minimum amount of memory
in the worst case while maintaining O(1) amortized time bounds.  The data
structure described in [Brodnik99resizablearrays] has a maximum backing
array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
over ArrayList do you see in your data structure?

Regarding:  "Also, with regard to a Deque implementation, it seems that the
simplest solution would be to simply have two lists, with one accepting
inserts for near the beginning and being ordered in reverse whilst the other
accepted inserts for near to the end."

What happens with your structure when you add n elements and then remove
element 0 n times?  I think that once you work out all the kinks you'll end
up with the two stacks approach, which is mentioned in
[Brodnik99resizablearrays] and which I mentioned in an earlier email, or
you'll end up with the circular list approach, which is not friendly to O(1)
amortized time bounds in a data structure that resizes more often than O(n)
due to the 'unshift' to the front = 0 position.  I think the best approach
is the one mentioned in [Brodnik99resizablearrays], which is the approach
that I am currently working on.  Incidentally, this approach also provides
for a much improved index unpacking procedure using only bit shifts and bit
masks, although it is at the expense of (O(1)) additional work during
resize.

Regards,

Kevin


On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <lists at laerad.com>wrote:

> Hi,
>
> I hope you don't consider it rude to involve myself in this conversation
> towards the end - I joined the mailing list only recently.
>
> I'm not sure if this offers a huge amount to the discussion, but I have
> tinkered with a "chunked" array list which seems to offer better time
> performance in general at the cost of greater (worst case) memory
> utilisation. It is easier to understand IMHO as well, although this is not
> necessarily a great benefit here. It turns out the idea is very similar to
> the one implemented already by Kevin, though; but perhaps simpler. The idea
> is to simply double the new array size each time a new array needs to be
> allocated, or in effect allocate an array that is the size of all existing
> arrays put together. With this scheme the calculation for array and offset
> are really very straight forward ( floor(log(i)) and 1 + i -
> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
> obviously inserts at the end are much quicker.
>
> I have prototyped the data structure this evening and benchmarked additions
> at the end of the list, for which the performance is pretty impressive.
>
> Some random statistics for addition only on the client JVM (I have quickly
> dubbed my implementation ExpArrayList)
> All statistics were run in two rounds with ~1000 runs per round per
> statistic per list, and the second round results were used.
>
> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
> 10 items Chunked / ExpArray = 1.12
> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
> 100 items Chunked / ExpArray = 1.45
> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
> 1000 items Chunked / ExpArray = 2.02
> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
> 10000 items Chunked / ExpArray = 1.79
> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
> 100000 items Chunked / ExpArray = 1.64
>
> and server JVM:
> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
> 10 items Chunked / ExpArray = 0.86
> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
> 100 items Chunked / ExpArray = 1.34
> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
> 1000 items Chunked / ExpArray = 1.27
> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
> 10000 items Chunked / ExpArray = 1.12
> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
> 100000 items Chunked / ExpArray = 1.10
>
> Interestingly insertion at the beginning of the list appears to be quicker
> with ExpArrayList, at least on the server JVM, whereas I would have expected
> them to be fairly close.
> Amazingly ExpArrayList is faster even than ArrayList for insertion at the
> beginning of large lists, which I haven't yet tried to understand. Insertion
> in the middle is similar.
>
> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
> 10 items Chunked / ExpArray = 2.59
> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
> 100 items Chunked / ExpArray = 2.14
> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
> 1000 items Chunked / ExpArray = 2.59
> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
> 10000 items Chunked / ExpArray = 2.16
>
> Finally, there are promising results for get() from the ExpArrayList as
> well (server JVM), again somehow beating ArrayList for larger lists:
> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
> 10 items get Chunked / ExpArray = 1.10
> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
> 100 items get Chunked / ExpArray = 1.25
> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
> 1000 items get Chunked / ExpArray = 1.33
> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
> 10000 items get Chunked / ExpArray = 1.24
> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
> 100000 items get Chunked / ExpArray = 1.22
>
>
> I'm willing to explore this further but I'm not sure how desirable that is,
> given that Kevin's data structure appears to perform pretty well already wrt
> to CPU time, and better wrt to memory utilisation, and in effect this mostly
> changes only the function to determine which array to use, not the body of
> the implementation. Let me know if you would like a copy of the source code
> and I will find somewhere to upload it.
>
> Also, with regard to a Deque implementation, it seems that the simplest
> solution would be to simply have two lists, with one accepting inserts for
> near the beginning and being ordered in reverse whilst the other accepted
> inserts for near to the end. The only trick would be having the list at the
> beginning support iteration in reverse order cheaply, but this could easily
> be achieved by creating an extension of List with a reverseIterator()
> method.
>
>
> Anyway, not sure if this helped at all but fancied joining in...
>
>
>
>
> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com> wrote:
>
>> Hi Kevin,
>>
>> It implements List, as well as Deque. It is indeed based on ArrayDeque,
>> with the added operations to implement list. It does so reasonably
>> efficiently, moving the fewest elements possible on each operation, that is
>> zero for the queue operations, at most n/2 for the rest and all of them for
>> a backing array resize.
>>
>> The idea is to get a replacement for arraylist that performs like
>> arraydeque on remove(0). As a side effect, we should be able to get better
>> performance on other operations by requiring fewer elements to be moved.
>>
>> Thanks,
>> Joe
>>
>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>
>> Hi Joe,
>>>
>>> I was referring to the ChunkedArrayList when I stated that add does not
>>> amortize to constant time when the data structure employs the circular list
>>> trick to achieve deque behavior; ChunkedArrayList potentially resizes every
>>> n^(1/2) operations.
>>>
>>> Regarding your CircularArrayList, does it differ from Java's ArrayDeque?
>>> I took only a cursory look at it, so please understand if I have missed your
>>> reason for creating CircularArrayList altogether.
>>>
>>> Regards,
>>>
>>> Kevin
>>>
>>>
>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>> joe.j.kearney at googlemail.com> wrote:
>>>
>>>> Hi Kevin, Martin,
>>>>
>>>> To add another discussion point, I've been writing a
>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>> This works over the raw array, it doesn't use the fancier structures being
>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>
>>>>
>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>
>>>> I'd be interested if you have any comments in the context of this
>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>
>>>> Tests show performance to be close to ArrayList for the O(1)
>>>> operations. Timings for indexed reads and writes showed
>>>> no discernible difference between implementations last time I ran the
>>>> tests. I don't understand at the moment why the iterator add at index
>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>> operations that become O(1) in a circular implementation (that are
>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>> at most half of the elements, except when resizing the array.
>>>>
>>>> Kevin, I don't fully understand your point about not amortizing to O(1).
>>>> Certainly that's true for insert not at head or tail. Otherwise this
>>>> implementation only moves array elements to the front on an array resize
>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>
>>>> Some performance results below, code for these is in the repository
>>>> above too. This was the second run, after a warmup.
>>>>
>>>> Thanks,
>>>> Joe
>>>>
>>>> ------------------------------------------------ CircularArrayList
>>>> ------------------------------------------------
>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>      10          20          67          70         125         102
>>>>      90         240         191         138
>>>>    100          19          67          70         166         138
>>>>    94         230         194         118
>>>>   1000          28          64          67         681         538
>>>>    91         324         382         119
>>>>   10000          30          65          67        5884        4425
>>>>      94        1296        2330         124
>>>> ---------------------------------------------------- ArrayList
>>>> ----------------------------------------------------
>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>      10          23          68          70         100          69
>>>>   32913         162         130         105
>>>>    100          20          67          70         129         104
>>>> 21944         169         134         135
>>>>   1000          29          63          67         651         506
>>>>  9602         364         333         526
>>>>   10000          30          63          66        5878        4414
>>>>    9947        2312        2280        4437
>>>>
>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>
>>>> Hi Martin,
>>>>>
>>>>> I had intended to address your request for absolute O(1) operations in
>>>>> the previous email.  The approach to achieving this suggested in
>>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>> a constant number of entries from the main array each time add is called.
>>>>> Although this distributes the work done during a resize across the n
>>>>> operations required to enter a resize-required state, it is at the expense
>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>> this would be a fine approach for a real-time application that requires hard
>>>>> guarantees on performance but would be a liability in so many Java
>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>> hearing your thoughts on the matter, though.
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> It's interesting to note that the old circular list trick will not
>>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>>> beginning or the end; this is the approach that I will take to create a
>>>>>> deque.
>>>>>>
>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine
>>>>>> and J. Ian Munro and Robert Sedgewick},
>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>     year = {1999}
>>>>>>
>>>>>> }
>>>>>>
>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <martinrb at google.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> Thanks for your continuing work on this.
>>>>>>>
>>>>>>> I like the test results, and agree with your analysis.
>>>>>>> I'm especially happy that you're beating
>>>>>>> ArrayList at some operations.
>>>>>>>
>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>> implement both List and Deque (I regret
>>>>>>> our not having done this with ArrayDeque).
>>>>>>>
>>>>>>> An additional property that would be nice to
>>>>>>> have (but don't try too hard)
>>>>>>> is to provide some kind of real-time
>>>>>>> guarantees on the cost of an individual operation,
>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>> in some real-time applications.
>>>>>>>
>>>>>>> I will help get your changes into the obvious
>>>>>>> software distributions.  I assume you're happy
>>>>>>> with having this class included in any of
>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>> You should sign a Sun contributor agreement,
>>>>>>> or whatever the Oracle equivalent is,
>>>>>>> if you have not done so yet.
>>>>>>>
>>>>>>> Doug Lea likes public domain,
>>>>>>> guava-libraries likes the Apache license.
>>>>>>>
>>>>>>> We should get various people a chance to give
>>>>>>> a thumbs up on the design of this class -
>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>> > Hello Martin,
>>>>>>> >
>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>> shape.  I have
>>>>>>> > also gathered some data on the performance of ChunkedArrayList over
>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below (note
>>>>>>> that the
>>>>>>> > numbers represent the time spent performing the specified operation
>>>>>>> with
>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>> indicates
>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is
>>>>>>> less
>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>>>>> noticed
>>>>>>> > relatively significant variability in a few of the numbers when I
>>>>>>> switch
>>>>>>> > hardware; though, these data do seem to represent rough performance
>>>>>>> > expectations.  For my test I generated x elements and then timed
>>>>>>> the process
>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a
>>>>>>> get
>>>>>>> > operation on each for indices 0 through x-1 and finally I used the
>>>>>>> iterator
>>>>>>> > mechanism to retrieve the first through xth element (of course, I
>>>>>>> performed
>>>>>>> > each of these operations multiple times throwing away the timing
>>>>>>> for the
>>>>>>> > first few iterations to warm up the JVM).
>>>>>>> >
>>>>>>> > Regarding the question of whether or not this belongs in java.util,
>>>>>>> I would
>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>> eliminate the
>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>> achieving this by
>>>>>>> > way of a data structure that is both time and space optimal is a
>>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>>> backing
>>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>>> dynamic
>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>> ArrayList, and
>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>> ArrayList.  Of
>>>>>>> > course, this data structure does not do everything better than
>>>>>>> ArrayList; in
>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>> decomposition
>>>>>>> > of the index into backing array index and offset and the additional
>>>>>>> memory
>>>>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>>>>> multiple
>>>>>>> > array copies necessary to complete the shift.  That being said, I
>>>>>>> think that
>>>>>>> > the additional cost of indexed access is partially mitigated by the
>>>>>>> > availability of iterator and listIterator, whose implementations do
>>>>>>> not use
>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>> ArrayList due
>>>>>>> > to its linear time complexity.
>>>>>>> >
>>>>>>> > Kevin
>>>>>>> >
>>>>>>> > 1000000 elements:
>>>>>>> > Client JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>> >
>>>>>>> > Server JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>> >
>>>>>>> > 100000 elements:
>>>>>>> > Client JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>> >
>>>>>>> > Server JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>> >
>>>>>>> > 10000 elements:
>>>>>>> > Client JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>> >
>>>>>>> > Server JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>> >
>>>>>>> > 1000 elements:
>>>>>>> > Client JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>> >
>>>>>>> > Server JVM:
>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>> martinrb at google.com> wrote:
>>>>>>> >>
>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>> >> should really do this.  My idea is that we can
>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>> >> performance traps of existing collection classes.
>>>>>>> >>
>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>> >> implementation.  Random access will of course
>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>> >> We can do some benchmarking and look for
>>>>>>> >> micro-optimizations now.
>>>>>>> >>
>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>> >>
>>>>>>> >> Martin
>>>>>>> >>
>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>> kevin.l.stern at gmail.com>
>>>>>>> >> wrote:
>>>>>>> >> > The data structure is available at the second link that I
>>>>>>> originally
>>>>>>> >> > provided (once again, it is
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>> ).
>>>>>>> >> > This does not have O(1) time insertion at the front as yet as it
>>>>>>> was
>>>>>>> >> > unclear
>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>> >> > _________________
>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>> arrays
>>>>>>> >> > rather
>>>>>>> >> > than the traditional single large array.
>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>> >> >
>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>> java.util.ArrayList
>>>>>>> >> > with
>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>> add-at-front or
>>>>>>> >> > other
>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>>>>> important
>>>>>>> >> > and
>>>>>>> >> > popular collection, it's the primary "straight replacement for
>>>>>>> primitive
>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>> >> > _________________
>>>>>>> >> >
>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>> updating the
>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>> >> >
>>>>>>> >> > Thoughts?
>>>>>>> >> >
>>>>>>> >> > Thanks,
>>>>>>> >> >
>>>>>>> >> > Kevin
>>>>>>> >> >
>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>> martinrb at google.com>
>>>>>>> >> > wrote:
>>>>>>> >> >>
>>>>>>> >> >> Hi Kevin,
>>>>>>> >> >>
>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>> >> >> taking a research paper into production would
>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>> >> >>
>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>> >> >> They would probably also be interested to hear
>>>>>>> >> >> about your implementation.
>>>>>>> >> >>
>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> >> >>
>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>> >> >>
>>>>>>> >> >> Martin
>>>>>>> >> >>
>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>> kevin.l.stern at gmail.com>
>>>>>>> >> >> wrote:
>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The
>>>>>>> code below
>>>>>>> >> >> > gives
>>>>>>> >> >> > me the appropriate index into the index array and the offset
>>>>>>> into the
>>>>>>> >> >> > data
>>>>>>> >> >> > block.  That being said, remember when I mentioned that this
>>>>>>> will
>>>>>>> >> >> > include a
>>>>>>> >> >> > bit more work to access an element than a simple bit shift
>>>>>>> and a bit
>>>>>>> >> >> > mask?
>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this each
>>>>>>> time an
>>>>>>> >> >> > index
>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the
>>>>>>> bits to see
>>>>>>> >> >> > if
>>>>>>> >> >> > I
>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>> >> >> >
>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>> >> >> >             int k = lg(r);
>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>> >> >> >             int ceilKO2;
>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>> >> >> >             } else {
>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>> >> >> >             }
>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>> >> >> >
>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " "
>>>>>>> + e);
>>>>>>> >> >> >         }
>>>>>>> >> >> >
>>>>>>> >> >> > Kevin
>>>>>>> >> >> >
>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>> >> >> > wrote:
>>>>>>> >> >> >>
>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>> numbered
>>>>>>> >> >> >> superblocks,
>>>>>>> >> >> >> the odd numbered superblocks need an additional term added
>>>>>>> (the
>>>>>>> >> >> >> number
>>>>>>> >> >> >> of
>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>>> anyhow, I
>>>>>>> >> >> >> also
>>>>>>> >> >> >> came
>>>>>>> >> >> >> across an alternative characterization of superblock in the
>>>>>>> paper
>>>>>>> >> >> >> which
>>>>>>> >> >> >> states that data blocks are grouped within a superblock when
>>>>>>> they
>>>>>>> >> >> >> are
>>>>>>> >> >> >> the
>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>> structure
>>>>>>> >> >> >> below
>>>>>>> >> >> >> would be
>>>>>>> >> >> >>
>>>>>>> >> >> >> SB_0: [1]
>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>> >> >> >>
>>>>>>> >> >> >> which seems to contradict my understanding of (1) below.  I
>>>>>>> must be
>>>>>>> >> >> >> reading this upside down.
>>>>>>> >> >> >>
>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>> >> >> >> wrote:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal
>>>>>>> time and
>>>>>>> >> >> >>> space"
>>>>>>> >> >> >>> the authors define their data structure with the following
>>>>>>> >> >> >>> property:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists
>>>>>>> of
>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies the
>>>>>>> >> >> >>> following
>>>>>>> >> >> >>> structure:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> SB_0: [1]
>>>>>>> >> >> >>> SB_1: [2]
>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>> >> >> >>> [...]
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> What concerns me is their statement that p represents "the
>>>>>>> number
>>>>>>> >> >> >>> of
>>>>>>> >> >> >>> data
>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two
>>>>>>> data
>>>>>>> >> >> >>> blocks
>>>>>>> >> >> >>> in
>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>>> unless I'm
>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>> superblocks prior
>>>>>>> >> >> >>> to
>>>>>>> >> >> >>> SB_k
>>>>>>> >> >> >>> should be:
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>> example above,
>>>>>>> >> >> >>> giving the correct answer to my interpretation of their
>>>>>>> data
>>>>>>> >> >> >>> structure, but
>>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>>> rather than
>>>>>>> >> >> >>> my
>>>>>>> >> >> >>> misinterpretation.
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> Thoughts?
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> Kevin
>>>>>>> >> >> >>>
>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>> >> >> >>> wrote:
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>> >> >> >>>> wrote:
>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>> >> >> >>>> >
>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>>>>>> comes to
>>>>>>> >> >> >>>> > mind
>>>>>>> >> >> >>>> > to
>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is
>>>>>>> to create
>>>>>>> >> >> >>>> > a
>>>>>>> >> >> >>>> > cyclic
>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert at
>>>>>>> the
>>>>>>> >> >> >>>> > front
>>>>>>> >> >> >>>> > requires
>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>>>>>> insert
>>>>>>> >> >> >>>> > at
>>>>>>> >> >> >>>> > the
>>>>>>> >> >> >>>> > rear
>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>> size).  We
>>>>>>> >> >> >>>> > need
>>>>>>> >> >> >>>> > to
>>>>>>> >> >> >>>> > resize
>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>>>>> explain
>>>>>>> >> >> >>>> > more
>>>>>>> >> >> >>>> > about
>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>> shared by the
>>>>>>> >> >> >>>> > front
>>>>>>> >> >> >>>> > and
>>>>>>> >> >> >>>> > the rear?
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way
>>>>>>> to turn
>>>>>>> >> >> >>>> it
>>>>>>> >> >> >>>> into
>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>> >> >> >>>> implementation,
>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>>>> better
>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that
>>>>>>> you
>>>>>>> >> >> >>>> > reference,
>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>>>> deque so
>>>>>>> >> >> >>>> > if
>>>>>>> >> >> >>>> > we
>>>>>>> >> >> >>>> > take
>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>> >> >> >>>>
>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque operations
>>>>>>> -
>>>>>>> >> >> >>>> just not efficiently.
>>>>>>> >> >> >>>
>>>>>>> >> >> >>
>>>>>>> >> >> >
>>>>>>> >> >> >
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100414/3a1c64c1/attachment.html>

From lists at laerad.com  Thu Apr 15 09:07:03 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Thu, 15 Apr 2010 10:07:03 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
Message-ID: <p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>

Hi Kevin,

Yes, as I was going to bed last night I realised I had not fully addressed
the problem that was originally being visited; only reduced the constant
factor for addition to the end of the list. A trivial modification fixes
that, however; same scheme but up to some maximum arraylet size (of a power
of 2), after which the array is increased in size linearly. Performance
doesn't seem to have been affected appreciably, although not been exhaustive
in the benchmarking:

10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
10 items inserts Chunked / ExpArray = 0.99
10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
10 items get Chunked / ExpArray = 0.99
100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
100 items inserts Chunked / ExpArray = 1.23
100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
100 items get Chunked / ExpArray = 1.23
1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
1000 items inserts Chunked / ExpArray = 1.19
1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
1000 items get Chunked / ExpArray = 1.19
10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
10000 items inserts Chunked / ExpArray = 1.18
10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
10000 items get Chunked / ExpArray = 1.18
100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
100000 items inserts Chunked / ExpArray = 1.09
100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
100000 items get Chunked / ExpArray = 1.09

The nice thing about this is that the maximum amount of wasted memory is
user configurable. Even with a low setting as above (65K) performance seems
pretty consistent.

Code for calculating index and array offset are pretty straight forward;
haven't given much thought to optimisations just yet:

private final int indexFor(int a, int i) {
return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
maxArraySizeShift : 1 << a) ;
}
private final int arrayFor(int i) {
return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
}

Regarding the double list idea - yes, I agree, I certainly didn't think that
one through fully!



On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> Like you, I am relatively new to this mailing list; I am also trying to
> tread lightly so as not to step on any toes.  That being said, I think that
> I can offer a response to your inquiry.
>
> Regarding:  "The idea is to simply double the new array size each time a
> new array needs to be allocated"
>
> It seems this would not address the desire of offering an alternative to
> the allocation of a large backing array for ArrayList (your largest backing
> array could still reach a size of 1/2 * Integer.MAX_VALUE) and would not
> address the desire of wasting the (asymptotically) minimum amount of memory
> in the worst case while maintaining O(1) amortized time bounds.  The data
> structure described in [Brodnik99resizablearrays] has a maximum backing
> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
> over ArrayList do you see in your data structure?
>
> Regarding:  "Also, with regard to a Deque implementation, it seems that the
> simplest solution would be to simply have two lists, with one accepting
> inserts for near the beginning and being ordered in reverse whilst the other
> accepted inserts for near to the end."
>
> What happens with your structure when you add n elements and then remove
> element 0 n times?  I think that once you work out all the kinks you'll end
> up with the two stacks approach, which is mentioned in
> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
> you'll end up with the circular list approach, which is not friendly to O(1)
> amortized time bounds in a data structure that resizes more often than O(n)
> due to the 'unshift' to the front = 0 position.  I think the best approach
> is the one mentioned in [Brodnik99resizablearrays], which is the approach
> that I am currently working on.  Incidentally, this approach also provides
> for a much improved index unpacking procedure using only bit shifts and bit
> masks, although it is at the expense of (O(1)) additional work during
> resize.
>
> Regards,
>
> Kevin
>
>
>
> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <lists at laerad.com>wrote:
>
>> Hi,
>>
>> I hope you don't consider it rude to involve myself in this conversation
>> towards the end - I joined the mailing list only recently.
>>
>> I'm not sure if this offers a huge amount to the discussion, but I have
>> tinkered with a "chunked" array list which seems to offer better time
>> performance in general at the cost of greater (worst case) memory
>> utilisation. It is easier to understand IMHO as well, although this is not
>> necessarily a great benefit here. It turns out the idea is very similar to
>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>> is to simply double the new array size each time a new array needs to be
>> allocated, or in effect allocate an array that is the size of all existing
>> arrays put together. With this scheme the calculation for array and offset
>> are really very straight forward ( floor(log(i)) and 1 + i -
>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>> obviously inserts at the end are much quicker.
>>
>> I have prototyped the data structure this evening and benchmarked
>> additions at the end of the list, for which the performance is pretty
>> impressive.
>>
>> Some random statistics for addition only on the client JVM (I have quickly
>> dubbed my implementation ExpArrayList)
>> All statistics were run in two rounds with ~1000 runs per round per
>> statistic per list, and the second round results were used.
>>
>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>> 10 items Chunked / ExpArray = 1.12
>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>> 100 items Chunked / ExpArray = 1.45
>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>> 1000 items Chunked / ExpArray = 2.02
>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>> 10000 items Chunked / ExpArray = 1.79
>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>> 100000 items Chunked / ExpArray = 1.64
>>
>> and server JVM:
>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>> 10 items Chunked / ExpArray = 0.86
>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>> 100 items Chunked / ExpArray = 1.34
>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>> 1000 items Chunked / ExpArray = 1.27
>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>> 10000 items Chunked / ExpArray = 1.12
>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>> 100000 items Chunked / ExpArray = 1.10
>>
>> Interestingly insertion at the beginning of the list appears to be quicker
>> with ExpArrayList, at least on the server JVM, whereas I would have expected
>> them to be fairly close.
>> Amazingly ExpArrayList is faster even than ArrayList for insertion at the
>> beginning of large lists, which I haven't yet tried to understand. Insertion
>> in the middle is similar.
>>
>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>> 10 items Chunked / ExpArray = 2.59
>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>> 100 items Chunked / ExpArray = 2.14
>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>> 1000 items Chunked / ExpArray = 2.59
>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>> 10000 items Chunked / ExpArray = 2.16
>>
>> Finally, there are promising results for get() from the ExpArrayList as
>> well (server JVM), again somehow beating ArrayList for larger lists:
>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>> 10 items get Chunked / ExpArray = 1.10
>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>> 100 items get Chunked / ExpArray = 1.25
>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>> 1000 items get Chunked / ExpArray = 1.33
>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>> 10000 items get Chunked / ExpArray = 1.24
>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>> 100000 items get Chunked / ExpArray = 1.22
>>
>>
>> I'm willing to explore this further but I'm not sure how desirable that
>> is, given that Kevin's data structure appears to perform pretty well already
>> wrt to CPU time, and better wrt to memory utilisation, and in effect this
>> mostly changes only the function to determine which array to use, not the
>> body of the implementation. Let me know if you would like a copy of the
>> source code and I will find somewhere to upload it.
>>
>> Also, with regard to a Deque implementation, it seems that the simplest
>> solution would be to simply have two lists, with one accepting inserts for
>> near the beginning and being ordered in reverse whilst the other accepted
>> inserts for near to the end. The only trick would be having the list at the
>> beginning support iteration in reverse order cheaply, but this could easily
>> be achieved by creating an extension of List with a reverseIterator()
>> method.
>>
>>
>> Anyway, not sure if this helped at all but fancied joining in...
>>
>>
>>
>>
>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> It implements List, as well as Deque. It is indeed based on ArrayDeque,
>>> with the added operations to implement list. It does so reasonably
>>> efficiently, moving the fewest elements possible on each operation, that is
>>> zero for the queue operations, at most n/2 for the rest and all of them for
>>> a backing array resize.
>>>
>>> The idea is to get a replacement for arraylist that performs like
>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>> performance on other operations by requiring fewer elements to be moved.
>>>
>>> Thanks,
>>> Joe
>>>
>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>
>>> Hi Joe,
>>>>
>>>> I was referring to the ChunkedArrayList when I stated that add does not
>>>> amortize to constant time when the data structure employs the circular list
>>>> trick to achieve deque behavior; ChunkedArrayList potentially resizes every
>>>> n^(1/2) operations.
>>>>
>>>> Regarding your CircularArrayList, does it differ from Java's
>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>> have missed your reason for creating CircularArrayList altogether.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>> joe.j.kearney at googlemail.com> wrote:
>>>>
>>>>> Hi Kevin, Martin,
>>>>>
>>>>> To add another discussion point, I've been writing a
>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>
>>>>>
>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>
>>>>> I'd be interested if you have any comments in the context of this
>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>
>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>> operations. Timings for indexed reads and writes showed
>>>>> no discernible difference between implementations last time I ran the
>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>> operations that become O(1) in a circular implementation (that are
>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>> at most half of the elements, except when resizing the array.
>>>>>
>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>> implementation only moves array elements to the front on an array resize
>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>
>>>>> Some performance results below, code for these is in the repository
>>>>> above too. This was the second run, after a warmup.
>>>>>
>>>>> Thanks,
>>>>> Joe
>>>>>
>>>>> ------------------------------------------------ CircularArrayList
>>>>> ------------------------------------------------
>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>      10          20          67          70         125         102
>>>>>        90         240         191         138
>>>>>    100          19          67          70         166         138
>>>>>      94         230         194         118
>>>>>   1000          28          64          67         681         538
>>>>>      91         324         382         119
>>>>>   10000          30          65          67        5884        4425
>>>>>        94        1296        2330         124
>>>>> ---------------------------------------------------- ArrayList
>>>>> ----------------------------------------------------
>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>      10          23          68          70         100          69
>>>>>     32913         162         130         105
>>>>>    100          20          67          70         129         104
>>>>>   21944         169         134         135
>>>>>   1000          29          63          67         651         506
>>>>>    9602         364         333         526
>>>>>   10000          30          63          66        5878        4414
>>>>>      9947        2312        2280        4437
>>>>>
>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>
>>>>> Hi Martin,
>>>>>>
>>>>>> I had intended to address your request for absolute O(1) operations in
>>>>>> the previous email.  The approach to achieving this suggested in
>>>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>> a constant number of entries from the main array each time add is called.
>>>>>> Although this distributes the work done during a resize across the n
>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>> hearing your thoughts on the matter, though.
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> It's interesting to note that the old circular list trick will not
>>>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>>>> beginning or the end; this is the approach that I will take to create a
>>>>>>> deque.
>>>>>>>
>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine
>>>>>>> and J. Ian Munro and Robert Sedgewick},
>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>     year = {1999}
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>> martinrb at google.com> wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>
>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>> I'm especially happy that you're beating
>>>>>>>> ArrayList at some operations.
>>>>>>>>
>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>> implement both List and Deque (I regret
>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>
>>>>>>>> An additional property that would be nice to
>>>>>>>> have (but don't try too hard)
>>>>>>>> is to provide some kind of real-time
>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>> in some real-time applications.
>>>>>>>>
>>>>>>>> I will help get your changes into the obvious
>>>>>>>> software distributions.  I assume you're happy
>>>>>>>> with having this class included in any of
>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>> if you have not done so yet.
>>>>>>>>
>>>>>>>> Doug Lea likes public domain,
>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>
>>>>>>>> We should get various people a chance to give
>>>>>>>> a thumbs up on the design of this class -
>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>> > Hello Martin,
>>>>>>>> >
>>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>> shape.  I have
>>>>>>>> > also gathered some data on the performance of ChunkedArrayList
>>>>>>>> over
>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>> (note that the
>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>> operation with
>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>> indicates
>>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is
>>>>>>>> less
>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>>>>>> noticed
>>>>>>>> > relatively significant variability in a few of the numbers when I
>>>>>>>> switch
>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>> performance
>>>>>>>> > expectations.  For my test I generated x elements and then timed
>>>>>>>> the process
>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a
>>>>>>>> get
>>>>>>>> > operation on each for indices 0 through x-1 and finally I used the
>>>>>>>> iterator
>>>>>>>> > mechanism to retrieve the first through xth element (of course, I
>>>>>>>> performed
>>>>>>>> > each of these operations multiple times throwing away the timing
>>>>>>>> for the
>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>> >
>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>> java.util, I would
>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>> eliminate the
>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>> achieving this by
>>>>>>>> > way of a data structure that is both time and space optimal is a
>>>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>>>> backing
>>>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>>>> dynamic
>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>> ArrayList, and
>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>> ArrayList.  Of
>>>>>>>> > course, this data structure does not do everything better than
>>>>>>>> ArrayList; in
>>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>>> decomposition
>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>> additional memory
>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>>>>>> multiple
>>>>>>>> > array copies necessary to complete the shift.  That being said, I
>>>>>>>> think that
>>>>>>>> > the additional cost of indexed access is partially mitigated by
>>>>>>>> the
>>>>>>>> > availability of iterator and listIterator, whose implementations
>>>>>>>> do not use
>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>> ArrayList due
>>>>>>>> > to its linear time complexity.
>>>>>>>> >
>>>>>>>> > Kevin
>>>>>>>> >
>>>>>>>> > 1000000 elements:
>>>>>>>> > Client JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>> >
>>>>>>>> > Server JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>> >
>>>>>>>> > 100000 elements:
>>>>>>>> > Client JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>> >
>>>>>>>> > Server JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>> >
>>>>>>>> > 10000 elements:
>>>>>>>> > Client JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>> >
>>>>>>>> > Server JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>> >
>>>>>>>> > 1000 elements:
>>>>>>>> > Client JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>> >
>>>>>>>> > Server JVM:
>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>> martinrb at google.com> wrote:
>>>>>>>> >>
>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>> >>
>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>> >> implementation.  Random access will of course
>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>> >> micro-optimizations now.
>>>>>>>> >>
>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>> >>
>>>>>>>> >> Martin
>>>>>>>> >>
>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>> >> wrote:
>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>> originally
>>>>>>>> >> > provided (once again, it is
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>> ).
>>>>>>>> >> > This does not have O(1) time insertion at the front as yet as
>>>>>>>> it was
>>>>>>>> >> > unclear
>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>> >> > _________________
>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>>> arrays
>>>>>>>> >> > rather
>>>>>>>> >> > than the traditional single large array.
>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>> >> >
>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>> java.util.ArrayList
>>>>>>>> >> > with
>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>> add-at-front or
>>>>>>>> >> > other
>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>>>>>> important
>>>>>>>> >> > and
>>>>>>>> >> > popular collection, it's the primary "straight replacement for
>>>>>>>> primitive
>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>> >> > _________________
>>>>>>>> >> >
>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>> updating the
>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>> >> >
>>>>>>>> >> > Thoughts?
>>>>>>>> >> >
>>>>>>>> >> > Thanks,
>>>>>>>> >> >
>>>>>>>> >> > Kevin
>>>>>>>> >> >
>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>> martinrb at google.com>
>>>>>>>> >> > wrote:
>>>>>>>> >> >>
>>>>>>>> >> >> Hi Kevin,
>>>>>>>> >> >>
>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>> >> >> taking a research paper into production would
>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>> >> >>
>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>> >> >> about your implementation.
>>>>>>>> >> >>
>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> >> >>
>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>> >> >>
>>>>>>>> >> >> Martin
>>>>>>>> >> >>
>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>> >> >> wrote:
>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The
>>>>>>>> code below
>>>>>>>> >> >> > gives
>>>>>>>> >> >> > me the appropriate index into the index array and the offset
>>>>>>>> into the
>>>>>>>> >> >> > data
>>>>>>>> >> >> > block.  That being said, remember when I mentioned that this
>>>>>>>> will
>>>>>>>> >> >> > include a
>>>>>>>> >> >> > bit more work to access an element than a simple bit shift
>>>>>>>> and a bit
>>>>>>>> >> >> > mask?
>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this each
>>>>>>>> time an
>>>>>>>> >> >> > index
>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the
>>>>>>>> bits to see
>>>>>>>> >> >> > if
>>>>>>>> >> >> > I
>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>> >> >> >
>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>> >> >> >             } else {
>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>> >> >> >             }
>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>> >> >> >
>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + " "
>>>>>>>> + e);
>>>>>>>> >> >> >         }
>>>>>>>> >> >> >
>>>>>>>> >> >> > Kevin
>>>>>>>> >> >> >
>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>> >> >> > wrote:
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>> numbered
>>>>>>>> >> >> >> superblocks,
>>>>>>>> >> >> >> the odd numbered superblocks need an additional term added
>>>>>>>> (the
>>>>>>>> >> >> >> number
>>>>>>>> >> >> >> of
>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>>>> anyhow, I
>>>>>>>> >> >> >> also
>>>>>>>> >> >> >> came
>>>>>>>> >> >> >> across an alternative characterization of superblock in the
>>>>>>>> paper
>>>>>>>> >> >> >> which
>>>>>>>> >> >> >> states that data blocks are grouped within a superblock
>>>>>>>> when they
>>>>>>>> >> >> >> are
>>>>>>>> >> >> >> the
>>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>>> structure
>>>>>>>> >> >> >> below
>>>>>>>> >> >> >> would be
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> which seems to contradict my understanding of (1) below.  I
>>>>>>>> must be
>>>>>>>> >> >> >> reading this upside down.
>>>>>>>> >> >> >>
>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>> >> >> >> wrote:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal
>>>>>>>> time and
>>>>>>>> >> >> >>> space"
>>>>>>>> >> >> >>> the authors define their data structure with the following
>>>>>>>> >> >> >>> property:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it consists
>>>>>>>> of
>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies
>>>>>>>> the
>>>>>>>> >> >> >>> following
>>>>>>>> >> >> >>> structure:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>> >> >> >>> [...]
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> What concerns me is their statement that p represents "the
>>>>>>>> number
>>>>>>>> >> >> >>> of
>>>>>>>> >> >> >>> data
>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two
>>>>>>>> data
>>>>>>>> >> >> >>> blocks
>>>>>>>> >> >> >>> in
>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>>>> unless I'm
>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>> superblocks prior
>>>>>>>> >> >> >>> to
>>>>>>>> >> >> >>> SB_k
>>>>>>>> >> >> >>> should be:
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>> example above,
>>>>>>>> >> >> >>> giving the correct answer to my interpretation of their
>>>>>>>> data
>>>>>>>> >> >> >>> structure, but
>>>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>>>> rather than
>>>>>>>> >> >> >>> my
>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> Thoughts?
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> Kevin
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>> >> >> >>> wrote:
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>> >> >> >>>> wrote:
>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>> >> >> >>>> >
>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach that
>>>>>>>> comes to
>>>>>>>> >> >> >>>> > mind
>>>>>>>> >> >> >>>> > to
>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is
>>>>>>>> to create
>>>>>>>> >> >> >>>> > a
>>>>>>>> >> >> >>>> > cyclic
>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert at
>>>>>>>> the
>>>>>>>> >> >> >>>> > front
>>>>>>>> >> >> >>>> > requires
>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and to
>>>>>>>> insert
>>>>>>>> >> >> >>>> > at
>>>>>>>> >> >> >>>> > the
>>>>>>>> >> >> >>>> > rear
>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>> size).  We
>>>>>>>> >> >> >>>> > need
>>>>>>>> >> >> >>>> > to
>>>>>>>> >> >> >>>> > resize
>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>>>>>> explain
>>>>>>>> >> >> >>>> > more
>>>>>>>> >> >> >>>> > about
>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>> shared by the
>>>>>>>> >> >> >>>> > front
>>>>>>>> >> >> >>>> > and
>>>>>>>> >> >> >>>> > the rear?
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way
>>>>>>>> to turn
>>>>>>>> >> >> >>>> it
>>>>>>>> >> >> >>>> into
>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>> >> >> >>>> implementation,
>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>>>>> better
>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that
>>>>>>>> you
>>>>>>>> >> >> >>>> > reference,
>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>>>>> deque so
>>>>>>>> >> >> >>>> > if
>>>>>>>> >> >> >>>> > we
>>>>>>>> >> >> >>>> > take
>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>> >> >> >>>>
>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque operations
>>>>>>>> -
>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>> >> >> >>>
>>>>>>>> >> >> >>
>>>>>>>> >> >> >
>>>>>>>> >> >> >
>>>>>>>> >> >
>>>>>>>> >> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100415/88d6510a/attachment.html>

From kevin.l.stern at gmail.com  Thu Apr 15 11:12:45 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Thu, 15 Apr 2010 06:12:45 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
Message-ID: <g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>

Hi Benedict,

Unless I am misreading your post, this now has a very similar feel to the
first data structure that I posted to the list.  Martin Buchholz then
pointed out that we can incorporate the ideas from
[Brodnik99resizablearrays] and reap additional benefits.

Regards,

Kevin

On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <lists at laerad.com>wrote:

> Hi Kevin,
>
> Yes, as I was going to bed last night I realised I had not fully addressed
> the problem that was originally being visited; only reduced the constant
> factor for addition to the end of the list. A trivial modification fixes
> that, however; same scheme but up to some maximum arraylet size (of a power
> of 2), after which the array is increased in size linearly. Performance
> doesn't seem to have been affected appreciably, although not been exhaustive
> in the benchmarking:
>
> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
> 10 items inserts Chunked / ExpArray = 0.99
> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
> 10 items get Chunked / ExpArray = 0.99
> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
> 100 items inserts Chunked / ExpArray = 1.23
> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
> 100 items get Chunked / ExpArray = 1.23
> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
> 1000 items inserts Chunked / ExpArray = 1.19
> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
> 1000 items get Chunked / ExpArray = 1.19
> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
> 10000 items inserts Chunked / ExpArray = 1.18
> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
> 10000 items get Chunked / ExpArray = 1.18
> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
> 100000 items inserts Chunked / ExpArray = 1.09
> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
> 100000 items get Chunked / ExpArray = 1.09
>
> The nice thing about this is that the maximum amount of wasted memory is
> user configurable. Even with a low setting as above (65K) performance seems
> pretty consistent.
>
> Code for calculating index and array offset are pretty straight forward;
> haven't given much thought to optimisations just yet:
>
> private final int indexFor(int a, int i) {
> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
> maxArraySizeShift : 1 << a) ;
>  }
> private final int arrayFor(int i) {
> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>  }
>
> Regarding the double list idea - yes, I agree, I certainly didn't think
> that one through fully!
>
>
>
> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Hi Benedict,
>>
>> Like you, I am relatively new to this mailing list; I am also trying to
>> tread lightly so as not to step on any toes.  That being said, I think that
>> I can offer a response to your inquiry.
>>
>> Regarding:  "The idea is to simply double the new array size each time a
>> new array needs to be allocated"
>>
>> It seems this would not address the desire of offering an alternative to
>> the allocation of a large backing array for ArrayList (your largest backing
>> array could still reach a size of 1/2 * Integer.MAX_VALUE) and would not
>> address the desire of wasting the (asymptotically) minimum amount of memory
>> in the worst case while maintaining O(1) amortized time bounds.  The data
>> structure described in [Brodnik99resizablearrays] has a maximum backing
>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>> over ArrayList do you see in your data structure?
>>
>> Regarding:  "Also, with regard to a Deque implementation, it seems that
>> the simplest solution would be to simply have two lists, with one accepting
>> inserts for near the beginning and being ordered in reverse whilst the other
>> accepted inserts for near to the end."
>>
>> What happens with your structure when you add n elements and then remove
>> element 0 n times?  I think that once you work out all the kinks you'll end
>> up with the two stacks approach, which is mentioned in
>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>> you'll end up with the circular list approach, which is not friendly to O(1)
>> amortized time bounds in a data structure that resizes more often than O(n)
>> due to the 'unshift' to the front = 0 position.  I think the best approach
>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>> that I am currently working on.  Incidentally, this approach also provides
>> for a much improved index unpacking procedure using only bit shifts and bit
>> masks, although it is at the expense of (O(1)) additional work during
>> resize.
>>
>> Regards,
>>
>> Kevin
>>
>>
>>
>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <lists at laerad.com
>> > wrote:
>>
>>> Hi,
>>>
>>> I hope you don't consider it rude to involve myself in this conversation
>>> towards the end - I joined the mailing list only recently.
>>>
>>> I'm not sure if this offers a huge amount to the discussion, but I have
>>> tinkered with a "chunked" array list which seems to offer better time
>>> performance in general at the cost of greater (worst case) memory
>>> utilisation. It is easier to understand IMHO as well, although this is not
>>> necessarily a great benefit here. It turns out the idea is very similar to
>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>> is to simply double the new array size each time a new array needs to be
>>> allocated, or in effect allocate an array that is the size of all existing
>>> arrays put together. With this scheme the calculation for array and offset
>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>> obviously inserts at the end are much quicker.
>>>
>>> I have prototyped the data structure this evening and benchmarked
>>> additions at the end of the list, for which the performance is pretty
>>> impressive.
>>>
>>> Some random statistics for addition only on the client JVM (I have
>>> quickly dubbed my implementation ExpArrayList)
>>> All statistics were run in two rounds with ~1000 runs per round per
>>> statistic per list, and the second round results were used.
>>>
>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>> 10 items Chunked / ExpArray = 1.12
>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>> 100 items Chunked / ExpArray = 1.45
>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>> 1000 items Chunked / ExpArray = 2.02
>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>> 10000 items Chunked / ExpArray = 1.79
>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>> 100000 items Chunked / ExpArray = 1.64
>>>
>>> and server JVM:
>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>> 10 items Chunked / ExpArray = 0.86
>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>> 100 items Chunked / ExpArray = 1.34
>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>> 1000 items Chunked / ExpArray = 1.27
>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>> 10000 items Chunked / ExpArray = 1.12
>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>> 100000 items Chunked / ExpArray = 1.10
>>>
>>> Interestingly insertion at the beginning of the list appears to be
>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>> expected them to be fairly close.
>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at the
>>> beginning of large lists, which I haven't yet tried to understand. Insertion
>>> in the middle is similar.
>>>
>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>> 10 items Chunked / ExpArray = 2.59
>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>> 100 items Chunked / ExpArray = 2.14
>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>> 1000 items Chunked / ExpArray = 2.59
>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>> 10000 items Chunked / ExpArray = 2.16
>>>
>>> Finally, there are promising results for get() from the ExpArrayList as
>>> well (server JVM), again somehow beating ArrayList for larger lists:
>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>> 10 items get Chunked / ExpArray = 1.10
>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>> 100 items get Chunked / ExpArray = 1.25
>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>> 1000 items get Chunked / ExpArray = 1.33
>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>> 10000 items get Chunked / ExpArray = 1.24
>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>> 100000 items get Chunked / ExpArray = 1.22
>>>
>>>
>>> I'm willing to explore this further but I'm not sure how desirable that
>>> is, given that Kevin's data structure appears to perform pretty well already
>>> wrt to CPU time, and better wrt to memory utilisation, and in effect this
>>> mostly changes only the function to determine which array to use, not the
>>> body of the implementation. Let me know if you would like a copy of the
>>> source code and I will find somewhere to upload it.
>>>
>>> Also, with regard to a Deque implementation, it seems that the simplest
>>> solution would be to simply have two lists, with one accepting inserts for
>>> near the beginning and being ordered in reverse whilst the other accepted
>>> inserts for near to the end. The only trick would be having the list at the
>>> beginning support iteration in reverse order cheaply, but this could easily
>>> be achieved by creating an extension of List with a reverseIterator()
>>> method.
>>>
>>>
>>> Anyway, not sure if this helped at all but fancied joining in...
>>>
>>>
>>>
>>>
>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> It implements List, as well as Deque. It is indeed based on ArrayDeque,
>>>> with the added operations to implement list. It does so reasonably
>>>> efficiently, moving the fewest elements possible on each operation, that is
>>>> zero for the queue operations, at most n/2 for the rest and all of them for
>>>> a backing array resize.
>>>>
>>>> The idea is to get a replacement for arraylist that performs like
>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>> performance on other operations by requiring fewer elements to be moved.
>>>>
>>>> Thanks,
>>>> Joe
>>>>
>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>
>>>> Hi Joe,
>>>>>
>>>>> I was referring to the ChunkedArrayList when I stated that add does not
>>>>> amortize to constant time when the data structure employs the circular list
>>>>> trick to achieve deque behavior; ChunkedArrayList potentially resizes every
>>>>> n^(1/2) operations.
>>>>>
>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>
>>>>>> Hi Kevin, Martin,
>>>>>>
>>>>>> To add another discussion point, I've been writing a
>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>
>>>>>>
>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>
>>>>>> I'd be interested if you have any comments in the context of this
>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>
>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>> operations. Timings for indexed reads and writes showed
>>>>>> no discernible difference between implementations last time I ran the
>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>> at most half of the elements, except when resizing the array.
>>>>>>
>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>> implementation only moves array elements to the front on an array resize
>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>
>>>>>> Some performance results below, code for these is in the repository
>>>>>> above too. This was the second run, after a warmup.
>>>>>>
>>>>>> Thanks,
>>>>>> Joe
>>>>>>
>>>>>> ------------------------------------------------ CircularArrayList
>>>>>> ------------------------------------------------
>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>      10          20          67          70         125         102
>>>>>>        90         240         191         138
>>>>>>    100          19          67          70         166         138
>>>>>>      94         230         194         118
>>>>>>   1000          28          64          67         681         538
>>>>>>      91         324         382         119
>>>>>>   10000          30          65          67        5884        4425
>>>>>>        94        1296        2330         124
>>>>>> ---------------------------------------------------- ArrayList
>>>>>> ----------------------------------------------------
>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>      10          23          68          70         100          69
>>>>>>     32913         162         130         105
>>>>>>    100          20          67          70         129         104
>>>>>>   21944         169         134         135
>>>>>>   1000          29          63          67         651         506
>>>>>>    9602         364         333         526
>>>>>>   10000          30          63          66        5878        4414
>>>>>>      9947        2312        2280        4437
>>>>>>
>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>
>>>>>> Hi Martin,
>>>>>>>
>>>>>>> I had intended to address your request for absolute O(1) operations
>>>>>>> in the previous email.  The approach to achieving this suggested in
>>>>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Martin,
>>>>>>>>
>>>>>>>> It's interesting to note that the old circular list trick will not
>>>>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>>>>> beginning or the end; this is the approach that I will take to create a
>>>>>>>> deque.
>>>>>>>>
>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D. Demaine
>>>>>>>> and J. Ian Munro and Robert Sedgewick},
>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>     year = {1999}
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>>
>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>
>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>> ArrayList at some operations.
>>>>>>>>>
>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>
>>>>>>>>> An additional property that would be nice to
>>>>>>>>> have (but don't try too hard)
>>>>>>>>> is to provide some kind of real-time
>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>> in some real-time applications.
>>>>>>>>>
>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>> with having this class included in any of
>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>> if you have not done so yet.
>>>>>>>>>
>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>
>>>>>>>>> We should get various people a chance to give
>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>> > Hello Martin,
>>>>>>>>> >
>>>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>> shape.  I have
>>>>>>>>> > also gathered some data on the performance of ChunkedArrayList
>>>>>>>>> over
>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>> (note that the
>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>> operation with
>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>> indicates
>>>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList is
>>>>>>>>> less
>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).  I've
>>>>>>>>> noticed
>>>>>>>>> > relatively significant variability in a few of the numbers when I
>>>>>>>>> switch
>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>> performance
>>>>>>>>> > expectations.  For my test I generated x elements and then timed
>>>>>>>>> the process
>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a
>>>>>>>>> get
>>>>>>>>> > operation on each for indices 0 through x-1 and finally I used
>>>>>>>>> the iterator
>>>>>>>>> > mechanism to retrieve the first through xth element (of course, I
>>>>>>>>> performed
>>>>>>>>> > each of these operations multiple times throwing away the timing
>>>>>>>>> for the
>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>> >
>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>> java.util, I would
>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>> eliminate the
>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>> achieving this by
>>>>>>>>> > way of a data structure that is both time and space optimal is a
>>>>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>>>>> backing
>>>>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>>>>> dynamic
>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>> ArrayList, and
>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>> ArrayList.  Of
>>>>>>>>> > course, this data structure does not do everything better than
>>>>>>>>> ArrayList; in
>>>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>>>> decomposition
>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>> additional memory
>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to the
>>>>>>>>> multiple
>>>>>>>>> > array copies necessary to complete the shift.  That being said, I
>>>>>>>>> think that
>>>>>>>>> > the additional cost of indexed access is partially mitigated by
>>>>>>>>> the
>>>>>>>>> > availability of iterator and listIterator, whose implementations
>>>>>>>>> do not use
>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>> ArrayList due
>>>>>>>>> > to its linear time complexity.
>>>>>>>>> >
>>>>>>>>> > Kevin
>>>>>>>>> >
>>>>>>>>> > 1000000 elements:
>>>>>>>>> > Client JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>> >
>>>>>>>>> > Server JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>> >
>>>>>>>>> > 100000 elements:
>>>>>>>>> > Client JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>> >
>>>>>>>>> > Server JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>> >
>>>>>>>>> > 10000 elements:
>>>>>>>>> > Client JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>> >
>>>>>>>>> > Server JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>> >
>>>>>>>>> > 1000 elements:
>>>>>>>>> > Client JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>> >
>>>>>>>>> > Server JVM:
>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>> >>
>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>> >> micro-optimizations now.
>>>>>>>>> >>
>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>> >>
>>>>>>>>> >> Martin
>>>>>>>>> >>
>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>> >> wrote:
>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>> originally
>>>>>>>>> >> > provided (once again, it is
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>> ).
>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet as
>>>>>>>>> it was
>>>>>>>>> >> > unclear
>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>> >> > _________________
>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>>>> arrays
>>>>>>>>> >> > rather
>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>> >> >
>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>> java.util.ArrayList
>>>>>>>>> >> > with
>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>> add-at-front or
>>>>>>>>> >> > other
>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much more
>>>>>>>>> important
>>>>>>>>> >> > and
>>>>>>>>> >> > popular collection, it's the primary "straight replacement for
>>>>>>>>> primitive
>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>> >> > _________________
>>>>>>>>> >> >
>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>> updating the
>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>> >> >
>>>>>>>>> >> > Thoughts?
>>>>>>>>> >> >
>>>>>>>>> >> > Thanks,
>>>>>>>>> >> >
>>>>>>>>> >> > Kevin
>>>>>>>>> >> >
>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>> martinrb at google.com>
>>>>>>>>> >> > wrote:
>>>>>>>>> >> >>
>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>> >> >>
>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>> >> >>
>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>> >> >> about your implementation.
>>>>>>>>> >> >>
>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>> >> >>
>>>>>>>>> >> >>
>>>>>>>>> >> >>
>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>> >> >>
>>>>>>>>> >> >> Martin
>>>>>>>>> >> >>
>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>> >> >> wrote:
>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The
>>>>>>>>> code below
>>>>>>>>> >> >> > gives
>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>> offset into the
>>>>>>>>> >> >> > data
>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>> this will
>>>>>>>>> >> >> > include a
>>>>>>>>> >> >> > bit more work to access an element than a simple bit shift
>>>>>>>>> and a bit
>>>>>>>>> >> >> > mask?
>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>> each time an
>>>>>>>>> >> >> > index
>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the
>>>>>>>>> bits to see
>>>>>>>>> >> >> > if
>>>>>>>>> >> >> > I
>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>> >> >> >             } else {
>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>> >> >> >             }
>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>> >> >> >
>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + "
>>>>>>>>> " + e);
>>>>>>>>> >> >> >         }
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > Kevin
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>> >> >> > wrote:
>>>>>>>>> >> >> >>
>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>> numbered
>>>>>>>>> >> >> >> superblocks,
>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term added
>>>>>>>>> (the
>>>>>>>>> >> >> >> number
>>>>>>>>> >> >> >> of
>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>>>>> anyhow, I
>>>>>>>>> >> >> >> also
>>>>>>>>> >> >> >> came
>>>>>>>>> >> >> >> across an alternative characterization of superblock in
>>>>>>>>> the paper
>>>>>>>>> >> >> >> which
>>>>>>>>> >> >> >> states that data blocks are grouped within a superblock
>>>>>>>>> when they
>>>>>>>>> >> >> >> are
>>>>>>>>> >> >> >> the
>>>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>>>> structure
>>>>>>>>> >> >> >> below
>>>>>>>>> >> >> >> would be
>>>>>>>>> >> >> >>
>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>> >> >> >>
>>>>>>>>> >> >> >> which seems to contradict my understanding of (1) below.
>>>>>>>>> I must be
>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>> >> >> >>
>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>> >> >> >> wrote:
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal
>>>>>>>>> time and
>>>>>>>>> >> >> >>> space"
>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>> following
>>>>>>>>> >> >> >>> property:
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>> consists of
>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies
>>>>>>>>> the
>>>>>>>>> >> >> >>> following
>>>>>>>>> >> >> >>> structure:
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>> >> >> >>> [...]
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>> "the number
>>>>>>>>> >> >> >>> of
>>>>>>>>> >> >> >>> data
>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only two
>>>>>>>>> data
>>>>>>>>> >> >> >>> blocks
>>>>>>>>> >> >> >>> in
>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>>>>> unless I'm
>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>> superblocks prior
>>>>>>>>> >> >> >>> to
>>>>>>>>> >> >> >>> SB_k
>>>>>>>>> >> >> >>> should be:
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>> example above,
>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of their
>>>>>>>>> data
>>>>>>>>> >> >> >>> structure, but
>>>>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>>>>> rather than
>>>>>>>>> >> >> >>> my
>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> Kevin
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>> >> >> >>> wrote:
>>>>>>>>> >> >> >>>>
>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>> >> >> >>>> >
>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>> that comes to
>>>>>>>>> >> >> >>>> > mind
>>>>>>>>> >> >> >>>> > to
>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion is
>>>>>>>>> to create
>>>>>>>>> >> >> >>>> > a
>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert
>>>>>>>>> at the
>>>>>>>>> >> >> >>>> > front
>>>>>>>>> >> >> >>>> > requires
>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and
>>>>>>>>> to insert
>>>>>>>>> >> >> >>>> > at
>>>>>>>>> >> >> >>>> > the
>>>>>>>>> >> >> >>>> > rear
>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>> size).  We
>>>>>>>>> >> >> >>>> > need
>>>>>>>>> >> >> >>>> > to
>>>>>>>>> >> >> >>>> > resize
>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could you
>>>>>>>>> explain
>>>>>>>>> >> >> >>>> > more
>>>>>>>>> >> >> >>>> > about
>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>> shared by the
>>>>>>>>> >> >> >>>> > front
>>>>>>>>> >> >> >>>> > and
>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>> >> >> >>>>
>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a way
>>>>>>>>> to turn
>>>>>>>>> >> >> >>>> it
>>>>>>>>> >> >> >>>> into
>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>> >> >> >>>>
>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>>>>>> better
>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper that
>>>>>>>>> you
>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>>>>>> deque so
>>>>>>>>> >> >> >>>> > if
>>>>>>>>> >> >> >>>> > we
>>>>>>>>> >> >> >>>> > take
>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>> >> >> >>>>
>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>> operations -
>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>> >> >> >>>
>>>>>>>>> >> >> >>
>>>>>>>>> >> >> >
>>>>>>>>> >> >> >
>>>>>>>>> >> >
>>>>>>>>> >> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100415/e89225df/attachment.html>

From lists at laerad.com  Thu Apr 15 13:00:21 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Thu, 15 Apr 2010 14:00:21 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
Message-ID: <s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>

Sorry Kevin - it sounds like I might be being of more hindrance than help.
that part of the discussion was clearly truncated by the time I had joined
the list - I haven't been able to find the history in the archives either...

I was just wondering about the worst case cost of add() as described by your
javadoc; admittedly it is optimal with respect to unused memory, but the
worst case cost of an add is still sqrt(n), with a relatively high constant
factor. I had been thinking that once n passed a threshold the cost of
additions in this other structure would become a constant factor, offering
nice algorithmic complexity guarantees for large n; however since
sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
allocations would have to be unrealistically small (assuming linear cost for
allocation) for this to be the case. It would still be nice to have a data
structure that avoids needing to copy data with each grow, whilst still
maintaining good memory performance.

That *all* being said, I had been going by your javadoc and emails to
ascertain the behaviour of this class, as I couldn't locate a free copy
of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
sqrt(n) cost appears to be associated with growing the backing array, rather
than with what I assumed to be copying data between arraylets, and it seems
this cost is pretty optimal. That will teach me to post to a list without
getting my facts straight first. The interesting thing is simply that the
constant factor for this implementation still seems to be quite high,
although perhaps that is simply because I was not benchmarking sufficiently
large values of n.



On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> Unless I am misreading your post, this now has a very similar feel to the
> first data structure that I posted to the list.  Martin Buchholz then
> pointed out that we can incorporate the ideas from
> [Brodnik99resizablearrays] and reap additional benefits.
>
> Regards,
>
> Kevin
>
>
> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <lists at laerad.com>wrote:
>
>> Hi Kevin,
>>
>> Yes, as I was going to bed last night I realised I had not fully addressed
>> the problem that was originally being visited; only reduced the constant
>> factor for addition to the end of the list. A trivial modification fixes
>> that, however; same scheme but up to some maximum arraylet size (of a power
>> of 2), after which the array is increased in size linearly. Performance
>> doesn't seem to have been affected appreciably, although not been exhaustive
>> in the benchmarking:
>>
>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>> 10 items inserts Chunked / ExpArray = 0.99
>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>> 10 items get Chunked / ExpArray = 0.99
>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>> 100 items inserts Chunked / ExpArray = 1.23
>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>> 100 items get Chunked / ExpArray = 1.23
>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>> 1000 items inserts Chunked / ExpArray = 1.19
>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>> 1000 items get Chunked / ExpArray = 1.19
>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>> 10000 items inserts Chunked / ExpArray = 1.18
>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>> 10000 items get Chunked / ExpArray = 1.18
>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>> 100000 items inserts Chunked / ExpArray = 1.09
>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>> 100000 items get Chunked / ExpArray = 1.09
>>
>> The nice thing about this is that the maximum amount of wasted memory is
>> user configurable. Even with a low setting as above (65K) performance seems
>> pretty consistent.
>>
>> Code for calculating index and array offset are pretty straight forward;
>> haven't given much thought to optimisations just yet:
>>
>> private final int indexFor(int a, int i) {
>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>> maxArraySizeShift : 1 << a) ;
>>  }
>> private final int arrayFor(int i) {
>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>  }
>>
>> Regarding the double list idea - yes, I agree, I certainly didn't think
>> that one through fully!
>>
>>
>>
>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Hi Benedict,
>>>
>>> Like you, I am relatively new to this mailing list; I am also trying to
>>> tread lightly so as not to step on any toes.  That being said, I think that
>>> I can offer a response to your inquiry.
>>>
>>> Regarding:  "The idea is to simply double the new array size each time a
>>> new array needs to be allocated"
>>>
>>> It seems this would not address the desire of offering an alternative to
>>> the allocation of a large backing array for ArrayList (your largest backing
>>> array could still reach a size of 1/2 * Integer.MAX_VALUE) and would not
>>> address the desire of wasting the (asymptotically) minimum amount of memory
>>> in the worst case while maintaining O(1) amortized time bounds.  The data
>>> structure described in [Brodnik99resizablearrays] has a maximum backing
>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>> over ArrayList do you see in your data structure?
>>>
>>> Regarding:  "Also, with regard to a Deque implementation, it seems that
>>> the simplest solution would be to simply have two lists, with one accepting
>>> inserts for near the beginning and being ordered in reverse whilst the other
>>> accepted inserts for near to the end."
>>>
>>> What happens with your structure when you add n elements and then remove
>>> element 0 n times?  I think that once you work out all the kinks you'll end
>>> up with the two stacks approach, which is mentioned in
>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>> amortized time bounds in a data structure that resizes more often than O(n)
>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>> that I am currently working on.  Incidentally, this approach also provides
>>> for a much improved index unpacking procedure using only bit shifts and bit
>>> masks, although it is at the expense of (O(1)) additional work during
>>> resize.
>>>
>>> Regards,
>>>
>>> Kevin
>>>
>>>
>>>
>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I hope you don't consider it rude to involve myself in this conversation
>>>> towards the end - I joined the mailing list only recently.
>>>>
>>>> I'm not sure if this offers a huge amount to the discussion, but I have
>>>> tinkered with a "chunked" array list which seems to offer better time
>>>> performance in general at the cost of greater (worst case) memory
>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>> is to simply double the new array size each time a new array needs to be
>>>> allocated, or in effect allocate an array that is the size of all existing
>>>> arrays put together. With this scheme the calculation for array and offset
>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>> obviously inserts at the end are much quicker.
>>>>
>>>> I have prototyped the data structure this evening and benchmarked
>>>> additions at the end of the list, for which the performance is pretty
>>>> impressive.
>>>>
>>>> Some random statistics for addition only on the client JVM (I have
>>>> quickly dubbed my implementation ExpArrayList)
>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>> statistic per list, and the second round results were used.
>>>>
>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>> 10 items Chunked / ExpArray = 1.12
>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>> 100 items Chunked / ExpArray = 1.45
>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>> 1000 items Chunked / ExpArray = 2.02
>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>> 10000 items Chunked / ExpArray = 1.79
>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>> 100000 items Chunked / ExpArray = 1.64
>>>>
>>>> and server JVM:
>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>> 10 items Chunked / ExpArray = 0.86
>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>> 100 items Chunked / ExpArray = 1.34
>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>> 1000 items Chunked / ExpArray = 1.27
>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>> 10000 items Chunked / ExpArray = 1.12
>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>> 100000 items Chunked / ExpArray = 1.10
>>>>
>>>> Interestingly insertion at the beginning of the list appears to be
>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>> expected them to be fairly close.
>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at
>>>> the beginning of large lists, which I haven't yet tried to understand.
>>>> Insertion in the middle is similar.
>>>>
>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>> 10 items Chunked / ExpArray = 2.59
>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>> 100 items Chunked / ExpArray = 2.14
>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>> 1000 items Chunked / ExpArray = 2.59
>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>> 10000 items Chunked / ExpArray = 2.16
>>>>
>>>> Finally, there are promising results for get() from the ExpArrayList as
>>>> well (server JVM), again somehow beating ArrayList for larger lists:
>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>> 10 items get Chunked / ExpArray = 1.10
>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>> 100 items get Chunked / ExpArray = 1.25
>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>> 1000 items get Chunked / ExpArray = 1.33
>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>> 10000 items get Chunked / ExpArray = 1.24
>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>
>>>>
>>>> I'm willing to explore this further but I'm not sure how desirable that
>>>> is, given that Kevin's data structure appears to perform pretty well already
>>>> wrt to CPU time, and better wrt to memory utilisation, and in effect this
>>>> mostly changes only the function to determine which array to use, not the
>>>> body of the implementation. Let me know if you would like a copy of the
>>>> source code and I will find somewhere to upload it.
>>>>
>>>> Also, with regard to a Deque implementation, it seems that the simplest
>>>> solution would be to simply have two lists, with one accepting inserts for
>>>> near the beginning and being ordered in reverse whilst the other accepted
>>>> inserts for near to the end. The only trick would be having the list at the
>>>> beginning support iteration in reverse order cheaply, but this could easily
>>>> be achieved by creating an extension of List with a reverseIterator()
>>>> method.
>>>>
>>>>
>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>
>>>>
>>>>
>>>>
>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> It implements List, as well as Deque. It is indeed based on ArrayDeque,
>>>>> with the added operations to implement list. It does so reasonably
>>>>> efficiently, moving the fewest elements possible on each operation, that is
>>>>> zero for the queue operations, at most n/2 for the rest and all of them for
>>>>> a backing array resize.
>>>>>
>>>>> The idea is to get a replacement for arraylist that performs like
>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>
>>>>> Thanks,
>>>>> Joe
>>>>>
>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>
>>>>> Hi Joe,
>>>>>>
>>>>>> I was referring to the ChunkedArrayList when I stated that add does
>>>>>> not amortize to constant time when the data structure employs the circular
>>>>>> list trick to achieve deque behavior; ChunkedArrayList potentially resizes
>>>>>> every n^(1/2) operations.
>>>>>>
>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>
>>>>>>> Hi Kevin, Martin,
>>>>>>>
>>>>>>> To add another discussion point, I've been writing a
>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>
>>>>>>>
>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>
>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>
>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>
>>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>>> implementation only moves array elements to the front on an array resize
>>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>
>>>>>>> Some performance results below, code for these is in the repository
>>>>>>> above too. This was the second run, after a warmup.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Joe
>>>>>>>
>>>>>>> ------------------------------------------------ CircularArrayList
>>>>>>> ------------------------------------------------
>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>      10          20          67          70         125         102
>>>>>>>          90         240         191         138
>>>>>>>    100          19          67          70         166         138
>>>>>>>        94         230         194         118
>>>>>>>   1000          28          64          67         681         538
>>>>>>>        91         324         382         119
>>>>>>>   10000          30          65          67        5884        4425
>>>>>>>          94        1296        2330         124
>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>> ----------------------------------------------------
>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>      10          23          68          70         100          69
>>>>>>>       32913         162         130         105
>>>>>>>    100          20          67          70         129         104
>>>>>>>     21944         169         134         135
>>>>>>>   1000          29          63          67         651         506
>>>>>>>      9602         364         333         526
>>>>>>>   10000          30          63          66        5878        4414
>>>>>>>        9947        2312        2280        4437
>>>>>>>
>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>
>>>>>>> Hi Martin,
>>>>>>>>
>>>>>>>> I had intended to address your request for absolute O(1) operations
>>>>>>>> in the previous email.  The approach to achieving this suggested in
>>>>>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> It's interesting to note that the old circular list trick will not
>>>>>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>>>>>> beginning or the end; this is the approach that I will take to create a
>>>>>>>>> deque.
>>>>>>>>>
>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>     year = {1999}
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin,
>>>>>>>>>>
>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>
>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>
>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>
>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>> in some real-time applications.
>>>>>>>>>>
>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>> with having this class included in any of
>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>
>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>
>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>> > Hello Martin,
>>>>>>>>>> >
>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>> shape.  I have
>>>>>>>>>> > also gathered some data on the performance of ChunkedArrayList
>>>>>>>>>> over
>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>>> (note that the
>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>> operation with
>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>> indicates
>>>>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList
>>>>>>>>>> is less
>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>> I've noticed
>>>>>>>>>> > relatively significant variability in a few of the numbers when
>>>>>>>>>> I switch
>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>> performance
>>>>>>>>>> > expectations.  For my test I generated x elements and then timed
>>>>>>>>>> the process
>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed a
>>>>>>>>>> get
>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I used
>>>>>>>>>> the iterator
>>>>>>>>>> > mechanism to retrieve the first through xth element (of course,
>>>>>>>>>> I performed
>>>>>>>>>> > each of these operations multiple times throwing away the timing
>>>>>>>>>> for the
>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>> >
>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>> java.util, I would
>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>> eliminate the
>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>> achieving this by
>>>>>>>>>> > way of a data structure that is both time and space optimal is a
>>>>>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>>>>>> backing
>>>>>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>>>>>> dynamic
>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>> ArrayList, and
>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>> ArrayList.  Of
>>>>>>>>>> > course, this data structure does not do everything better than
>>>>>>>>>> ArrayList; in
>>>>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>>>>> decomposition
>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>> additional memory
>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to
>>>>>>>>>> the multiple
>>>>>>>>>> > array copies necessary to complete the shift.  That being said,
>>>>>>>>>> I think that
>>>>>>>>>> > the additional cost of indexed access is partially mitigated by
>>>>>>>>>> the
>>>>>>>>>> > availability of iterator and listIterator, whose implementations
>>>>>>>>>> do not use
>>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>> ArrayList due
>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>> >
>>>>>>>>>> > Kevin
>>>>>>>>>> >
>>>>>>>>>> > 1000000 elements:
>>>>>>>>>> > Client JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>> >
>>>>>>>>>> > Server JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>> >
>>>>>>>>>> > 100000 elements:
>>>>>>>>>> > Client JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>> >
>>>>>>>>>> > Server JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>> >
>>>>>>>>>> > 10000 elements:
>>>>>>>>>> > Client JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>> >
>>>>>>>>>> > Server JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>> >
>>>>>>>>>> > 1000 elements:
>>>>>>>>>> > Client JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>> >
>>>>>>>>>> > Server JVM:
>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>> >>
>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>> >>
>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>> >>
>>>>>>>>>> >> Martin
>>>>>>>>>> >>
>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>> >> wrote:
>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>> originally
>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>> >> >
>>>>>>>>>> >> >
>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>> ).
>>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet as
>>>>>>>>>> it was
>>>>>>>>>> >> > unclear
>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>> >> > _________________
>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>>>>> arrays
>>>>>>>>>> >> > rather
>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>> >> >
>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>> java.util.ArrayList
>>>>>>>>>> >> > with
>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>> add-at-front or
>>>>>>>>>> >> > other
>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>> more important
>>>>>>>>>> >> > and
>>>>>>>>>> >> > popular collection, it's the primary "straight replacement
>>>>>>>>>> for primitive
>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>> >> > _________________
>>>>>>>>>> >> >
>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>> updating the
>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>> >> >
>>>>>>>>>> >> > Thoughts?
>>>>>>>>>> >> >
>>>>>>>>>> >> > Thanks,
>>>>>>>>>> >> >
>>>>>>>>>> >> > Kevin
>>>>>>>>>> >> >
>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>> martinrb at google.com>
>>>>>>>>>> >> > wrote:
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>> >> >>
>>>>>>>>>> >> >>
>>>>>>>>>> >> >>
>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> Martin
>>>>>>>>>> >> >>
>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>> >> >> wrote:
>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.  The
>>>>>>>>>> code below
>>>>>>>>>> >> >> > gives
>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>> offset into the
>>>>>>>>>> >> >> > data
>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>>> this will
>>>>>>>>>> >> >> > include a
>>>>>>>>>> >> >> > bit more work to access an element than a simple bit shift
>>>>>>>>>> and a bit
>>>>>>>>>> >> >> > mask?
>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>>> each time an
>>>>>>>>>> >> >> > index
>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the
>>>>>>>>>> bits to see
>>>>>>>>>> >> >> > if
>>>>>>>>>> >> >> > I
>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>> >> >> >             } else {
>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>> >> >> >             }
>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) + "
>>>>>>>>>> " + e);
>>>>>>>>>> >> >> >         }
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > Kevin
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>> >> >> > wrote:
>>>>>>>>>> >> >> >>
>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>> numbered
>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>> added (the
>>>>>>>>>> >> >> >> number
>>>>>>>>>> >> >> >> of
>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>>>>>> anyhow, I
>>>>>>>>>> >> >> >> also
>>>>>>>>>> >> >> >> came
>>>>>>>>>> >> >> >> across an alternative characterization of superblock in
>>>>>>>>>> the paper
>>>>>>>>>> >> >> >> which
>>>>>>>>>> >> >> >> states that data blocks are grouped within a superblock
>>>>>>>>>> when they
>>>>>>>>>> >> >> >> are
>>>>>>>>>> >> >> >> the
>>>>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>>>>> structure
>>>>>>>>>> >> >> >> below
>>>>>>>>>> >> >> >> would be
>>>>>>>>>> >> >> >>
>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>> >> >> >>
>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1) below.
>>>>>>>>>> I must be
>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>> >> >> >>
>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in optimal
>>>>>>>>>> time and
>>>>>>>>>> >> >> >>> space"
>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>> following
>>>>>>>>>> >> >> >>> property:
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>> consists of
>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size 2^(ceil(k/2))."
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies
>>>>>>>>>> the
>>>>>>>>>> >> >> >>> following
>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i = 3:
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>>> "the number
>>>>>>>>>> >> >> >>> of
>>>>>>>>>> >> >> >>> data
>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only
>>>>>>>>>> two data
>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>> >> >> >>> in
>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>>>>>> unless I'm
>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>> superblocks prior
>>>>>>>>>> >> >> >>> to
>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>> example above,
>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of their
>>>>>>>>>> data
>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>>>>>> rather than
>>>>>>>>>> >> >> >>> my
>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>> >> >> >>>>
>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>> >> >> >>>> >
>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>>> that comes to
>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion
>>>>>>>>>> is to create
>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert
>>>>>>>>>> at the
>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and
>>>>>>>>>> to insert
>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>>> size).  We
>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could
>>>>>>>>>> you explain
>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>>> shared by the
>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>> >> >> >>>>
>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a
>>>>>>>>>> way to turn
>>>>>>>>>> >> >> >>>> it
>>>>>>>>>> >> >> >>>> into
>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>> >> >> >>>>
>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be a
>>>>>>>>>> better
>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>> that you
>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives a
>>>>>>>>>> deque so
>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>> >> >> >>>>
>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>> operations -
>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>> >> >> >>>
>>>>>>>>>> >> >> >>
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >> >
>>>>>>>>>> >> >
>>>>>>>>>> >> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100415/3a2b0aab/attachment.html>

From chris.hegarty at oracle.com  Thu Apr 15 15:39:19 2010
From: chris.hegarty at oracle.com (chris.hegarty at oracle.com)
Date: Thu, 15 Apr 2010 15:39:19 +0000
Subject: hg: jdk7/tl/jdk: 6943219:
	test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/DNSIdentities.java
	fail in linux
Message-ID: <20100415154114.C888544575@hg.openjdk.java.net>

Changeset: ed61accf772e
Author:    chegar
Date:      2010-04-15 16:37 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ed61accf772e

6943219: test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/DNSIdentities.java fail in linux
Reviewed-by: andrew

! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/DNSIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPAddressDNSIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPAddressIPIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/IPIdentities.java
! test/sun/security/ssl/sun/net/www/protocol/https/HttpsURLConnection/Identities.java



From martinrb at google.com  Thu Apr 15 22:41:54 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 15 Apr 2010 15:41:54 -0700
Subject: A few char-fiddling optimizations for Pattern.java
Message-ID: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>

Hi Xueming and Ulf,

Please review a few more optimizations to be added to the
current crop:

http://cr.openjdk.java.net/~martin/webrevs/openjdk7/Pattern-opt/

Thanks,

Martin


From kevin.l.stern at gmail.com  Thu Apr 15 23:18:50 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Thu, 15 Apr 2010 18:18:50 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<x2y1704b7a21004130427we2bf1fa2mae67962b84ab4510@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
Message-ID: <n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>

Oh no worries Benedict, thanks for your interest in the topic.  Let me know
if you have any other questions or if you have any related ideas or
concerns.

On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <lists at laerad.com>wrote:

> Sorry Kevin - it sounds like I might be being of more hindrance than help.
> that part of the discussion was clearly truncated by the time I had joined
> the list - I haven't been able to find the history in the archives either...
>
> I was just wondering about the worst case cost of add() as described by
> your javadoc; admittedly it is optimal with respect to unused memory, but
> the worst case cost of an add is still sqrt(n), with a relatively high
> constant factor. I had been thinking that once n passed a threshold the cost
> of additions in this other structure would become a constant factor,
> offering nice algorithmic complexity guarantees for large n; however since
> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
> allocations would have to be unrealistically small (assuming linear cost for
> allocation) for this to be the case. It would still be nice to have a data
> structure that avoids needing to copy data with each grow, whilst still
> maintaining good memory performance.
>
> That *all* being said, I had been going by your javadoc and emails to
> ascertain the behaviour of this class, as I couldn't locate a free copy
> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
> sqrt(n) cost appears to be associated with growing the backing array, rather
> than with what I assumed to be copying data between arraylets, and it seems
> this cost is pretty optimal. That will teach me to post to a list without
> getting my facts straight first. The interesting thing is simply that the
> constant factor for this implementation still seems to be quite high,
> although perhaps that is simply because I was not benchmarking sufficiently
> large values of n.
>
>
>
> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Hi Benedict,
>>
>> Unless I am misreading your post, this now has a very similar feel to the
>> first data structure that I posted to the list.  Martin Buchholz then
>> pointed out that we can incorporate the ideas from
>> [Brodnik99resizablearrays] and reap additional benefits.
>>
>> Regards,
>>
>> Kevin
>>
>>
>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <lists at laerad.com
>> > wrote:
>>
>>> Hi Kevin,
>>>
>>> Yes, as I was going to bed last night I realised I had not fully
>>> addressed the problem that was originally being visited; only reduced the
>>> constant factor for addition to the end of the list. A trivial modification
>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>> power of 2), after which the array is increased in size linearly.
>>> Performance doesn't seem to have been affected appreciably, although not
>>> been exhaustive in the benchmarking:
>>>
>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>> 10 items inserts Chunked / ExpArray = 0.99
>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>> 10 items get Chunked / ExpArray = 0.99
>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>> 100 items inserts Chunked / ExpArray = 1.23
>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>> 100 items get Chunked / ExpArray = 1.23
>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>> 1000 items inserts Chunked / ExpArray = 1.19
>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>> 1000 items get Chunked / ExpArray = 1.19
>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>> 10000 items inserts Chunked / ExpArray = 1.18
>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>> 10000 items get Chunked / ExpArray = 1.18
>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>> 100000 items inserts Chunked / ExpArray = 1.09
>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>> 100000 items get Chunked / ExpArray = 1.09
>>>
>>> The nice thing about this is that the maximum amount of wasted memory is
>>> user configurable. Even with a low setting as above (65K) performance seems
>>> pretty consistent.
>>>
>>> Code for calculating index and array offset are pretty straight forward;
>>> haven't given much thought to optimisations just yet:
>>>
>>> private final int indexFor(int a, int i) {
>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>> maxArraySizeShift : 1 << a) ;
>>>  }
>>> private final int arrayFor(int i) {
>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>  }
>>>
>>> Regarding the double list idea - yes, I agree, I certainly didn't think
>>> that one through fully!
>>>
>>>
>>>
>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Hi Benedict,
>>>>
>>>> Like you, I am relatively new to this mailing list; I am also trying to
>>>> tread lightly so as not to step on any toes.  That being said, I think that
>>>> I can offer a response to your inquiry.
>>>>
>>>> Regarding:  "The idea is to simply double the new array size each time a
>>>> new array needs to be allocated"
>>>>
>>>> It seems this would not address the desire of offering an alternative to
>>>> the allocation of a large backing array for ArrayList (your largest backing
>>>> array could still reach a size of 1/2 * Integer.MAX_VALUE) and would not
>>>> address the desire of wasting the (asymptotically) minimum amount of memory
>>>> in the worst case while maintaining O(1) amortized time bounds.  The data
>>>> structure described in [Brodnik99resizablearrays] has a maximum backing
>>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>>> over ArrayList do you see in your data structure?
>>>>
>>>> Regarding:  "Also, with regard to a Deque implementation, it seems that
>>>> the simplest solution would be to simply have two lists, with one accepting
>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>> accepted inserts for near to the end."
>>>>
>>>> What happens with your structure when you add n elements and then remove
>>>> element 0 n times?  I think that once you work out all the kinks you'll end
>>>> up with the two stacks approach, which is mentioned in
>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>> that I am currently working on.  Incidentally, this approach also provides
>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>> masks, although it is at the expense of (O(1)) additional work during
>>>> resize.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>>
>>>>
>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I hope you don't consider it rude to involve myself in this
>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>
>>>>> I'm not sure if this offers a huge amount to the discussion, but I have
>>>>> tinkered with a "chunked" array list which seems to offer better time
>>>>> performance in general at the cost of greater (worst case) memory
>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>> is to simply double the new array size each time a new array needs to be
>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>> obviously inserts at the end are much quicker.
>>>>>
>>>>> I have prototyped the data structure this evening and benchmarked
>>>>> additions at the end of the list, for which the performance is pretty
>>>>> impressive.
>>>>>
>>>>> Some random statistics for addition only on the client JVM (I have
>>>>> quickly dubbed my implementation ExpArrayList)
>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>> statistic per list, and the second round results were used.
>>>>>
>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>> 10 items Chunked / ExpArray = 1.12
>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>> 100 items Chunked / ExpArray = 1.45
>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>
>>>>> and server JVM:
>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>> 10 items Chunked / ExpArray = 0.86
>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>> 100 items Chunked / ExpArray = 1.34
>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>
>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>> expected them to be fairly close.
>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at
>>>>> the beginning of large lists, which I haven't yet tried to understand.
>>>>> Insertion in the middle is similar.
>>>>>
>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>> 10 items Chunked / ExpArray = 2.59
>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>> 100 items Chunked / ExpArray = 2.14
>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>
>>>>> Finally, there are promising results for get() from the ExpArrayList as
>>>>> well (server JVM), again somehow beating ArrayList for larger lists:
>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>
>>>>>
>>>>> I'm willing to explore this further but I'm not sure how desirable that
>>>>> is, given that Kevin's data structure appears to perform pretty well already
>>>>> wrt to CPU time, and better wrt to memory utilisation, and in effect this
>>>>> mostly changes only the function to determine which array to use, not the
>>>>> body of the implementation. Let me know if you would like a copy of the
>>>>> source code and I will find somewhere to upload it.
>>>>>
>>>>> Also, with regard to a Deque implementation, it seems that the simplest
>>>>> solution would be to simply have two lists, with one accepting inserts for
>>>>> near the beginning and being ordered in reverse whilst the other accepted
>>>>> inserts for near to the end. The only trick would be having the list at the
>>>>> beginning support iteration in reverse order cheaply, but this could easily
>>>>> be achieved by creating an extension of List with a reverseIterator()
>>>>> method.
>>>>>
>>>>>
>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>> and all of them for a backing array resize.
>>>>>>
>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>
>>>>>> Thanks,
>>>>>> Joe
>>>>>>
>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>
>>>>>> Hi Joe,
>>>>>>>
>>>>>>> I was referring to the ChunkedArrayList when I stated that add does
>>>>>>> not amortize to constant time when the data structure employs the circular
>>>>>>> list trick to achieve deque behavior; ChunkedArrayList potentially resizes
>>>>>>> every n^(1/2) operations.
>>>>>>>
>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>
>>>>>>>> Hi Kevin, Martin,
>>>>>>>>
>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>
>>>>>>>>
>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>
>>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>
>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>
>>>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>>>> implementation only moves array elements to the front on an array resize
>>>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>
>>>>>>>> Some performance results below, code for these is in the repository
>>>>>>>> above too. This was the second run, after a warmup.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> ------------------------------------------------ CircularArrayList
>>>>>>>> ------------------------------------------------
>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>      10          20          67          70         125         102
>>>>>>>>          90         240         191         138
>>>>>>>>    100          19          67          70         166         138
>>>>>>>>        94         230         194         118
>>>>>>>>   1000          28          64          67         681         538
>>>>>>>>        91         324         382         119
>>>>>>>>   10000          30          65          67        5884        4425
>>>>>>>>          94        1296        2330         124
>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>> ----------------------------------------------------
>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>> insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>      10          23          68          70         100          69
>>>>>>>>       32913         162         130         105
>>>>>>>>    100          20          67          70         129         104
>>>>>>>>     21944         169         134         135
>>>>>>>>   1000          29          63          67         651         506
>>>>>>>>      9602         364         333         526
>>>>>>>>   10000          30          63          66        5878        4414
>>>>>>>>        9947        2312        2280        4437
>>>>>>>>
>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>
>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> I had intended to address your request for absolute O(1) operations
>>>>>>>>> in the previous email.  The approach to achieving this suggested in
>>>>>>>>> [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Martin,
>>>>>>>>>>
>>>>>>>>>> It's interesting to note that the old circular list trick will not
>>>>>>>>>> suffice to turn this data structure into a deque since we might be copying
>>>>>>>>>> all n elements back to the front = 0 position every n^(1/2) operations (add
>>>>>>>>>> wouldn't amortize to O(1)).  We could use the old two stacks trick (push
>>>>>>>>>> elements onto one stack, flip (the bottom) half (of) the elements to the
>>>>>>>>>> 'other' stack when the 'other' stack becomes empty), mentioned in
>>>>>>>>>> [Brodnik99resizablearrays], but I find this to be a bit CS 101.  In
>>>>>>>>>> [Brodnik99resizablearrays] the authors suggest a method for making all
>>>>>>>>>> blocks roughly the same size, allowing us to expand/shrink capacity at the
>>>>>>>>>> beginning or the end; this is the approach that I will take to create a
>>>>>>>>>> deque.
>>>>>>>>>>
>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>     year = {1999}
>>>>>>>>>>
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>
>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>
>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>
>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>
>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>
>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>
>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>
>>>>>>>>>>> Martin
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>> >
>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>>> shape.  I have
>>>>>>>>>>> > also gathered some data on the performance of ChunkedArrayList
>>>>>>>>>>> over
>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>>>> (note that the
>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>> operation with
>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>>> indicates
>>>>>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList
>>>>>>>>>>> is less
>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>>> I've noticed
>>>>>>>>>>> > relatively significant variability in a few of the numbers when
>>>>>>>>>>> I switch
>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>> performance
>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>> timed the process
>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed
>>>>>>>>>>> a get
>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I used
>>>>>>>>>>> the iterator
>>>>>>>>>>> > mechanism to retrieve the first through xth element (of course,
>>>>>>>>>>> I performed
>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>> timing for the
>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>> >
>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>> java.util, I would
>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>> eliminate the
>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>> achieving this by
>>>>>>>>>>> > way of a data structure that is both time and space optimal is
>>>>>>>>>>> a
>>>>>>>>>>> > particularly elegant solution as it not only guarantees that no
>>>>>>>>>>> backing
>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also provides
>>>>>>>>>>> dynamic
>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>> ArrayList, and
>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>> > course, this data structure does not do everything better than
>>>>>>>>>>> ArrayList; in
>>>>>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>>>>>> decomposition
>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>> additional memory
>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to
>>>>>>>>>>> the multiple
>>>>>>>>>>> > array copies necessary to complete the shift.  That being said,
>>>>>>>>>>> I think that
>>>>>>>>>>> > the additional cost of indexed access is partially mitigated by
>>>>>>>>>>> the
>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>> implementations do not use
>>>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>>> ArrayList due
>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>> >
>>>>>>>>>>> > Kevin
>>>>>>>>>>> >
>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>> > Client JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>> >
>>>>>>>>>>> > Server JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>> >
>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>> > Client JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>> >
>>>>>>>>>>> > Server JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>> >
>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>> > Client JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>> >
>>>>>>>>>>> > Server JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>> >
>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>> > Client JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>> >
>>>>>>>>>>> > Server JVM:
>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>> >>
>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>> >>
>>>>>>>>>>> >> Martin
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>> >> wrote:
>>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>>> originally
>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>> ).
>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet
>>>>>>>>>>> as it was
>>>>>>>>>>> >> > unclear
>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>> >> > _________________
>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>>>>>> arrays
>>>>>>>>>>> >> > rather
>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>> >> > with
>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>> add-at-front or
>>>>>>>>>>> >> > other
>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>>> more important
>>>>>>>>>>> >> > and
>>>>>>>>>>> >> > popular collection, it's the primary "straight replacement
>>>>>>>>>>> for primitive
>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>> >> > _________________
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>>> updating the
>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > Kevin
>>>>>>>>>>> >> >
>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>> >> > wrote:
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >>
>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> Martin
>>>>>>>>>>> >> >>
>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.
>>>>>>>>>>> The code below
>>>>>>>>>>> >> >> > gives
>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>> offset into the
>>>>>>>>>>> >> >> > data
>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>>>> this will
>>>>>>>>>>> >> >> > include a
>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>> shift and a bit
>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>>>> each time an
>>>>>>>>>>> >> >> > index
>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle the
>>>>>>>>>>> bits to see
>>>>>>>>>>> >> >> > if
>>>>>>>>>>> >> >> > I
>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>> >> >> >             }
>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) +
>>>>>>>>>>> " " + e);
>>>>>>>>>>> >> >> >         }
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>> >> >> >>
>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>> numbered
>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>> added (the
>>>>>>>>>>> >> >> >> number
>>>>>>>>>>> >> >> >> of
>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my interpretation;
>>>>>>>>>>> anyhow, I
>>>>>>>>>>> >> >> >> also
>>>>>>>>>>> >> >> >> came
>>>>>>>>>>> >> >> >> across an alternative characterization of superblock in
>>>>>>>>>>> the paper
>>>>>>>>>>> >> >> >> which
>>>>>>>>>>> >> >> >> states that data blocks are grouped within a superblock
>>>>>>>>>>> when they
>>>>>>>>>>> >> >> >> are
>>>>>>>>>>> >> >> >> the
>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>>>>>> structure
>>>>>>>>>>> >> >> >> below
>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>> >> >> >>
>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>> >> >> >>
>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>> below.  I must be
>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>> >> >> >>
>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>> optimal time and
>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>> following
>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>> consists of
>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this implies
>>>>>>>>>>> the
>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i =
>>>>>>>>>>> 3:
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>>>> "the number
>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only
>>>>>>>>>>> two data
>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1) above,
>>>>>>>>>>> unless I'm
>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>> superblocks prior
>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>>> example above,
>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of their
>>>>>>>>>>> data
>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their mistake
>>>>>>>>>>> rather than
>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>>>> that comes to
>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion
>>>>>>>>>>> is to create
>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to insert
>>>>>>>>>>> at the
>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size) and
>>>>>>>>>>> to insert
>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>>>> size).  We
>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could
>>>>>>>>>>> you explain
>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>>>> shared by the
>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a
>>>>>>>>>>> way to turn
>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be
>>>>>>>>>>> a better
>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>>> that you
>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives
>>>>>>>>>>> a deque so
>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>> operations -
>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>> >> >> >>>
>>>>>>>>>>> >> >> >>
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100415/f46fc80d/attachment.html>

From xueming.shen at oracle.com  Fri Apr 16 00:03:07 2010
From: xueming.shen at oracle.com (xueming.shen at oracle.com)
Date: Fri, 16 Apr 2010 00:03:07 +0000
Subject: hg: jdk7/tl/jdk: 6931676: TEST_BUG: wrong value for bugid in comments
Message-ID: <20100416000326.04CFC44583@hg.openjdk.java.net>

Changeset: c70d4266d474
Author:    sherman
Date:      2010-04-15 17:00 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/c70d4266d474

6931676: TEST_BUG: wrong value for bugid in comments
Summary: update to the correct bugid
Reviewed-by: martin

! test/java/util/regex/RegExTest.java



From Ulf.Zibis at gmx.de  Fri Apr 16 00:23:39 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Fri, 16 Apr 2010 02:23:39 +0200
Subject: A few char-fiddling optimizations for Pattern.java
In-Reply-To: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>
References: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>
Message-ID: <4BC7AE0B.3080409@gmx.de>

>
> Hi Xueming and Ulf,
>
> Please review a few more optimizations to be added to the
> current crop:
>
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/Pattern-opt/
>
>    

Have you tried to disassemble? :

     private static final boolean isSupplementary(int cp) {
          returnCharacter.isSurrogate((char)cp)  ||
//  in this case method should be namedisSupplementaryOrHigher():
//                cp>= Character.MIN_SUPPLEMENTARY_CODE_POINT);
                  Character.isSupplementaryCodePoint(cp);

     }

Maybe HotSpot would compile to same code.
Anyway I think, such functionality should belong to class Character.
And don't forget the 8-space indentation rule for line continuation. ;-)


Use Character.codePointCount(...) instead ;-)  :

countCodePoints(CharSequence seq)


-Ulf









From martinrb at google.com  Fri Apr 16 00:52:52 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 15 Apr 2010 17:52:52 -0700
Subject: A few char-fiddling optimizations for Pattern.java
In-Reply-To: <4BC7AE0B.3080409@gmx.de>
References: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>
	<4BC7AE0B.3080409@gmx.de>
Message-ID: <g2y1ccfd1c11004151752lc9bb3de4w217422a78170f279@mail.gmail.com>

On Thu, Apr 15, 2010 at 17:23, Ulf Zibis <Ulf.Zibis at gmx.de> wrote:
>>
>> Hi Xueming and Ulf,
>>
>> Please review a few more optimizations to be added to the
>> current crop:
>>
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/Pattern-opt/
>>
>>
>
> Have you tried to disassemble? :

No.  Have you?

> ? ?private static final boolean isSupplementary(int cp) {
> ? ? ? ? returnCharacter.isSurrogate((char)cp) ?||
> // ?in this case method should be namedisSupplementaryOrHigher():
> // ? ? ? ? ? ? ? ?cp>= Character.MIN_SUPPLEMENTARY_CODE_POINT);
> ? ? ? ? ? ? ? ? Character.isSupplementaryCodePoint(cp);
>
> ? ?}
>
> Maybe HotSpot would compile to same code.

That seems unlikely to me.

> Anyway I think, such functionality should belong to class Character.

I disagree.

> And don't forget the 8-space indentation rule for line continuation. ;-)
>
>
> Use Character.codePointCount(...) instead ;-) ?:

Good point!  I deleted countCodePoints,
especially since it is not used in performance-critical code.

Webrev regenerated.

Martin

> countCodePoints(CharSequence seq)
>
>
> -Ulf
>
>
>
>
>
>
>
>


From weijun.wang at sun.com  Fri Apr 16 02:06:34 2010
From: weijun.wang at sun.com (weijun.wang at sun.com)
Date: Fri, 16 Apr 2010 02:06:34 +0000
Subject: hg: jdk7/tl/jdk: 6937978: let keytool -gencert generate the chain
Message-ID: <20100416020653.5534E44589@hg.openjdk.java.net>

Changeset: db4fd2fdf196
Author:    weijun
Date:      2010-04-16 10:06 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/db4fd2fdf196

6937978: let keytool -gencert generate the chain
Reviewed-by: mullan

! src/share/classes/sun/security/tools/KeyTool.java
! test/sun/security/tools/keytool/selfissued.sh



From weijun.wang at sun.com  Fri Apr 16 02:14:57 2010
From: weijun.wang at sun.com (weijun.wang at sun.com)
Date: Fri, 16 Apr 2010 02:14:57 +0000
Subject: hg: jdk7/tl/jdk: 6939248: Jarsigner can't extract Extended Key Usage
	from Timestamp Reply correctly
Message-ID: <20100416021516.29A684458A@hg.openjdk.java.net>

Changeset: 0d989dc383d3
Author:    weijun
Date:      2010-04-16 10:13 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0d989dc383d3

6939248: Jarsigner can't extract Extended Key Usage from Timestamp Reply correctly
Reviewed-by: xuelei, mullan

! src/share/classes/sun/security/tools/TimestampedSigner.java
+ test/sun/security/tools/jarsigner/TimestampCheck.java
+ test/sun/security/tools/jarsigner/ts.sh



From lordpixel+core-libs-dev at mac.com  Fri Apr 16 03:02:25 2010
From: lordpixel+core-libs-dev at mac.com (Andrew Thompson)
Date: Thu, 15 Apr 2010 23:02:25 -0400
Subject: Disambiguating empty catch blocks
In-Reply-To: <4BC59DD2.4060900@paradise.net.nz>
References: <4BC428D3.1060004@paradise.net.nz>
	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
	<4BC59DD2.4060900@paradise.net.nz>
Message-ID: <D69455C4-288D-4A27-A5FE-DF6263719256@mac.com>


On Apr 14, 2010, at 6:49 AM, Bruce Chapman wrote:

>> One thing we definitely can do is to add a new constructor
>> to AssertionError, that takes a cause as an argument,
>> consistent with other exceptions.
>> 
> Yeah, that was a minor niggle - I'll check  for an RFE and get the ball started on that.


Actually, it does exist, it is just disguised as taking Object:

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/AssertionError.html#AssertionError(java.lang.Object)

It you want to RFE, I guess

AssertionError(String additionalDetails, Throwable cause) 

is the missing method.


AndyT (lordpixel - the cat who walks through walls)
A little bigger on the inside

	(see you later space cowboy, you can't take the sky from me)




From martinrb at google.com  Fri Apr 16 06:09:50 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 15 Apr 2010 23:09:50 -0700
Subject: Bugs in java.util.ArrayList, java.util.Hashtable and 
	java.io.ByteArrayOutputStream
In-Reply-To: <4B9633EA.8070101@sun.com>
References: <1704b7a21003031741m734545f1gb0170ed5fa6f6d68@mail.gmail.com>
	<1ccfd1c11003050104u61e776apc5fe2e5ec08e3dc0@mail.gmail.com>
	<1704b7a21003050248k1e893cedmd14f26cbecd45896@mail.gmail.com>
	<1ccfd1c11003081810u54fb22e6k25230f4eb5ca1b18@mail.gmail.com>
	<4B9633EA.8070101@sun.com>
Message-ID: <l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>

Hi Chris,

I recently discovered another place to handle huge arrays better - in
AbstractCollection.
I've put those changes into
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize2/
I propose to qfold these into the original changes for this bug
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize/
which have not yet been committed.

Martin

On Tue, Mar 9, 2010 at 04:41, Christopher Hegarty -Sun Microsystems
Ireland <Christopher.Hegarty at sun.com> wrote:
> Sorry Martin, I appear to have missed your original request to file this
> bug. I since filed the following:
>
> ?6933217: Huge arrays handled poorly in core libraries
>
> The changes you are proposing seem reasonable to me.
>
> -Chris.
>
> Martin Buchholz wrote:
>>
>> [Chris or Alan, please review and file a bug]
>>
>> OK, guys,
>>
>> Here's a patch:
>>
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize/
>>
>> Martin
>>
>> On Fri, Mar 5, 2010 at 02:48, Kevin L. Stern <kevin.l.stern at gmail.com>
>> wrote:
>>>
>>> Hi Martin,
>>>
>>> Thank you for your reply. ?If I may, PriorityQueue appears to employ the
>>> simple strategy that I suggested above in its grow method:
>>>
>>> ? ? ? ?int newCapacity = ((oldCapacity < 64)?
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ((oldCapacity + 1) * 2):
>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ((oldCapacity / 2) * 3));
>>> ? ? ? ?if (newCapacity < 0) // overflow
>>> ? ? ? ? ? ?newCapacity = Integer.MAX_VALUE;
>>>
>>> It might be desirable to set a common strategy for capacity increase for
>>> all
>>> collections.
>>>
>>> Regards,
>>>
>>> Kevin
>>>
>>> On Fri, Mar 5, 2010 at 3:04 AM, Martin Buchholz <martinrb at google.com>
>>> wrote:
>>>>
>>>> Hi Kevin,
>>>>
>>>> As you've noticed, creating objects within a factor of two of
>>>> their natural limits is a good way to expose lurking bugs.
>>>>
>>>> I'm the one responsible for the algorithm in ArrayList.
>>>> I'm a bit embarrassed, looking at that code today.
>>>> We could set the array size to Integer.MAX_VALUE,
>>>> but then you might hit an independent buglet in hotspot
>>>> that you cannot allocate an array with Integer.MAX_VALUE
>>>> elements, but Integer.MAX_VALUE - 5 (or so) works.
>>>>
>>>> It occurs to me that increasing the size by 50% is better done by
>>>> int newCapacity = oldCapacity + (oldCapacity >> 1) + 1;
>>>>
>>>> I agree with the plan of setting the capacity to something near
>>>> MAX_VALUE on overflow, and throw OutOfMemoryError on next resize.
>>>>
>>>> These bugs are not known.
>>>> Chris Hegarty, could you file a bug for us?
>>>>
>>>> Martin
>>>>
>>>> On Wed, Mar 3, 2010 at 17:41, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>> wrote:
>>>>>
>>>>> Greetings,
>>>>>
>>>>> I've noticed bugs in java.util.ArrayList, java.util.Hashtable and
>>>>> java.io.ByteArrayOutputStream which arise when the capacities of the
>>>>> data
>>>>> structures reach a particular threshold. ?More below.
>>>>>
>>>>> When the capacity of an ArrayList reaches (2/3)*Integer.MAX_VALUE its
>>>>> size
>>>>> reaches its capacity and an add or an insert operation is invoked, the
>>>>> capacity is increased by only one element. ?Notice that in the
>>>>> following
>>>>> excerpt from ArrayList.ensureCapacity the new capacity is set to (3/2)
>>>>> *
>>>>> oldCapacity + 1 unless this value would not suffice to accommodate the
>>>>> required capacity in which case it is set to the required capacity. ?If
>>>>> the
>>>>> current capacity is at least (2/3)*Integer.MAX_VALUE, then (oldCapacity
>>>>> *
>>>>> 3)/2 + 1 overflows and resolves to a negative number resulting in the
>>>>> new
>>>>> capacity being set to the required capacity. ?The major consequence of
>>>>> this
>>>>> is that each subsequent add/insert operation results in a full resize
>>>>> of
>>>>> the
>>>>> ArrayList causing performance to degrade significantly.
>>>>>
>>>>> ? ? ? ?int newCapacity = (oldCapacity * 3)/2 + 1;
>>>>> ? ? ? ? ? ?if (newCapacity < minCapacity)
>>>>> ? ? ? ?newCapacity = minCapacity;
>>>>>
>>>>> Hashtable breaks entirely when the size of its backing array reaches
>>>>> (1/2) *
>>>>> Integer.MAX_VALUE and a rehash is necessary as is evident from the
>>>>> following
>>>>> excerpt from rehash. ?Notice that rehash will attempt to create an
>>>>> array
>>>>> of
>>>>> negative size if the size of the backing array reaches (1/2) *
>>>>> Integer.MAX_VALUE since oldCapacity * 2 + 1 overflows and resolves to a
>>>>> negative number.
>>>>>
>>>>> ? ?int newCapacity = oldCapacity * 2 + 1;
>>>>> ? ?HashtableEntry newTable[] = new HashtableEntry[newCapacity];
>>>>>
>>>>> When the capacity of the backing array in a ByteArrayOutputStream
>>>>> reaches
>>>>> (1/2) * Integer.MAX_VALUE its size reaches its capacity and a write
>>>>> operation is invoked, the capacity of the backing array is increased
>>>>> only by
>>>>> the required number of elements. ?Notice that in the following excerpt
>>>>> from
>>>>> ByteArrayOutputStream.write(int) the new backing array capacity is set
>>>>> to 2
>>>>> * buf.length unless this value would not suffice to accommodate the
>>>>> required
>>>>> capacity in which case it is set to the required capacity. ?If the
>>>>> current
>>>>> backing array capacity is at least (1/2) * Integer.MAX_VALUE + 1, then
>>>>> buf.length << 1 overflows and resolves to a negative number resulting
>>>>> in
>>>>> the
>>>>> new capacity being set to the required capacity. ?The major consequence
>>>>> of
>>>>> this, like with ArrayList, is that each subsequent write operation
>>>>> results
>>>>> in a full resize of the ByteArrayOutputStream causing performance to
>>>>> degrade
>>>>> significantly.
>>>>>
>>>>> ? ?int newcount = count + 1;
>>>>> ? ?if (newcount > buf.length) {
>>>>> ? ? ? ? ? ?buf = Arrays.copyOf(buf, Math.max(buf.length << 1,
>>>>> newcount));
>>>>> ? ?}
>>>>>
>>>>> It is interesting to note that any statements about the amortized time
>>>>> complexity of add/insert operations, such as the one in the ArrayList
>>>>> javadoc, are invalidated by the performance related bugs. ?One solution
>>>>> to
>>>>> the above situations is to set the new capacity of the backing array to
>>>>> Integer.MAX_VALUE when the initial size calculation results in a
>>>>> negative
>>>>> number during a resize.
>>>>>
>>>>> Apologies if these bugs are already known.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Kevin
>>>>>
>>>
>


From martinrb at google.com  Fri Apr 16 06:33:53 2010
From: martinrb at google.com (Martin Buchholz)
Date: Thu, 15 Apr 2010 23:33:53 -0700
Subject: UNIXProcess improvements
Message-ID: <l2m1ccfd1c11004152333rfae23b4fgd77d60bc3ef4d353@mail.gmail.com>

Chris and/or Alan,
I am planning some improvements to Process.
Please file a bug.

Synopsis: Improvements to subprocess handling on Unix
Description:
Looking at UNIXProcess.java.linux, there are a number of possible improvements:

The "process reaper" thread should run in a thread pool to save on
thread creation in case of repeated process creation.  Its stack size
should be small.

If the process reaper thread throws a non-IOException (e.g. OOME)
the thread creating the subprocess will hang,
waiting for a signal from the reaper thread.

The JDK knows when the subprocess terminates.
At that time, we cannot yet close the streams,
because there may still be unread buffered data, but...
the file descriptors underlying the subprocess streams
can be drained, then closed, and the drained bytes can
be added to the internal buffer.  This will solve most cases
of (pandemic) poor file stream hygiene leading to
file descriptor exhaustion or simply poor performance.

Preliminary fix here:
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/


From Ulf.Zibis at gmx.de  Fri Apr 16 08:48:04 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Fri, 16 Apr 2010 10:48:04 +0200
Subject: A few char-fiddling optimizations for Pattern.java
In-Reply-To: <g2y1ccfd1c11004151752lc9bb3de4w217422a78170f279@mail.gmail.com>
References: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>	
	<4BC7AE0B.3080409@gmx.de>
	<g2y1ccfd1c11004151752lc9bb3de4w217422a78170f279@mail.gmail.com>
Message-ID: <4BC82444.4080507@gmx.de>

Am 16.04.2010 02:52, schrieb Martin Buchholz:
> On Thu, Apr 15, 2010 at 17:23, Ulf Zibis<Ulf.Zibis at gmx.de>  wrote:
>    
>>> Hi Xueming and Ulf,
>>>
>>> Please review a few more optimizations to be added to the
>>> current crop:
>>>
>>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/Pattern-opt/
>>>
>>>
>>>        
>> Have you tried to disassemble? :
>>      
> No.  Have you?
>    

No.
Can you write a main()-loop, which would trigger the isSurrogate-case 
enough, so I can examine the disassembly.
I've not enough knowledge about Pattern.java.

On a 2nd thought, I think a code point in the surrogate range should be 
rare, as it is kinda invalid. So why check it in 1. place.
What is the purpose of the method?
- isNotValidOrSurrogateBMPCodePoint --> includes negative and values > 
MAX_SUPPLEMENTARY_CODE_POINT
- needsSurrogateTreatment --> only includes surrogate and valid 
supplementary code points
For the 1st case I would code:
     return !Character.isBMPCodePoint(cp) || Character.isSurrogate((char)cp)

>>     private static final boolean isSupplementary(int cp) {
>>          returnCharacter.isSurrogate((char)cp)  ||
>> //  in this case method should be namedisSupplementaryOrHigher():
>> //                cp>= Character.MIN_SUPPLEMENTARY_CODE_POINT);
>>                  Character.isSupplementaryCodePoint(cp);
>>
>>     }
>>
>> Maybe HotSpot would compile to same code.
>>      
> That seems unlikely to me.
>    

When HotSpot examines cp to be less than MIN_SURROGATE, why should it 
later examine cp >= MIN_SUPPLEMENTARY_CODE_POINT ?

>> Anyway I think, such functionality should belong to class Character.
>>      
> I disagree.
>    

I'm thinking about:
     boolean isBMPCodePoint(int cp, boolean validateSurrogate) {...}

>> And don't forget the 8-space indentation rule for line continuation. ;-)
>>
>>
>> Use Character.codePointCount(...) instead ;-)  :
>>      
> Good point!  I deleted countCodePoints,
> especially since it is not used in performance-critical code.
>    

I was wondering, as that was you who had optimized 
Character.codePointCount(...) some days ago.
So why rewriting it on another place.
If countCodePoints has enough call sites, convenient method could reduce 
byte code size but decreases interpreter performance:
     private int countCodePoints(CharSequence chars) {
         return Character.codePointCount(chars, 0, chars.length());
     }

-Ulf




From ahughes at redhat.com  Fri Apr 16 09:14:11 2010
From: ahughes at redhat.com (ahughes at redhat.com)
Date: Fri, 16 Apr 2010 09:14:11 +0000
Subject: hg: jdk7/tl/jdk: 6944361: Missing CKR_ values in PKCS11Exception
Message-ID: <20100416091430.498054459C@hg.openjdk.java.net>

Changeset: c444651077d2
Author:    andrew
Date:      2010-04-16 09:54 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/c444651077d2

6944361: Missing CKR_ values in PKCS11Exception
Summary: Allow native NSS errors to be observed and correctly reported
Reviewed-by: wetmore, valeriep

! src/share/classes/sun/security/pkcs11/wrapper/PKCS11Exception.java
! src/share/classes/sun/security/x509/X509Key.java



From Ulf.Zibis at gmx.de  Fri Apr 16 09:38:58 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Fri, 16 Apr 2010 11:38:58 +0200
Subject: A few char-fiddling optimizations for Pattern.java
In-Reply-To: <4BC82444.4080507@gmx.de>
References: <r2z1ccfd1c11004151541j66421897g483932f676b7c60@mail.gmail.com>		<4BC7AE0B.3080409@gmx.de>	<g2y1ccfd1c11004151752lc9bb3de4w217422a78170f279@mail.gmail.com>
	<4BC82444.4080507@gmx.de>
Message-ID: <4BC83032.40507@gmx.de>

Am 16.04.2010 10:48, schrieb Ulf Zibis:
> On a 2nd thought, I think a code point in the surrogate range should 
> be rare, as it is kinda invalid. So why check it in 1. place.
> What is the purpose of the method?
> - isNotValidOrSurrogateBMPCodePoint --> includes negative and values > 
> MAX_SUPPLEMENTARY_CODE_POINT
> - needsSurrogateTreatment --> only includes surrogate and valid 
> supplementary code points

legacy isSupplementary() --> excludes negatives, but includes values > 
MAX_SUPPLEMENTARY_CODE_POINT

>
> I'm thinking about:
>     boolean isBMPCodePoint(int cp, boolean validateSurrogate) {...}

Thinking again, I would drop  isBMPCodePoint(int cp) against:

     boolean isBMPCodePoint(int cp, boolean validateSurrogate) {
         return (cp >>> 16) == 0 && !(validateSurrogate && 
isSurrogate((char)cp));
     }

It would serve more complete the possible use cases.
It would inline to (cp >>> 16) == 0, if validateSurrogate == false.

-Ulf




From chris.hegarty at oracle.com  Fri Apr 16 09:38:07 2010
From: chris.hegarty at oracle.com (Chris Hegarty)
Date: Fri, 16 Apr 2010 10:38:07 +0100
Subject: Bugs in java.util.ArrayList,
	java.util.Hashtable and  java.io.ByteArrayOutputStream
In-Reply-To: <l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>
References: <1704b7a21003031741m734545f1gb0170ed5fa6f6d68@mail.gmail.com>
	<1ccfd1c11003050104u61e776apc5fe2e5ec08e3dc0@mail.gmail.com>
	<1704b7a21003050248k1e893cedmd14f26cbecd45896@mail.gmail.com>
	<1ccfd1c11003081810u54fb22e6k25230f4eb5ca1b18@mail.gmail.com>
	<4B9633EA.8070101@sun.com>
	<l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>
Message-ID: <4BC82FFF.20502@oracle.com>


Martin Buchholz wrote:
> Hi Chris,
> 
> I recently discovered another place to handle huge arrays better - in
> AbstractCollection.
> I've put those changes into
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize2/
> I propose to qfold these into the original changes for this bug
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize/
> which have not yet been committed.

Good catch Martin, these additional changes look good.

-Chris.

> 
> Martin
> 
> On Tue, Mar 9, 2010 at 04:41, Christopher Hegarty -Sun Microsystems
> Ireland <Christopher.Hegarty at sun.com> wrote:
>> Sorry Martin, I appear to have missed your original request to file this
>> bug. I since filed the following:
>>
>>  6933217: Huge arrays handled poorly in core libraries
>>
>> The changes you are proposing seem reasonable to me.
>>
>> -Chris.
>>
>> Martin Buchholz wrote:
>>> [Chris or Alan, please review and file a bug]
>>>
>>> OK, guys,
>>>
>>> Here's a patch:
>>>
>>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize/
>>>
>>> Martin
>>>
>>> On Fri, Mar 5, 2010 at 02:48, Kevin L. Stern <kevin.l.stern at gmail.com>
>>> wrote:
>>>> Hi Martin,
>>>>
>>>> Thank you for your reply.  If I may, PriorityQueue appears to employ the
>>>> simple strategy that I suggested above in its grow method:
>>>>
>>>>        int newCapacity = ((oldCapacity < 64)?
>>>>                           ((oldCapacity + 1) * 2):
>>>>                           ((oldCapacity / 2) * 3));
>>>>        if (newCapacity < 0) // overflow
>>>>            newCapacity = Integer.MAX_VALUE;
>>>>
>>>> It might be desirable to set a common strategy for capacity increase for
>>>> all
>>>> collections.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>> On Fri, Mar 5, 2010 at 3:04 AM, Martin Buchholz <martinrb at google.com>
>>>> wrote:
>>>>> Hi Kevin,
>>>>>
>>>>> As you've noticed, creating objects within a factor of two of
>>>>> their natural limits is a good way to expose lurking bugs.
>>>>>
>>>>> I'm the one responsible for the algorithm in ArrayList.
>>>>> I'm a bit embarrassed, looking at that code today.
>>>>> We could set the array size to Integer.MAX_VALUE,
>>>>> but then you might hit an independent buglet in hotspot
>>>>> that you cannot allocate an array with Integer.MAX_VALUE
>>>>> elements, but Integer.MAX_VALUE - 5 (or so) works.
>>>>>
>>>>> It occurs to me that increasing the size by 50% is better done by
>>>>> int newCapacity = oldCapacity + (oldCapacity >> 1) + 1;
>>>>>
>>>>> I agree with the plan of setting the capacity to something near
>>>>> MAX_VALUE on overflow, and throw OutOfMemoryError on next resize.
>>>>>
>>>>> These bugs are not known.
>>>>> Chris Hegarty, could you file a bug for us?
>>>>>
>>>>> Martin
>>>>>
>>>>> On Wed, Mar 3, 2010 at 17:41, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>> wrote:
>>>>>> Greetings,
>>>>>>
>>>>>> I've noticed bugs in java.util.ArrayList, java.util.Hashtable and
>>>>>> java.io.ByteArrayOutputStream which arise when the capacities of the
>>>>>> data
>>>>>> structures reach a particular threshold.  More below.
>>>>>>
>>>>>> When the capacity of an ArrayList reaches (2/3)*Integer.MAX_VALUE its
>>>>>> size
>>>>>> reaches its capacity and an add or an insert operation is invoked, the
>>>>>> capacity is increased by only one element.  Notice that in the
>>>>>> following
>>>>>> excerpt from ArrayList.ensureCapacity the new capacity is set to (3/2)
>>>>>> *
>>>>>> oldCapacity + 1 unless this value would not suffice to accommodate the
>>>>>> required capacity in which case it is set to the required capacity.  If
>>>>>> the
>>>>>> current capacity is at least (2/3)*Integer.MAX_VALUE, then (oldCapacity
>>>>>> *
>>>>>> 3)/2 + 1 overflows and resolves to a negative number resulting in the
>>>>>> new
>>>>>> capacity being set to the required capacity.  The major consequence of
>>>>>> this
>>>>>> is that each subsequent add/insert operation results in a full resize
>>>>>> of
>>>>>> the
>>>>>> ArrayList causing performance to degrade significantly.
>>>>>>
>>>>>>        int newCapacity = (oldCapacity * 3)/2 + 1;
>>>>>>            if (newCapacity < minCapacity)
>>>>>>        newCapacity = minCapacity;
>>>>>>
>>>>>> Hashtable breaks entirely when the size of its backing array reaches
>>>>>> (1/2) *
>>>>>> Integer.MAX_VALUE and a rehash is necessary as is evident from the
>>>>>> following
>>>>>> excerpt from rehash.  Notice that rehash will attempt to create an
>>>>>> array
>>>>>> of
>>>>>> negative size if the size of the backing array reaches (1/2) *
>>>>>> Integer.MAX_VALUE since oldCapacity * 2 + 1 overflows and resolves to a
>>>>>> negative number.
>>>>>>
>>>>>>    int newCapacity = oldCapacity * 2 + 1;
>>>>>>    HashtableEntry newTable[] = new HashtableEntry[newCapacity];
>>>>>>
>>>>>> When the capacity of the backing array in a ByteArrayOutputStream
>>>>>> reaches
>>>>>> (1/2) * Integer.MAX_VALUE its size reaches its capacity and a write
>>>>>> operation is invoked, the capacity of the backing array is increased
>>>>>> only by
>>>>>> the required number of elements.  Notice that in the following excerpt
>>>>>> from
>>>>>> ByteArrayOutputStream.write(int) the new backing array capacity is set
>>>>>> to 2
>>>>>> * buf.length unless this value would not suffice to accommodate the
>>>>>> required
>>>>>> capacity in which case it is set to the required capacity.  If the
>>>>>> current
>>>>>> backing array capacity is at least (1/2) * Integer.MAX_VALUE + 1, then
>>>>>> buf.length << 1 overflows and resolves to a negative number resulting
>>>>>> in
>>>>>> the
>>>>>> new capacity being set to the required capacity.  The major consequence
>>>>>> of
>>>>>> this, like with ArrayList, is that each subsequent write operation
>>>>>> results
>>>>>> in a full resize of the ByteArrayOutputStream causing performance to
>>>>>> degrade
>>>>>> significantly.
>>>>>>
>>>>>>    int newcount = count + 1;
>>>>>>    if (newcount > buf.length) {
>>>>>>            buf = Arrays.copyOf(buf, Math.max(buf.length << 1,
>>>>>> newcount));
>>>>>>    }
>>>>>>
>>>>>> It is interesting to note that any statements about the amortized time
>>>>>> complexity of add/insert operations, such as the one in the ArrayList
>>>>>> javadoc, are invalidated by the performance related bugs.  One solution
>>>>>> to
>>>>>> the above situations is to set the new capacity of the backing array to
>>>>>> Integer.MAX_VALUE when the initial size calculation results in a
>>>>>> negative
>>>>>> number during a resize.
>>>>>>
>>>>>> Apologies if these bugs are already known.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>


From David.Holmes at oracle.com  Fri Apr 16 10:17:26 2010
From: David.Holmes at oracle.com (David Holmes)
Date: Fri, 16 Apr 2010 20:17:26 +1000
Subject: Bugs in java.util.ArrayList,
	java.util.Hashtable and  java.io.ByteArrayOutputStream
In-Reply-To: <l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>
References: <1704b7a21003031741m734545f1gb0170ed5fa6f6d68@mail.gmail.com>	<1ccfd1c11003050104u61e776apc5fe2e5ec08e3dc0@mail.gmail.com>	<1704b7a21003050248k1e893cedmd14f26cbecd45896@mail.gmail.com>	<1ccfd1c11003081810u54fb22e6k25230f4eb5ca1b18@mail.gmail.com>	<4B9633EA.8070101@sun.com>
	<l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>
Message-ID: <4BC83936.6090106@oracle.com>

Hi Martin,

Martin Buchholz said the following on 04/16/10 16:09:
> I recently discovered another place to handle huge arrays better - in
> AbstractCollection.
> I've put those changes into
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize2/

I don't understand what you are doing here:

194      * The maximum size of array to allocate.
195      * Some VMs reserve some header words in an array.
196      * Attempts to allocate larger arrays may result in
197      * OutOfMemoryError: Requested array size exceeds VM limit
198      */
199     private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

On 32-bit you can't get anywhere near this limit. On 64-bit Hotspot 
currently allows you to allocate the entire array up to 
Integer.MAX_VALUE elements for types <= 32-bit; and MAX_VALUE-3 for 64-bit.

Regardless this is VM specific and I don't see what purpose this serves.
???

David

> Martin
> 
> On Tue, Mar 9, 2010 at 04:41, Christopher Hegarty -Sun Microsystems
> Ireland <Christopher.Hegarty at sun.com> wrote:
>> Sorry Martin, I appear to have missed your original request to file this
>> bug. I since filed the following:
>>
>>  6933217: Huge arrays handled poorly in core libraries
>>
>> The changes you are proposing seem reasonable to me.
>>
>> -Chris.
>>
>> Martin Buchholz wrote:
>>> [Chris or Alan, please review and file a bug]
>>>
>>> OK, guys,
>>>
>>> Here's a patch:
>>>
>>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize/
>>>
>>> Martin
>>>
>>> On Fri, Mar 5, 2010 at 02:48, Kevin L. Stern <kevin.l.stern at gmail.com>
>>> wrote:
>>>> Hi Martin,
>>>>
>>>> Thank you for your reply.  If I may, PriorityQueue appears to employ the
>>>> simple strategy that I suggested above in its grow method:
>>>>
>>>>        int newCapacity = ((oldCapacity < 64)?
>>>>                           ((oldCapacity + 1) * 2):
>>>>                           ((oldCapacity / 2) * 3));
>>>>        if (newCapacity < 0) // overflow
>>>>            newCapacity = Integer.MAX_VALUE;
>>>>
>>>> It might be desirable to set a common strategy for capacity increase for
>>>> all
>>>> collections.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>> On Fri, Mar 5, 2010 at 3:04 AM, Martin Buchholz <martinrb at google.com>
>>>> wrote:
>>>>> Hi Kevin,
>>>>>
>>>>> As you've noticed, creating objects within a factor of two of
>>>>> their natural limits is a good way to expose lurking bugs.
>>>>>
>>>>> I'm the one responsible for the algorithm in ArrayList.
>>>>> I'm a bit embarrassed, looking at that code today.
>>>>> We could set the array size to Integer.MAX_VALUE,
>>>>> but then you might hit an independent buglet in hotspot
>>>>> that you cannot allocate an array with Integer.MAX_VALUE
>>>>> elements, but Integer.MAX_VALUE - 5 (or so) works.
>>>>>
>>>>> It occurs to me that increasing the size by 50% is better done by
>>>>> int newCapacity = oldCapacity + (oldCapacity >> 1) + 1;
>>>>>
>>>>> I agree with the plan of setting the capacity to something near
>>>>> MAX_VALUE on overflow, and throw OutOfMemoryError on next resize.
>>>>>
>>>>> These bugs are not known.
>>>>> Chris Hegarty, could you file a bug for us?
>>>>>
>>>>> Martin
>>>>>
>>>>> On Wed, Mar 3, 2010 at 17:41, Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>> wrote:
>>>>>> Greetings,
>>>>>>
>>>>>> I've noticed bugs in java.util.ArrayList, java.util.Hashtable and
>>>>>> java.io.ByteArrayOutputStream which arise when the capacities of the
>>>>>> data
>>>>>> structures reach a particular threshold.  More below.
>>>>>>
>>>>>> When the capacity of an ArrayList reaches (2/3)*Integer.MAX_VALUE its
>>>>>> size
>>>>>> reaches its capacity and an add or an insert operation is invoked, the
>>>>>> capacity is increased by only one element.  Notice that in the
>>>>>> following
>>>>>> excerpt from ArrayList.ensureCapacity the new capacity is set to (3/2)
>>>>>> *
>>>>>> oldCapacity + 1 unless this value would not suffice to accommodate the
>>>>>> required capacity in which case it is set to the required capacity.  If
>>>>>> the
>>>>>> current capacity is at least (2/3)*Integer.MAX_VALUE, then (oldCapacity
>>>>>> *
>>>>>> 3)/2 + 1 overflows and resolves to a negative number resulting in the
>>>>>> new
>>>>>> capacity being set to the required capacity.  The major consequence of
>>>>>> this
>>>>>> is that each subsequent add/insert operation results in a full resize
>>>>>> of
>>>>>> the
>>>>>> ArrayList causing performance to degrade significantly.
>>>>>>
>>>>>>        int newCapacity = (oldCapacity * 3)/2 + 1;
>>>>>>            if (newCapacity < minCapacity)
>>>>>>        newCapacity = minCapacity;
>>>>>>
>>>>>> Hashtable breaks entirely when the size of its backing array reaches
>>>>>> (1/2) *
>>>>>> Integer.MAX_VALUE and a rehash is necessary as is evident from the
>>>>>> following
>>>>>> excerpt from rehash.  Notice that rehash will attempt to create an
>>>>>> array
>>>>>> of
>>>>>> negative size if the size of the backing array reaches (1/2) *
>>>>>> Integer.MAX_VALUE since oldCapacity * 2 + 1 overflows and resolves to a
>>>>>> negative number.
>>>>>>
>>>>>>    int newCapacity = oldCapacity * 2 + 1;
>>>>>>    HashtableEntry newTable[] = new HashtableEntry[newCapacity];
>>>>>>
>>>>>> When the capacity of the backing array in a ByteArrayOutputStream
>>>>>> reaches
>>>>>> (1/2) * Integer.MAX_VALUE its size reaches its capacity and a write
>>>>>> operation is invoked, the capacity of the backing array is increased
>>>>>> only by
>>>>>> the required number of elements.  Notice that in the following excerpt
>>>>>> from
>>>>>> ByteArrayOutputStream.write(int) the new backing array capacity is set
>>>>>> to 2
>>>>>> * buf.length unless this value would not suffice to accommodate the
>>>>>> required
>>>>>> capacity in which case it is set to the required capacity.  If the
>>>>>> current
>>>>>> backing array capacity is at least (1/2) * Integer.MAX_VALUE + 1, then
>>>>>> buf.length << 1 overflows and resolves to a negative number resulting
>>>>>> in
>>>>>> the
>>>>>> new capacity being set to the required capacity.  The major consequence
>>>>>> of
>>>>>> this, like with ArrayList, is that each subsequent write operation
>>>>>> results
>>>>>> in a full resize of the ByteArrayOutputStream causing performance to
>>>>>> degrade
>>>>>> significantly.
>>>>>>
>>>>>>    int newcount = count + 1;
>>>>>>    if (newcount > buf.length) {
>>>>>>            buf = Arrays.copyOf(buf, Math.max(buf.length << 1,
>>>>>> newcount));
>>>>>>    }
>>>>>>
>>>>>> It is interesting to note that any statements about the amortized time
>>>>>> complexity of add/insert operations, such as the one in the ArrayList
>>>>>> javadoc, are invalidated by the performance related bugs.  One solution
>>>>>> to
>>>>>> the above situations is to set the new capacity of the backing array to
>>>>>> Integer.MAX_VALUE when the initial size calculation results in a
>>>>>> negative
>>>>>> number during a resize.
>>>>>>
>>>>>> Apologies if these bugs are already known.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>


From Alan.Bateman at Sun.COM  Fri Apr 16 14:54:55 2010
From: Alan.Bateman at Sun.COM (Alan Bateman)
Date: Fri, 16 Apr 2010 15:54:55 +0100
Subject: Need reviewer - Change to jdk/test/makefile for shared library
	execute permissions
In-Reply-To: <95C9525A-2AEC-48D3-BBAE-50CCB734182A@oracle.com>
References: <95C9525A-2AEC-48D3-BBAE-50CCB734182A@oracle.com>
Message-ID: <4BC87A3F.7070305@sun.com>

Kelly O'Hair wrote:
> Need reviewer - Change to jdk/test/makefile for shared library execute 
> permissions
>
> We been having some spurious test failures on windows and finally 
> tracked it down to this
> issue. Fixing it in the jdk/test/Makefile seemed like the best place, 
> even though none of us
> really want shared libraries in the repositories, this seems to be the 
> best fix.
>
> This allows more tests to reliably run, although the shared libraries 
> do need to exist
> for the test to pass, so anyone removing these binary files should of 
> course not run
> the associated tests.
>
> 6943915: Adjust jdk/test/Makefile to deal with .dll and .so libraries 
> needing execute permissions
> http://cr.openjdk.java.net/~ohair/openjdk7/testMakefile-6943915/webrev/
>
> I'll be testing this on all platforms tonight.
>
> -kto
>
This looks a reasonable solution for now.

-Alan.


From chris.hegarty at oracle.com  Fri Apr 16 16:00:34 2010
From: chris.hegarty at oracle.com (Chris Hegarty)
Date: Fri, 16 Apr 2010 17:00:34 +0100
Subject: UNIXProcess improvements
In-Reply-To: <l2m1ccfd1c11004152333rfae23b4fgd77d60bc3ef4d353@mail.gmail.com>
References: <l2m1ccfd1c11004152333rfae23b4fgd77d60bc3ef4d353@mail.gmail.com>
Message-ID: <4BC889A2.6090401@oracle.com>

Martin,

I filed the following to track this:
   6944584: Improvements to subprocess handling on Unix

I took a look at the preliminary changes.
  * I'm not sure we can remove any previous authors names.
  * I assume the second part, buffering of data after process
    termination has not been implemented yet, right?

Otherwise look good.

Michael:
   You may want to take a look at these changes also.

-Chris.

Martin Buchholz wrote:
> Chris and/or Alan,
> I am planning some improvements to Process.
> Please file a bug.
> 
> Synopsis: Improvements to subprocess handling on Unix
> Description:
> Looking at UNIXProcess.java.linux, there are a number of possible improvements:
> 
> The "process reaper" thread should run in a thread pool to save on
> thread creation in case of repeated process creation.  Its stack size
> should be small.
> 
> If the process reaper thread throws a non-IOException (e.g. OOME)
> the thread creating the subprocess will hang,
> waiting for a signal from the reaper thread.
> 
> The JDK knows when the subprocess terminates.
> At that time, we cannot yet close the streams,
> because there may still be unread buffered data, but...
> the file descriptors underlying the subprocess streams
> can be drained, then closed, and the drained bytes can
> be added to the internal buffer.  This will solve most cases
> of (pandemic) poor file stream hygiene leading to
> file descriptor exhaustion or simply poor performance.
> 
> Preliminary fix here:
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/


From martinrb at google.com  Fri Apr 16 16:09:31 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 16 Apr 2010 09:09:31 -0700
Subject: UNIXProcess improvements
In-Reply-To: <4BC889A2.6090401@oracle.com>
References: <l2m1ccfd1c11004152333rfae23b4fgd77d60bc3ef4d353@mail.gmail.com>
	<4BC889A2.6090401@oracle.com>
Message-ID: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>

I should have explicitly invited Michael and Xueming to review, as well.

On Fri, Apr 16, 2010 at 09:00, Chris Hegarty <chris.hegarty at oracle.com> wrote:
> Martin,
>
> I filed the following to track this:
> ?6944584: Improvements to subprocess handling on Unix

Thanks.

> I took a look at the preliminary changes.
> ?* I'm not sure we can remove any previous authors names.

We've considered doing that en masse.

I think it's only old copyrights that can never be removed.

> ?* I assume the second part, buffering of data after process
> ? termination has not been implemented yet, right?

Right.

> Otherwise look good.
>
> Michael:
> ?You may want to take a look at these changes also.
>
> -Chris.
>
> Martin Buchholz wrote:
>>
>> Chris and/or Alan,
>> I am planning some improvements to Process.
>> Please file a bug.
>>
>> Synopsis: Improvements to subprocess handling on Unix
>> Description:
>> Looking at UNIXProcess.java.linux, there are a number of possible
>> improvements:
>>
>> The "process reaper" thread should run in a thread pool to save on
>> thread creation in case of repeated process creation. ?Its stack size
>> should be small.
>>
>> If the process reaper thread throws a non-IOException (e.g. OOME)
>> the thread creating the subprocess will hang,
>> waiting for a signal from the reaper thread.
>>
>> The JDK knows when the subprocess terminates.
>> At that time, we cannot yet close the streams,
>> because there may still be unread buffered data, but...
>> the file descriptors underlying the subprocess streams
>> can be drained, then closed, and the drained bytes can
>> be added to the internal buffer. ?This will solve most cases
>> of (pandemic) poor file stream hygiene leading to
>> file descriptor exhaustion or simply poor performance.
>>
>> Preliminary fix here:
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/
>


From mr at sun.com  Fri Apr 16 16:18:31 2010
From: mr at sun.com (Mark Reinhold)
Date: Fri, 16 Apr 2010 09:18:31 -0700
Subject: UNIXProcess improvements 
In-Reply-To: martinrb@google.com; Fri, 16 Apr 2010 09:09:31 PDT;
	<v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com> 
Message-ID: <20100416161831.C21CE511@eggemoggin.niobe.net>

> Date: Fri, 16 Apr 2010 09:09:31 -0700
> From: Martin Buchholz <martinrb at google.com>

> On Fri, Apr 16, 2010 at 09:00, chris.hegarty at oracle.com wrote:
>> I took a look at the preliminary changes.
>> ?* I'm not sure we can remove any previous authors names.
> 
> We've considered doing that en masse.

Yes, and we might consider that again.

> I think it's only old copyrights that can never be removed.

Correct -- old copyright can never, ever be removed.

For now I suggest leaving old @author tags as-is.

- Mark


From martinrb at google.com  Fri Apr 16 17:01:09 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 16 Apr 2010 10:01:09 -0700
Subject: UNIXProcess improvements
In-Reply-To: <20100416161831.C21CE511@eggemoggin.niobe.net>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
Message-ID: <r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>

On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:

> For now I suggest leaving old @author tags as-is.

OK, done.

Version 0.2 of the webrev is published.

Martin


From martinrb at google.com  Fri Apr 16 19:03:33 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 16 Apr 2010 12:03:33 -0700
Subject: Bugs in java.util.ArrayList, java.util.Hashtable and 
	java.io.ByteArrayOutputStream
In-Reply-To: <4BC83936.6090106@oracle.com>
References: <1704b7a21003031741m734545f1gb0170ed5fa6f6d68@mail.gmail.com>
	<1ccfd1c11003050104u61e776apc5fe2e5ec08e3dc0@mail.gmail.com>
	<1704b7a21003050248k1e893cedmd14f26cbecd45896@mail.gmail.com>
	<1ccfd1c11003081810u54fb22e6k25230f4eb5ca1b18@mail.gmail.com>
	<4B9633EA.8070101@sun.com>
	<l2m1ccfd1c11004152309g23f3e239k1cbec0a93d48aef7@mail.gmail.com>
	<4BC83936.6090106@oracle.com>
Message-ID: <h2z1ccfd1c11004161203r5cf8f848w95816ff92c71ee4b@mail.gmail.com>

On Fri, Apr 16, 2010 at 03:17, David Holmes <David.Holmes at oracle.com> wrote:
> Hi Martin,
>
> Martin Buchholz said the following on 04/16/10 16:09:
>>
>> I recently discovered another place to handle huge arrays better - in
>> AbstractCollection.
>> I've put those changes into
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/ArrayResize2/
>
> I don't understand what you are doing here:
>
> 194 ? ? ?* The maximum size of array to allocate.
> 195 ? ? ?* Some VMs reserve some header words in an array.
> 196 ? ? ?* Attempts to allocate larger arrays may result in
> 197 ? ? ?* OutOfMemoryError: Requested array size exceeds VM limit
> 198 ? ? ?*/
> 199 ? ? private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
>
> On 32-bit you can't get anywhere near this limit. On 64-bit Hotspot
> currently allows you to allocate the entire array up to Integer.MAX_VALUE
> elements for types <= 32-bit; and MAX_VALUE-3 for 64-bit.
>
> Regardless this is VM specific and I don't see what purpose this serves.
> ???

I'm trying to avoid OOME due to exceeding the MAX_VALUE-3 limit
on a 64-bit VM,
when the number of actual elements is far smaller,
say 75% of MAX_VALUE.

Yes, you can think of this as a VM-specific bug workaround.

Martin


From martinrb at google.com  Fri Apr 16 19:07:20 2010
From: martinrb at google.com (Martin Buchholz)
Date: Fri, 16 Apr 2010 12:07:20 -0700
Subject: Disambiguating empty catch blocks
In-Reply-To: <D69455C4-288D-4A27-A5FE-DF6263719256@mac.com>
References: <4BC428D3.1060004@paradise.net.nz>
	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>
	<4BC59DD2.4060900@paradise.net.nz>
	<D69455C4-288D-4A27-A5FE-DF6263719256@mac.com>
Message-ID: <w2y1ccfd1c11004161207t57248a30z129cad3ec472df49@mail.gmail.com>

On Thu, Apr 15, 2010 at 20:02, Andrew Thompson
<lordpixel+core-libs-dev at mac.com> wrote:
>
> On Apr 14, 2010, at 6:49 AM, Bruce Chapman wrote:
>
>>> One thing we definitely can do is to add a new constructor
>>> to AssertionError, that takes a cause as an argument,
>>> consistent with other exceptions.
>>>
>> Yeah, that was a minor niggle - I'll check ?for an RFE and get the ball started on that.
>
>
> Actually, it does exist, it is just disguised as taking Object:
>
> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/AssertionError.html#AssertionError(java.lang.Object)

An awesome find, Andrew.
I had skimmed the (many) constructors for AssertionError,
and never suspected that this method is the one we want.

> It you want to RFE, I guess
>
> AssertionError(String additionalDetails, Throwable cause)

I'm guessing it's not critical, because the message of the cause
is a pretty good message for the wrapped exception.

The best reason for the new constructor is that humans can find it!

Martin


From joe.darcy at oracle.com  Fri Apr 16 20:02:39 2010
From: joe.darcy at oracle.com (Joe Darcy)
Date: Fri, 16 Apr 2010 13:02:39 -0700
Subject: Disambiguating empty catch blocks
In-Reply-To: <w2y1ccfd1c11004161207t57248a30z129cad3ec472df49@mail.gmail.com>
References: <4BC428D3.1060004@paradise.net.nz>	<s2p1ccfd1c11004131226k86f7827cq1f073321a11c06b7@mail.gmail.com>	<4BC59DD2.4060900@paradise.net.nz>	<D69455C4-288D-4A27-A5FE-DF6263719256@mac.com>
	<w2y1ccfd1c11004161207t57248a30z129cad3ec472df49@mail.gmail.com>
Message-ID: <4BC8C25F.9000205@oracle.com>

Martin Buchholz wrote:
> On Thu, Apr 15, 2010 at 20:02, Andrew Thompson
> <lordpixel+core-libs-dev at mac.com> wrote:
>   
>> On Apr 14, 2010, at 6:49 AM, Bruce Chapman wrote:
>>
>>     
>>>> One thing we definitely can do is to add a new constructor
>>>> to AssertionError, that takes a cause as an argument,
>>>> consistent with other exceptions.
>>>>
>>>>         
>>> Yeah, that was a minor niggle - I'll check  for an RFE and get the ball started on that.
>>>       
>> Actually, it does exist, it is just disguised as taking Object:
>>
>> http://java.sun.com/j2se/1.4.2/docs/api/java/lang/AssertionError.html#AssertionError(java.lang.Object)
>>     
>
> An awesome find, Andrew.
> I had skimmed the (many) constructors for AssertionError,
> and never suspected that this method is the one we want.
>
>   
>> It you want to RFE, I guess
>>
>> AssertionError(String additionalDetails, Throwable cause)
>>     
>
> I'm guessing it's not critical, because the message of the cause
> is a pretty good message for the wrapped exception.
>
> The best reason for the new constructor is that humans can find it!
>
> Martin
>   

Perhaps this is the bug you're looking for:
6935997 Please add a nested throwable constructor to AssertionError

-Joe



From assembling.signals at yandex.ru  Sat Apr 17 09:17:35 2010
From: assembling.signals at yandex.ru (assembling signals)
Date: Sat, 17 Apr 2010 13:17:35 +0400
Subject: How about Collections.emptySorted[Set|Map] ?
Message-ID: <150291271495855@web80.yandex.ru>

Hello, everyone!

In the Collections utility class we have

* emptyMap()
* emptySet()

* unmodifiableSet(...)
* unmodifiableMap(...)
* unmodifiableSortedSet(...)
* unmodifiableSortedMap(...)

So we have no

* emptySortedMap()
* emptySortedSet()

I have several times encountered situations where methods needed either SortedSet or SortedMap.

Do you think in would be appropriate to introduce the two proposed methods?

Best regards,
Ivan G Shevchenko.


From lists at laerad.com  Sat Apr 17 16:48:07 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sat, 17 Apr 2010 17:48:07 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<m2lbb01431e1004130452yfbac0bebw988433ecc9c8d3f3@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
Message-ID: <z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>

Hi Kevin,

As it happens I might have something useful still to contribute. As an
exercise in saving face I revisited the problem to see if I could achieve
the same complexity bounds as ChunkedArrayList but with a lower overhead. I
must admit I still didn't fully appreciate how the algorithm in
ChunkedArrayList worked until I tried to come up with an algorithm with
similar properties. What I have ended up with is almost identical except
adds I think a couple of incremental improvements, simply by redefining the
arrayIndex() method. I should note that I have not yet implemented more than
a prototype as it seems to me your implementation is excellent already, and
if it is decided to include my modifications the changes should be modest.

Firstly, (I hope that) what I have produced is a little more CPU pipe-line
friendly; there is less dependency on immediately preceding calculations at
each stage (i.e. so more operations should be able to proceed simultaneously
in the pipeline), and consists exclusively of shifts, addition/subtraction
and bit-wise (&)ands (except for the conditionals in
Integer.numberOfLeadingZeros(i)), although the total number of instructions
is approximately the same.

Secondly, I have modified the algorithm so that a "seed" size can be
specified (although I expect hard coding a suitable one will ultimately be
best). Whereas ChunkedArrayList currently requires that the pattern of array
allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12), 16
*s*(..*24)] etc. although when put in simple text like that it does appear
to trivialise the change. The benefit of this, though, is two fold: 1) for
small n the constant factor is reduced (both CPU and memory wise); and 2)
the sqrt(n) bounds are reached more quickly also.

As an illustration, consider setting *s* to 4, and assume the backing array
is size two and doubles in size with each growth; with ChunkedArrayList we
would resize at i=2, i=6, i=20, i=72; with *s* as 4 we would instead resize
at i=8,i=24,i=80,i=288; the cost at each would be some multiple of 2,4,8,16
respectively. As you can see the latter is much closer to the sqrt(n) cost -
both approach it eventually, but my suggestion is to reach it more quickly.
This is at the expense of more slowly reaching the sqrt(n) wasted memory
condition, but given the high constant factor cost wrt to memory at this
early stage, this seems a very sensible trade off. It seems likely this
should also have a positive impact on cache performance for smaller lists as
well.

Finally, after playing with this idea in my head I am confident I can extend
the core ideas of this data structure to hashing relatively easily, getting
the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n)) wasted
memory guarantees. I notice that this case hasn't been addressed yet,
although I see from Martin's recent mail that this was raised before. Unless
there are better suggestions for solving the hash table problem I will have
a go at it as it seems an interesting problem - that is, assuming there are
no objections?

I'm interested to hear your thoughts. I hope this time I've been a bit more
considered in what I've put forward, and hence less of a waste of time!

Code snippet for calculation of array index and item offset:

final int arraySizeShiftMinusSeed = ((31 -
Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3) - (1
<< arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
final int indexRemainder = index - ((1 << seed) << arraySizeShiftMinusSeed +
arraySizeShiftMinusSeed) ;
final int arrayOffset = indexRemainder >>> arraySizeShift ;

final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
final int itemIndex = index & ((1 << arraySizeShift) - 1) ;

the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* + 1);
seed only works for values for 2 or more at this moment, fyi



On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Oh no worries Benedict, thanks for your interest in the topic.  Let me know
> if you have any other questions or if you have any related ideas or
> concerns.
>
>
> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <lists at laerad.com>wrote:
>
>> Sorry Kevin - it sounds like I might be being of more hindrance than help.
>> that part of the discussion was clearly truncated by the time I had joined
>> the list - I haven't been able to find the history in the archives either...
>>
>> I was just wondering about the worst case cost of add() as described by
>> your javadoc; admittedly it is optimal with respect to unused memory, but
>> the worst case cost of an add is still sqrt(n), with a relatively high
>> constant factor. I had been thinking that once n passed a threshold the cost
>> of additions in this other structure would become a constant factor,
>> offering nice algorithmic complexity guarantees for large n; however since
>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>> allocations would have to be unrealistically small (assuming linear cost for
>> allocation) for this to be the case. It would still be nice to have a data
>> structure that avoids needing to copy data with each grow, whilst still
>> maintaining good memory performance.
>>
>> That *all* being said, I had been going by your javadoc and emails to
>> ascertain the behaviour of this class, as I couldn't locate a free copy
>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>> sqrt(n) cost appears to be associated with growing the backing array, rather
>> than with what I assumed to be copying data between arraylets, and it seems
>> this cost is pretty optimal. That will teach me to post to a list without
>> getting my facts straight first. The interesting thing is simply that the
>> constant factor for this implementation still seems to be quite high,
>> although perhaps that is simply because I was not benchmarking sufficiently
>> large values of n.
>>
>>
>>
>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Hi Benedict,
>>>
>>> Unless I am misreading your post, this now has a very similar feel to the
>>> first data structure that I posted to the list.  Martin Buchholz then
>>> pointed out that we can incorporate the ideas from
>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>
>>> Regards,
>>>
>>> Kevin
>>>
>>>
>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> Yes, as I was going to bed last night I realised I had not fully
>>>> addressed the problem that was originally being visited; only reduced the
>>>> constant factor for addition to the end of the list. A trivial modification
>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>> power of 2), after which the array is increased in size linearly.
>>>> Performance doesn't seem to have been affected appreciably, although not
>>>> been exhaustive in the benchmarking:
>>>>
>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>> 10 items get Chunked / ExpArray = 0.99
>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>> 100 items get Chunked / ExpArray = 1.23
>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>> 1000 items get Chunked / ExpArray = 1.19
>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>> 10000 items get Chunked / ExpArray = 1.18
>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>
>>>> The nice thing about this is that the maximum amount of wasted memory is
>>>> user configurable. Even with a low setting as above (65K) performance seems
>>>> pretty consistent.
>>>>
>>>> Code for calculating index and array offset are pretty straight forward;
>>>> haven't given much thought to optimisations just yet:
>>>>
>>>> private final int indexFor(int a, int i) {
>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>> maxArraySizeShift : 1 << a) ;
>>>>  }
>>>> private final int arrayFor(int i) {
>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>  }
>>>>
>>>> Regarding the double list idea - yes, I agree, I certainly didn't think
>>>> that one through fully!
>>>>
>>>>
>>>>
>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Hi Benedict,
>>>>>
>>>>> Like you, I am relatively new to this mailing list; I am also trying to
>>>>> tread lightly so as not to step on any toes.  That being said, I think that
>>>>> I can offer a response to your inquiry.
>>>>>
>>>>> Regarding:  "The idea is to simply double the new array size each time
>>>>> a new array needs to be allocated"
>>>>>
>>>>> It seems this would not address the desire of offering an alternative
>>>>> to the allocation of a large backing array for ArrayList (your largest
>>>>> backing array could still reach a size of 1/2 * Integer.MAX_VALUE) and would
>>>>> not address the desire of wasting the (asymptotically) minimum amount of
>>>>> memory in the worst case while maintaining O(1) amortized time bounds.  The
>>>>> data structure described in [Brodnik99resizablearrays] has a maximum backing
>>>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>>>> over ArrayList do you see in your data structure?
>>>>>
>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems that
>>>>> the simplest solution would be to simply have two lists, with one accepting
>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>> accepted inserts for near to the end."
>>>>>
>>>>> What happens with your structure when you add n elements and then
>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>> resize.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>> lists at laerad.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>
>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>> obviously inserts at the end are much quicker.
>>>>>>
>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>> impressive.
>>>>>>
>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>>> statistic per list, and the second round results were used.
>>>>>>
>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>
>>>>>> and server JVM:
>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>
>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>> expected them to be fairly close.
>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at
>>>>>> the beginning of large lists, which I haven't yet tried to understand.
>>>>>> Insertion in the middle is similar.
>>>>>>
>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>
>>>>>> Finally, there are promising results for get() from the ExpArrayList
>>>>>> as well (server JVM), again somehow beating ArrayList for larger lists:
>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>
>>>>>>
>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>> source code and I will find somewhere to upload it.
>>>>>>
>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>> could easily be achieved by creating an extension of List with a
>>>>>> reverseIterator() method.
>>>>>>
>>>>>>
>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>> and all of them for a backing array resize.
>>>>>>>
>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Joe
>>>>>>>
>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>
>>>>>>> Hi Joe,
>>>>>>>>
>>>>>>>> I was referring to the ChunkedArrayList when I stated that add does
>>>>>>>> not amortize to constant time when the data structure employs the circular
>>>>>>>> list trick to achieve deque behavior; ChunkedArrayList potentially resizes
>>>>>>>> every n^(1/2) operations.
>>>>>>>>
>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>
>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>
>>>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>
>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>
>>>>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>>>>> implementation only moves array elements to the front on an array resize
>>>>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>
>>>>>>>>> Some performance results below, code for these is in the repository
>>>>>>>>> above too. This was the second run, after a warmup.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>> ------------------------------------------------ CircularArrayList
>>>>>>>>> ------------------------------------------------
>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>      10          20          67          70         125
>>>>>>>>> 102          90         240         191         138
>>>>>>>>>    100          19          67          70         166         138
>>>>>>>>>          94         230         194         118
>>>>>>>>>   1000          28          64          67         681         538
>>>>>>>>>          91         324         382         119
>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>> ----------------------------------------------------
>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>      10          23          68          70         100
>>>>>>>>>  69       32913         162         130         105
>>>>>>>>>    100          20          67          70         129         104
>>>>>>>>>       21944         169         134         135
>>>>>>>>>   1000          29          63          67         651         506
>>>>>>>>>        9602         364         333         526
>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>
>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>
>>>>>>>>> Hi Martin,
>>>>>>>>>>
>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> It's interesting to note that the old circular list trick will
>>>>>>>>>>> not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>> deque.
>>>>>>>>>>>
>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>
>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>
>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>
>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>
>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>
>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>
>>>>>>>>>>>> Martin
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>> >
>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in the
>>>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>>>> shape.  I have
>>>>>>>>>>>> > also gathered some data on the performance of ChunkedArrayList
>>>>>>>>>>>> over
>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>>>>> (note that the
>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>> operation with
>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>>>> indicates
>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that ChunkedArrayList
>>>>>>>>>>>> is less
>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>>>> I've noticed
>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>> when I switch
>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>> performance
>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>> timed the process
>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I performed
>>>>>>>>>>>> a get
>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I used
>>>>>>>>>>>> the iterator
>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>> course, I performed
>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>> timing for the
>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>> >
>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>> eliminate the
>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>> achieving this by
>>>>>>>>>>>> > way of a data structure that is both time and space optimal is
>>>>>>>>>>>> a
>>>>>>>>>>>> > particularly elegant solution as it not only guarantees that
>>>>>>>>>>>> no backing
>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>>> > course, this data structure does not do everything better than
>>>>>>>>>>>> ArrayList; in
>>>>>>>>>>>> > particular, indexed access is more costly, due to the required
>>>>>>>>>>>> decomposition
>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>> additional memory
>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to
>>>>>>>>>>>> the multiple
>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>> said, I think that
>>>>>>>>>>>> > the additional cost of indexed access is partially mitigated
>>>>>>>>>>>> by the
>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>>>> ArrayList due
>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Kevin
>>>>>>>>>>>> >
>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>> >
>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>> >
>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>> >
>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>> >
>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>> >
>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>> >
>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>> >
>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK eventually
>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> Martin
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>>>> originally
>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>> ).
>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet
>>>>>>>>>>>> as it was
>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple small
>>>>>>>>>>>> arrays
>>>>>>>>>>>> >> > rather
>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>> >> > with
>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>> >> > other
>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>>>> more important
>>>>>>>>>>>> >> > and
>>>>>>>>>>>> >> > popular collection, it's the primary "straight replacement
>>>>>>>>>>>> for primitive
>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>>>> updating the
>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>> >> >>
>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.
>>>>>>>>>>>> The code below
>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>>> offset into the
>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>>>>> this will
>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>>>>> each time an
>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle
>>>>>>>>>>>> the bits to see
>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b) +
>>>>>>>>>>>> " " + e);
>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>>> numbered
>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>>> added (the
>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>> >> >> >> across an alternative characterization of superblock in
>>>>>>>>>>>> the paper
>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a superblock
>>>>>>>>>>>> when they
>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my example
>>>>>>>>>>>> structure
>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>> optimal time and
>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>> following
>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>> consists of
>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>> implies the
>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i =
>>>>>>>>>>>> 3:
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>>>>> "the number
>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only
>>>>>>>>>>>> two data
>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>>>> example above,
>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>> their data
>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>>>>> that comes to
>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear insertion
>>>>>>>>>>>> is to create
>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>> insert at the
>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size)
>>>>>>>>>>>> and to insert
>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>>>>> size).  We
>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could
>>>>>>>>>>>> you explain
>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>>>>> shared by the
>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a
>>>>>>>>>>>> way to turn
>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or be
>>>>>>>>>>>> a better
>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>>>> that you
>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space", gives
>>>>>>>>>>>> a deque so
>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>> operations -
>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100417/ff072137/attachment.html>

From opinali at gmail.com  Sat Apr 17 17:03:30 2010
From: opinali at gmail.com (Osvaldo Doederlein)
Date: Sat, 17 Apr 2010 14:03:30 -0300
Subject: How about Collections.emptySorted[Set|Map] ?
In-Reply-To: <150291271495855@web80.yandex.ru>
References: <150291271495855@web80.yandex.ru>
Message-ID: <l2nfb5ec5091004171003i1d6df4a2yd710a82a30b7fb5f@mail.gmail.com>

2010/4/17 assembling signals <assembling.signals at yandex.ru>

> Hello, everyone!
>
> In the Collections utility class we have
>
> * emptyMap()
> * emptySet()
>
> * unmodifiableSet(...)
> * unmodifiableMap(...)
> * unmodifiableSortedSet(...)
> * unmodifiableSortedMap(...)
>
> So we have no
>
> * emptySortedMap()
> * emptySortedSet()
>
> I have several times encountered situations where methods needed either
> SortedSet or SortedMap.
>
> Do you think in would be appropriate to introduce the two proposed methods?
>
>
Why not just changing the implementation of the existing emptyMap() and
emptySet(), so they return an object that is additionally sorted? This would
require a typecast, e.g. SortedMap<T> map =
(SortedMap<T>)Collections.emptyMap(); but wouldn't require changing the
existing API, or adding any new API. Just make very clear in the
documentation that the typecast is valid only @since 1.7.

A+
Osvaldo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100417/f371ef6c/attachment.html>

From assembling.signals at yandex.ru  Sat Apr 17 17:46:16 2010
From: assembling.signals at yandex.ru (assembling signals)
Date: Sat, 17 Apr 2010 21:46:16 +0400
Subject: How about Collections.emptySorted[Set|Map] ?
In-Reply-To: <l2nfb5ec5091004171003i1d6df4a2yd710a82a30b7fb5f@mail.gmail.com>
References: <150291271495855@web80.yandex.ru>
	<l2nfb5ec5091004171003i1d6df4a2yd710a82a30b7fb5f@mail.gmail.com>
Message-ID: <195501271526376@web77.yandex.ru>

This is a nice idea!

Additionally, (inspired by this idea,) emptyList() could return something that is castable to Deque
(which is a sub-interface of Queue). Think of an empty plus unmodifiable LinkedList (which implements List, Deque, Queue).

Deque<String> d = (Deque<String>) Collections.emptyList();

This, together with the Sorted[Set|Map] proposal, would cover most important collections interfaces, right?

17.04.10, 14:03, "Osvaldo Doederlein" <opinali at gmail.com>:
> 2010/4/17 assembling signals <assembling.signals at yandex.ru>
> > Hello, everyone!
> > 
> > In the Collections utility class we have
> > 
> > * emptyMap()
> > * emptySet()
> > 
> > * unmodifiableSet(...)
> > * unmodifiableMap(...)
> > * unmodifiableSortedSet(...)
> > * unmodifiableSortedMap(...)
> > 
> > So we have no
> > 
> > * emptySortedMap()
> > * emptySortedSet()
> > 
> > I have several times encountered situations where methods needed either SortedSet or SortedMap.
> > 
> > Do you think in would be appropriate to introduce the two proposed methods?
> > 
> 
> Why not just changing the implementation of the existing emptyMap() and emptySet(), so they return an object that is additionally sorted? This would require a typecast, e.g. SortedMap map = (SortedMap)Collections.emptyMap(); but wouldn't require changing the existing API, or adding any new API. Just make very clear in the documentation that the typecast is valid only @since 1.7.
> 
> A+
> Osvaldo


From assembling.signals at yandex.ru  Sat Apr 17 18:05:50 2010
From: assembling.signals at yandex.ru (assembling signals)
Date: Sat, 17 Apr 2010 22:05:50 +0400
Subject: One more proposal for "Collections": constantSized[List|Map]
Message-ID: <367551271527550@web58.yandex.ru>

Hello, everyone!

My next proposal for the "Collections" utility class is a bunch of methods (similar to unmodifiable[...])
which would return not an unmodifiable List|Map|Set, but a constant-sized one.
That is, all methods which get|set|... elements would work as earlier, but methods which
would alter the size of the underlying collection (add|put|remove|retain|...) would throw UnsupportedOperationException.

So the new logic hoerarchy would be:
1) Unmodifiable[something] -- very strict
2) ConstantSized[something] -- less strict
3) [Usual collections]

A constant-sized-list would be nearly same as an unmodifiable-list, with the exception that set(int index)
would be allowed.
A constant-sized-map would be nearly same as an unmodifiable-map, with the exception that put(K key, V value) on EXISTING
keys would be allowed.

I have no opinion of the usability of this proposal for Maps or Sets (it could make sense, but it could make none).
But I am very sure, that for Lists it is very useful.

What do you think of such a proposal? Not too specific? I, for myself, need such constant-sized collections often.

( I don't know about the name. ConstantSized[...]? NonResizable[...]? Sealed[...]? )

Please take a look at a proposal for a ConstantSizedList:

import java.util.*;

/**
 * Wrapps around an ArrayList and prevents all resizing methods by throwing
 * an UnsupportedOperationException. The underlying ArrayList still may be changed.
 *
 * The underlying ArrayList is sealed by following means:
 * - methods which return an iterator use Collections.unmodifiableList(...);
 * - adding, clearing, removing, retaining methods throw UnsupportedOperationException;
 * - all other methods are delegated to the underlying ArrayList.
 */
public final class ConstantSizeArrayList<T>
		implements List<T>, RandomAccess, Cloneable {

	private final static RuntimeException ex =
			new UnsupportedOperationException(ConstantSizeArrayList.class
			+ " doesn't allow methods which resize the underlying collection");

	private final ArrayList<T> underlying;

	public ConstantSizeArrayList(ArrayList<T> underlying) {
		this.underlying = underlying;
		underlying.trimToSize();
	}

	@Override
	public String toString() {
		return underlying.toString();
	}

	@Override
	public boolean containsAll(Collection<?> c) {
		return underlying.containsAll(c);
	}

	@Override
	public int hashCode() {
		return underlying.hashCode();
	}

	@Override
	public boolean equals(Object o) {
		return underlying.equals(o);
	}

	@Override
	public <T> T[] toArray(T[] a) {
		return underlying.toArray(a);
	}

	@Override
	public Object[] toArray() {
		return underlying.toArray();
	}

	@Override
	public List<T> subList(int fromIndex, int toIndex) {
		return underlying.subList(fromIndex, toIndex);
	}

	@Override
	public int size() {
		return underlying.size();
	}

	@Override
	public T set(int index, T element) {
		return underlying.set(index, element);
	}

	@Override
	public Iterator<T> iterator() {
		return Collections.unmodifiableList(underlying).iterator();
	}

	@Override
	public ListIterator<T> listIterator() {
		return Collections.unmodifiableList(underlying).listIterator();
	}

	@Override
	public ListIterator<T> listIterator(int index) {
		return Collections.unmodifiableList(underlying).listIterator(index);
	}

	@Override
	public int lastIndexOf(Object o) {
		return underlying.lastIndexOf(o);
	}

	@Override
	public boolean isEmpty() {
		return underlying.isEmpty();
	}

	@Override
	public int indexOf(Object o) {
		return underlying.indexOf(o);
	}

	@Override
	public T get(int index) {
		return underlying.get(index);
	}

	@Override
	@SuppressWarnings("element-type-mismatch")
	public boolean contains(Object o) {
		return underlying.contains(o);
	}

	// TODO delegating clone: is that OK?
	@Override
	public Object clone() {
		return underlying.clone();
	}

	@Override
	public void clear() {
		throw ex;
	}

	@Override
	public boolean addAll(int index, Collection<? extends T> c) {
		throw ex;
	}

	@Override
	public boolean addAll(Collection<? extends T> c) {
		throw ex;
	}

	@Override
	public void add(int index, T element) {
		throw ex;
	}

	@Override
	public boolean add(T e) {
		throw ex;
	}

	@Override
	public boolean retainAll(Collection<?> c) {
		throw ex;
	}

	@Override
	public boolean removeAll(Collection<?> c) {
		throw ex;
	}

	@Override
	public boolean remove(Object o) {
		throw ex;
	}

	@Override
	public T remove(int index) {
		throw ex;
	}

}



Best regards,
Ivan G Shevchenko


From jim.andreou at gmail.com  Sat Apr 17 18:40:23 2010
From: jim.andreou at gmail.com (Dimitris Andreou)
Date: Sat, 17 Apr 2010 21:40:23 +0300
Subject: One more proposal for "Collections": constantSized[List|Map]
In-Reply-To: <367551271527550@web58.yandex.ru>
References: <367551271527550@web58.yandex.ru>
Message-ID: <g2z7d7138c11004171140m40250575j2b710fcd39d6e17@mail.gmail.com>

>I have no opinion of the usability of this proposal for Maps or Sets (it
could make sense, but it could make none).
>But I am very sure, that for Lists it is very useful.

You can create a constant-sized mutable List like this:

public static List<T> constantSizeList(T[] array) {
  return Arrays.asList(array);
}

Thus what you describe as the most useful case is already covered.

Regards,
Dimitris

2010/4/17 assembling signals <assembling.signals at yandex.ru>

> Hello, everyone!
>
> My next proposal for the "Collections" utility class is a bunch of methods
> (similar to unmodifiable[...])
> which would return not an unmodifiable List|Map|Set, but a constant-sized
> one.
> That is, all methods which get|set|... elements would work as earlier, but
> methods which
> would alter the size of the underlying collection
> (add|put|remove|retain|...) would throw UnsupportedOperationException.
>
> So the new logic hoerarchy would be:
> 1) Unmodifiable[something] -- very strict
> 2) ConstantSized[something] -- less strict
> 3) [Usual collections]
>
> A constant-sized-list would be nearly same as an unmodifiable-list, with
> the exception that set(int index)
> would be allowed.
> A constant-sized-map would be nearly same as an unmodifiable-map, with the
> exception that put(K key, V value) on EXISTING
> keys would be allowed.
>
> I have no opinion of the usability of this proposal for Maps or Sets (it
> could make sense, but it could make none).
> But I am very sure, that for Lists it is very useful.


> What do you think of such a proposal? Not too specific? I, for myself, need
> such constant-sized collections often.
>
> ( I don't know about the name. ConstantSized[...]? NonResizable[...]?
> Sealed[...]? )
>
> Please take a look at a proposal for a ConstantSizedList:
>
> import java.util.*;
>
> /**
>  * Wrapps around an ArrayList and prevents all resizing methods by throwing
>  * an UnsupportedOperationException. The underlying ArrayList still may be
> changed.
>  *
>  * The underlying ArrayList is sealed by following means:
>  * - methods which return an iterator use
> Collections.unmodifiableList(...);
>  * - adding, clearing, removing, retaining methods throw
> UnsupportedOperationException;
>  * - all other methods are delegated to the underlying ArrayList.
>  */
> public final class ConstantSizeArrayList<T>
>                implements List<T>, RandomAccess, Cloneable {
>
>        private final static RuntimeException ex =
>                        new
> UnsupportedOperationException(ConstantSizeArrayList.class
>                        + " doesn't allow methods which resize the
> underlying collection");
>
>        private final ArrayList<T> underlying;
>
>        public ConstantSizeArrayList(ArrayList<T> underlying) {
>                this.underlying = underlying;
>                underlying.trimToSize();
>        }
>
>        @Override
>        public String toString() {
>                return underlying.toString();
>        }
>
>        @Override
>        public boolean containsAll(Collection<?> c) {
>                return underlying.containsAll(c);
>        }
>
>        @Override
>        public int hashCode() {
>                return underlying.hashCode();
>        }
>
>        @Override
>        public boolean equals(Object o) {
>                return underlying.equals(o);
>        }
>
>        @Override
>        public <T> T[] toArray(T[] a) {
>                return underlying.toArray(a);
>        }
>
>        @Override
>        public Object[] toArray() {
>                return underlying.toArray();
>        }
>
>        @Override
>        public List<T> subList(int fromIndex, int toIndex) {
>                return underlying.subList(fromIndex, toIndex);
>        }
>
>        @Override
>        public int size() {
>                return underlying.size();
>        }
>
>        @Override
>        public T set(int index, T element) {
>                return underlying.set(index, element);
>        }
>
>        @Override
>        public Iterator<T> iterator() {
>                return Collections.unmodifiableList(underlying).iterator();
>        }
>
>        @Override
>        public ListIterator<T> listIterator() {
>                return
> Collections.unmodifiableList(underlying).listIterator();
>        }
>
>        @Override
>        public ListIterator<T> listIterator(int index) {
>                return
> Collections.unmodifiableList(underlying).listIterator(index);
>        }
>
>        @Override
>        public int lastIndexOf(Object o) {
>                return underlying.lastIndexOf(o);
>        }
>
>        @Override
>        public boolean isEmpty() {
>                return underlying.isEmpty();
>        }
>
>        @Override
>        public int indexOf(Object o) {
>                return underlying.indexOf(o);
>        }
>
>        @Override
>        public T get(int index) {
>                return underlying.get(index);
>        }
>
>        @Override
>        @SuppressWarnings("element-type-mismatch")
>        public boolean contains(Object o) {
>                return underlying.contains(o);
>        }
>
>        // TODO delegating clone: is that OK?
>        @Override
>        public Object clone() {
>                return underlying.clone();
>        }
>
>        @Override
>        public void clear() {
>                throw ex;
>        }
>
>        @Override
>        public boolean addAll(int index, Collection<? extends T> c) {
>                throw ex;
>        }
>
>        @Override
>        public boolean addAll(Collection<? extends T> c) {
>                throw ex;
>        }
>
>        @Override
>        public void add(int index, T element) {
>                throw ex;
>        }
>
>        @Override
>        public boolean add(T e) {
>                throw ex;
>        }
>
>        @Override
>        public boolean retainAll(Collection<?> c) {
>                throw ex;
>        }
>
>        @Override
>        public boolean removeAll(Collection<?> c) {
>                throw ex;
>        }
>
>        @Override
>        public boolean remove(Object o) {
>                throw ex;
>        }
>
>        @Override
>        public T remove(int index) {
>                throw ex;
>        }
>
> }
>
>
>
> Best regards,
> Ivan G Shevchenko
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100417/ead4c15e/attachment.html>

From lana.steuck at sun.com  Sat Apr 17 22:11:50 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:11:50 +0000
Subject: hg: jdk7/tl: 6 new changesets
Message-ID: <20100417221151.766A8445E7@hg.openjdk.java.net>

Changeset: 6b1069f53fbc
Author:    mikejwre
Date:      2010-03-18 13:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/6b1069f53fbc

Added tag jdk7-b86 for changeset 433a60a9c0bf

! .hgtags

Changeset: 82135c848d5f
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/82135c848d5f

Added tag jdk7-b87 for changeset 6b1069f53fbc

! .hgtags

Changeset: 9d9097eb4b5a
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/9d9097eb4b5a

Added tag jdk7-b88 for changeset 82135c848d5f

! .hgtags

Changeset: b1c3b0e44b9d
Author:    lana
Date:      2010-04-08 15:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/b1c3b0e44b9d

Merge


Changeset: 7f1ba4459972
Author:    lana
Date:      2010-04-13 16:35 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/7f1ba4459972

Merge


Changeset: 425ba3efabbf
Author:    mikejwre
Date:      2010-04-15 13:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/rev/425ba3efabbf

Added tag jdk7-b89 for changeset 7f1ba4459972

! .hgtags



From lana.steuck at sun.com  Sat Apr 17 22:11:57 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:11:57 +0000
Subject: hg: jdk7/tl/corba: 4 new changesets
Message-ID: <20100417221202.D193E445E8@hg.openjdk.java.net>

Changeset: 09a41111a401
Author:    mikejwre
Date:      2010-03-18 13:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/corba/rev/09a41111a401

Added tag jdk7-b86 for changeset 6253e28826d1

! .hgtags

Changeset: 39e14d2da687
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/corba/rev/39e14d2da687

Added tag jdk7-b87 for changeset 09a41111a401

! .hgtags

Changeset: bb4424c5e778
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/corba/rev/bb4424c5e778

Added tag jdk7-b88 for changeset 39e14d2da687

! .hgtags

Changeset: 56ce07b0eb47
Author:    mikejwre
Date:      2010-04-15 13:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/corba/rev/56ce07b0eb47

Added tag jdk7-b89 for changeset bb4424c5e778

! .hgtags



From lana.steuck at sun.com  Sat Apr 17 22:13:30 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:13:30 +0000
Subject: hg: jdk7/tl/hotspot: 65 new changesets
Message-ID: <20100417221649.7512C445E9@hg.openjdk.java.net>

Changeset: 4b58861a3570
Author:    mikejwre
Date:      2010-03-18 13:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/4b58861a3570

Added tag jdk7-b86 for changeset bf823ef06b4f

! .hgtags

Changeset: 877a14af58e1
Author:    never
Date:      2010-02-18 15:05 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/877a14af58e1

6663854: assert(n != __null,"Bad immediate dominator info.") in C2 with -Xcomp
Reviewed-by: kvn

! src/share/vm/opto/split_if.cpp
+ test/compiler/6663854/Test6663854.java

Changeset: 2883969d09e7
Author:    kvn
Date:      2010-02-19 10:04 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/2883969d09e7

6910664: C2: java/util/Arrays/Sorting.java fails with DeoptimizeALot flag
Summary: Matcher::float_in_double should be true only when FPU is used for floats.
Reviewed-by: never, twisti

! src/cpu/sparc/vm/sparc.ad
! src/cpu/x86/vm/x86_32.ad
! src/cpu/x86/vm/x86_64.ad
! src/share/vm/opto/matcher.hpp
! src/share/vm/opto/output.cpp

Changeset: b71f13525cc8
Author:    never
Date:      2010-02-19 13:06 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/b71f13525cc8

6927049: assert(is_Loop(),"invalid node class")
Reviewed-by: kvn

! src/share/vm/opto/loopTransform.cpp

Changeset: 8b38237bae55
Author:    kvn
Date:      2010-02-22 16:56 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/8b38237bae55

6928717: HS17 fails to build with SS11 C++
Summary: Add missing handles.inline.hpp for codeCache.cpp.
Reviewed-by: never

! src/share/vm/includeDB_core

Changeset: 855c5171834c
Author:    twisti
Date:      2010-02-23 17:46 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/855c5171834c

6928839: JSR 292 typo in x86 _adapter_check_cast
Summary: There is a small typo in methodHandles_x86.cpp.
Reviewed-by: kvn

! src/cpu/x86/vm/methodHandles_x86.cpp

Changeset: da9559b49b84
Author:    never
Date:      2010-02-25 11:38 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/da9559b49b84

6915557: assert(_gvn.type(l)->higher_equal(type),"must constrain OSR typestate") with debug build
Reviewed-by: kvn

! src/share/vm/opto/parse1.cpp

Changeset: 2432acbee618
Author:    kvn
Date:      2010-02-25 15:55 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/2432acbee618

6930035: C2 type system incorrectly handles case j.l.Object->meet(constant AryPtr)
Summary: Add missing code.
Reviewed-by: never

! src/share/vm/opto/type.cpp

Changeset: 336c6c200f5f
Author:    kvn
Date:      2010-02-25 22:58 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/336c6c200f5f

6930116: loop predication code does not handle If nodes with only one projection
Summary: Add check for iff->outcnt() < 2.
Reviewed-by: never

! src/share/vm/opto/loopTransform.cpp

Changeset: 7d236a9688c5
Author:    never
Date:      2010-03-01 12:12 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/7d236a9688c5

6930398: fix for return address locals in OSR entries uses wrong test
Reviewed-by: kvn

! src/share/vm/opto/parse1.cpp

Changeset: b81f3572f355
Author:    tonyp
Date:      2010-02-23 23:13 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/b81f3572f355

6928059: G1: command line parameter renaming
Summary: Rename G1 parameters to make them more consistent.
Reviewed-by: jmasa, johnc

! src/share/vm/gc_implementation/g1/concurrentG1Refine.cpp
! src/share/vm/gc_implementation/g1/concurrentG1Refine.hpp
! src/share/vm/gc_implementation/g1/concurrentG1RefineThread.cpp
! src/share/vm/gc_implementation/g1/concurrentMark.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp
! src/share/vm/gc_implementation/g1/g1MMUTracker.cpp
! src/share/vm/gc_implementation/g1/g1MMUTracker.hpp
! src/share/vm/gc_implementation/g1/g1RemSet.cpp
! src/share/vm/gc_implementation/g1/g1_globals.hpp

Changeset: 1c72304f1885
Author:    tonyp
Date:      2010-02-23 23:14 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/1c72304f1885

6928073: G1: use existing command line parameters for marking cycle initiation
Summary: replace the combination of the G1SteadyStateUsed / G1SteadyStateUsedDelta parameteres to decide the marking initiation threshold and instead use InitiatingHeapOccupancyPercent.
Reviewed-by: ysr, johnc

! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp
! src/share/vm/gc_implementation/g1/g1_globals.hpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/globals.hpp

Changeset: 5f1f51edaff6
Author:    jmasa
Date:      2010-02-24 07:00 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/5f1f51edaff6

6928081: G1: rename parameters common with CMS
Summary: Rename marking stack sizing flags to be common between G1 and CMS
Reviewed-by: ysr, tonyp

! src/share/vm/gc_implementation/concurrentMarkSweep/cmsAdaptiveSizePolicy.cpp
! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp
! src/share/vm/gc_implementation/g1/concurrentMark.cpp
! src/share/vm/gc_implementation/g1/g1_globals.hpp
! src/share/vm/includeDB_core
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/arguments.hpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/utilities/taskqueue.hpp

Changeset: a1c410de27e4
Author:    tonyp
Date:      2010-02-24 14:56 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/a1c410de27e4

6928065: G1: use existing command line parameters to set the young generation size
Summary: see synopsis
Reviewed-by: johnc, jmasa

! src/share/vm/gc_implementation/g1/g1CollectorPolicy.cpp
! src/share/vm/gc_implementation/g1/g1_globals.hpp

Changeset: ab75c83d7c37
Author:    johnc
Date:      2010-03-02 13:57 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/ab75c83d7c37

Merge

! src/share/vm/includeDB_core

Changeset: 8911d8c0596f
Author:    phh
Date:      2010-02-26 16:40 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/8911d8c0596f

6923123: Hotspot refuses to start when -Xmx4m or -Xms4m is specified
Summary: Reduce NewSize from 4m to 1m.
Reviewed-by: tonyp, jmasa

! src/share/vm/runtime/globals.hpp

Changeset: c76ca382971b
Author:    johnc
Date:      2010-03-02 13:59 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/c76ca382971b

Merge

! src/share/vm/runtime/globals.hpp

Changeset: d47555d7aca8
Author:    jmasa
Date:      2010-03-03 08:10 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/d47555d7aca8

6910182: CMS: assert(_cursor[j] == _survivor_plab_array[j].end(),"Ctl pt invariant")
Summary: Calculation of the slicing of survivor spaces for MT was incorrect.
Reviewed-by: ysr

! src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp

Changeset: c8a467bf56ad
Author:    coleenp
Date:      2010-03-02 12:09 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/c8a467bf56ad

6914050: jvm assertion "guard pages must be in use" in -Xcomp mode
Summary: Move creating stack guard pages in jni attach thread before potential java call rather than after. Also cleanup stack guard pages when jni attach fails
Reviewed-by: never, dholmes

! src/share/vm/prims/jni.cpp
! src/share/vm/runtime/thread.cpp

Changeset: 4b0f2f4918ed
Author:    xlu
Date:      2010-03-10 21:42 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/4b0f2f4918ed

6933402: RFE: Improve PrintSafepointStatistics output to track cleanup time
Summary: Improve the usability of safepoint statistics data. See bug evaluation for more details.
Reviewed-by: ysr, dholmes

! src/share/vm/runtime/safepoint.cpp
! src/share/vm/runtime/safepoint.hpp

Changeset: 12d91eb0f579
Author:    acorn
Date:      2010-03-11 14:41 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/12d91eb0f579

Merge


Changeset: 51db1e4b379d
Author:    twisti
Date:      2010-03-08 04:46 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/51db1e4b379d

6932536: JSR 292 modified JDK MethodHandlesTest fails on x86_64
Summary: A modified MethodHandlesTest revealed two bugs on x86_64.
Reviewed-by: never, jrose

! src/cpu/x86/vm/methodHandles_x86.cpp

Changeset: 7de45b5044c3
Author:    never
Date:      2010-03-09 11:02 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/7de45b5044c3

6932270: Allow Java's ELF symtab reader to use separate debuginfo files
Reviewed-by: never
Contributed-by: Andrew Haley <aph at redhat.com>

! agent/src/os/linux/libproc_impl.c
! agent/src/os/linux/symtab.c
! agent/src/os/linux/symtab.h
+ make/linux/makefiles/build_vm_def.sh
! make/linux/makefiles/mapfile-vers-debug
! make/linux/makefiles/mapfile-vers-product
! make/linux/makefiles/vm.make

Changeset: 3cf667df43ef
Author:    twisti
Date:      2010-03-09 20:16 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3cf667df43ef

6919934: JSR 292 needs to support x86 C1
Summary: This implements JSR 292 support for C1 x86.
Reviewed-by: never, jrose, kvn

! src/cpu/sparc/vm/c1_CodeStubs_sparc.cpp
! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp
! src/cpu/sparc/vm/c1_MacroAssembler_sparc.cpp
! src/cpu/sparc/vm/c1_Runtime1_sparc.cpp
! src/cpu/sparc/vm/interp_masm_sparc.cpp
! src/cpu/sparc/vm/interp_masm_sparc.hpp
! src/cpu/sparc/vm/stubGenerator_sparc.cpp
! src/cpu/sparc/vm/templateInterpreter_sparc.cpp
! src/cpu/x86/vm/c1_CodeStubs_x86.cpp
! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp
! src/cpu/x86/vm/c1_MacroAssembler_x86.cpp
! src/cpu/x86/vm/c1_Runtime1_x86.cpp
! src/cpu/x86/vm/stubGenerator_x86_32.cpp
! src/cpu/x86/vm/stubGenerator_x86_64.cpp
! src/cpu/x86/vm/templateInterpreter_x86_32.cpp
! src/cpu/x86/vm/templateInterpreter_x86_64.cpp
! src/share/vm/c1/c1_Canonicalizer.cpp
! src/share/vm/c1/c1_CodeStubs.hpp
! src/share/vm/c1/c1_GraphBuilder.cpp
! src/share/vm/c1/c1_IR.cpp
! src/share/vm/c1/c1_IR.hpp
! src/share/vm/c1/c1_Instruction.cpp
! src/share/vm/c1/c1_Instruction.hpp
! src/share/vm/c1/c1_LIR.cpp
! src/share/vm/c1/c1_LIR.hpp
! src/share/vm/c1/c1_LIRAssembler.cpp
! src/share/vm/c1/c1_LIRAssembler.hpp
! src/share/vm/c1/c1_LIRGenerator.cpp
! src/share/vm/c1/c1_MacroAssembler.hpp
! src/share/vm/ci/ciCPCache.cpp
! src/share/vm/ci/ciCPCache.hpp
! src/share/vm/includeDB_compiler1
! src/share/vm/includeDB_core
! src/share/vm/opto/runtime.cpp
! src/share/vm/runtime/sharedRuntime.cpp
! src/share/vm/runtime/sharedRuntime.hpp
! src/share/vm/runtime/vframeArray.cpp

Changeset: d8e270c4f609
Author:    twisti
Date:      2010-03-09 23:57 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/d8e270c4f609

Merge


Changeset: c466efa608d5
Author:    roland
Date:      2010-03-05 13:58 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/c466efa608d5

6932496: c1: deoptimization of jsr subroutine fails on sparcv9
Summary: store jsr ret bci as intptr constant in c1 debug info
Reviewed-by: never

! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp
! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp
! src/share/vm/c1/c1_LIR.cpp
! src/share/vm/c1/c1_LIR.hpp
! src/share/vm/c1/c1_LinearScan.cpp
+ test/compiler/6932496/Test6932496.java

Changeset: da06d1795d84
Author:    twisti
Date:      2010-03-11 05:09 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/da06d1795d84

6934089: Zero 32-bit/64kb page fix
Summary: The fix for 6927165 increased the number of shadow pages for 32-bit platforms and this causes a problem on systems with 64kb pages.
Reviewed-by: twisti
Contributed-by: Gary Benson <gbenson at redhat.com>

! src/os_cpu/linux_zero/vm/globals_linux_zero.hpp

Changeset: 9eba43136cb5
Author:    twisti
Date:      2010-03-16 11:52 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/9eba43136cb5

6934494: JSR 292 MethodHandles adapters should be generated into their own CodeBlob
Summary: Passing a null pointer to an InvokeDynamic function call should lead to a NullPointerException.
Reviewed-by: kvn, never

! src/cpu/sparc/vm/stubRoutines_sparc.hpp
! src/cpu/x86/vm/stubGenerator_x86_32.cpp
! src/cpu/x86/vm/stubGenerator_x86_64.cpp
! src/cpu/x86/vm/stubRoutines_x86_32.hpp
! src/cpu/x86/vm/stubRoutines_x86_64.hpp
! src/share/vm/code/codeBlob.cpp
! src/share/vm/code/codeBlob.hpp
! src/share/vm/includeDB_core
! src/share/vm/prims/methodHandles.cpp
! src/share/vm/prims/methodHandles.hpp
! src/share/vm/runtime/init.cpp
! src/share/vm/runtime/sharedRuntime.cpp
! src/share/vm/runtime/sharedRuntime.hpp

Changeset: 428a9c451986
Author:    kvn
Date:      2010-03-16 15:35 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/428a9c451986

6935466: new CodeCache flushing code is not guarded by the flag
Summary: Add missing guard.
Reviewed-by: never

! src/share/vm/compiler/compileBroker.cpp

Changeset: fc2c71045ada
Author:    twisti
Date:      2010-03-17 10:22 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/fc2c71045ada

6934966: JSR 292 add C1 logic for saved SP over MethodHandle calls
Summary: The logic for x86 C1 to save the SP over MH calls is pretty straight forward but SPARC handles that differently.
Reviewed-by: never, jrose

! src/cpu/sparc/vm/c1_FrameMap_sparc.hpp
! src/cpu/sparc/vm/c1_LIRAssembler_sparc.cpp
! src/cpu/x86/vm/c1_FrameMap_x86.hpp
! src/cpu/x86/vm/c1_LIRAssembler_x86.cpp
! src/share/vm/c1/c1_LIR.cpp
! src/share/vm/c1/c1_LIRAssembler.cpp
! src/share/vm/c1/c1_LIRAssembler.hpp

Changeset: 2484f4d6a54e
Author:    kvn
Date:      2010-03-17 10:47 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/2484f4d6a54e

6935535: String.indexOf() returns incorrect result on x86 with SSE4.2
Summary: Added missing counter decrement when substring search restarted.
Reviewed-by: twisti

! src/cpu/x86/vm/assembler_x86.cpp
+ test/compiler/6935535/Test.java

Changeset: c047da02984c
Author:    never
Date:      2010-03-17 16:40 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/c047da02984c

6930043: C2: SIGSEGV in javasoft.sqe.tests.lang.arr017.arr01702.arr01702.loop_forw(II)I
Reviewed-by: kvn

! src/share/vm/opto/loopTransform.cpp
! src/share/vm/opto/loopnode.hpp
+ test/compiler/6930043/Test6930043.java

Changeset: 76c1d7d13ec5
Author:    twisti
Date:      2010-03-18 09:56 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/76c1d7d13ec5

6932091: JSR 292 x86 code cleanup
Summary: Some code cleanups found during the JSR 292 SPARC port.
Reviewed-by: kvn, never

! src/cpu/x86/vm/methodHandles_x86.cpp
! src/cpu/x86/vm/templateTable_x86_32.cpp
! src/cpu/x86/vm/templateTable_x86_64.cpp
! src/share/vm/c1/c1_LIRGenerator.cpp
! src/share/vm/prims/methodHandles.hpp
! src/share/vm/runtime/arguments.cpp

Changeset: 97fe2cc98b1d
Author:    twisti
Date:      2010-03-18 06:36 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/97fe2cc98b1d

Merge


Changeset: 6c94fe3c8df3
Author:    trims
Date:      2010-03-18 16:06 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/6c94fe3c8df3

Merge


Changeset: 07226e9eab8f
Author:    trims
Date:      2010-03-18 17:21 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/07226e9eab8f

6936366: Fork HS17 to HS18 - renumber Major and build numbers of JVM
Summary: Update the Major and build numbers for HS18 fork
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: 3deb84ecd19d
Author:    trims
Date:      2010-03-23 20:28 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3deb84ecd19d

Added tag hs18-b01 for changeset 07226e9eab8f

! .hgtags

Changeset: e7e7e36ccdb5
Author:    trims
Date:      2010-03-23 20:37 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/e7e7e36ccdb5

Merge

! .hgtags

Changeset: 465c39e1fb46
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/465c39e1fb46

Added tag jdk7-b87 for changeset e7e7e36ccdb5

! .hgtags

Changeset: 2a1472c30599
Author:    jcoomes
Date:      2010-03-03 14:48 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/2a1472c30599

4396719: Mark Sweep stack overflow on deeply nested Object arrays
Summary: Use an explicit stack for object arrays and process them in chunks.
Reviewed-by: iveresov, apetrusenko

! src/share/vm/gc_implementation/g1/concurrentMark.hpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp
! src/share/vm/gc_implementation/g1/g1MarkSweep.cpp
! src/share/vm/gc_implementation/includeDB_gc_parallelScavenge
! src/share/vm/gc_implementation/parallelScavenge/pcTasks.cpp
! src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp
! src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.hpp
+ src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.inline.hpp
! src/share/vm/gc_implementation/parallelScavenge/psMarkSweep.cpp
! src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp
! src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.hpp
! src/share/vm/gc_implementation/shared/markSweep.cpp
! src/share/vm/gc_implementation/shared/markSweep.hpp
! src/share/vm/gc_implementation/shared/markSweep.inline.hpp
! src/share/vm/includeDB_core
! src/share/vm/includeDB_gc_parallel
! src/share/vm/memory/genMarkSweep.cpp
! src/share/vm/memory/genOopClosures.hpp
! src/share/vm/oops/objArrayKlass.cpp
! src/share/vm/oops/objArrayKlass.hpp
+ src/share/vm/oops/objArrayKlass.inline.hpp
! src/share/vm/runtime/arguments.cpp
! src/share/vm/runtime/globals.hpp
! src/share/vm/utilities/globalDefinitions.hpp
! src/share/vm/utilities/taskqueue.cpp
! src/share/vm/utilities/taskqueue.hpp

Changeset: 94946bdf36bd
Author:    apetrusenko
Date:      2010-03-15 02:56 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/94946bdf36bd

Merge


Changeset: 664ae0c5e0e5
Author:    johnc
Date:      2010-03-11 11:44 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/664ae0c5e0e5

6755988: G1: assert(new_obj != 0 || ... "should be forwarded")
Summary: A TLAB became large enough to be considered a humongous object allowing multiple objects to be allocated in a humongous region, which violates a basic assumption about humongous regions. The changes ensure that TLABs cannot be regarded as humongous.
Reviewed-by: iveresov, tonyp

! src/share/vm/gc_implementation/g1/g1CollectedHeap.cpp
! src/share/vm/gc_implementation/g1/g1CollectedHeap.hpp

Changeset: 3f0549ed0c98
Author:    apetrusenko
Date:      2010-03-18 01:48 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3f0549ed0c98

6921710: G1: assert(new_finger >= _finger && new_finger < _region_limit,"invariant")
Summary: If CM task was aborted while scanning the last object of the specified region and the size of that object is equal to bitmap's granularity then the next offset would be equal or over the region limit which is exactly what the assertion states.
Reviewed-by: ysr, tonyp, jmasa

! src/share/vm/gc_implementation/g1/concurrentMark.cpp

Changeset: c385bf94cfb8
Author:    jcoomes
Date:      2010-03-18 13:31 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/c385bf94cfb8

6935839: excessive marking stack growth during full gcs
Summary: process one item at a time from the objarray stack/queue
Reviewed-by: apetrusenko, tonyp

! src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp
! src/share/vm/gc_implementation/shared/markSweep.cpp

Changeset: cc98cc548f51
Author:    apetrusenko
Date:      2010-03-22 02:40 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/cc98cc548f51

Merge

! src/share/vm/includeDB_core
! src/share/vm/runtime/arguments.cpp

Changeset: d4197f8d516a
Author:    tonyp
Date:      2010-03-18 12:14 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/d4197f8d516a

6935821: G1: threads created during marking do not active their SATB queues
Summary: Newly-created threads always had the active field of their SATB queue initialized to false, even if they were created during marking. As a result, updates from threads created during a marking cycle were never enqueued and never processed. The fix includes remaining a method from active() to is_active() for readability and naming consistency.
Reviewed-by: ysr, johnc

! src/share/vm/gc_implementation/g1/concurrentMark.cpp
! src/share/vm/gc_implementation/g1/g1SATBCardTableModRefBS.cpp
! src/share/vm/gc_implementation/g1/ptrQueue.cpp
! src/share/vm/gc_implementation/g1/ptrQueue.hpp
! src/share/vm/gc_implementation/g1/satbQueue.cpp
! src/share/vm/gc_implementation/g1/satbQueue.hpp

Changeset: 5c9df1575c39
Author:    trims
Date:      2010-04-01 16:10 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/5c9df1575c39

Merge


Changeset: 096b18156d91
Author:    trims
Date:      2010-04-01 16:15 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/096b18156d91

6940419: Bump the HS18 build number to 02
Summary: Update the HS18 build number to 02
Reviewed-by: jcoomes

! make/hotspot_version

Changeset: 3b3d12e645e7
Author:    coleenp
Date:      2010-03-12 10:42 -0500
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3b3d12e645e7

6929067: Stack guard pages should be removed when thread is detached
Summary: Add code to unmap stack guard area when thread is detached.
Reviewed-by: coleenp, kamg

! src/os/linux/vm/os_linux.cpp
! src/os/solaris/vm/os_solaris.cpp
! src/os/windows/vm/os_windows.cpp
! src/share/vm/runtime/os.hpp
! src/share/vm/runtime/thread.cpp
+ test/runtime/6929067/T.java
+ test/runtime/6929067/Test6929067.sh
+ test/runtime/6929067/invoke.c

Changeset: 0f6600cee529
Author:    xlu
Date:      2010-03-13 16:32 -0800
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/0f6600cee529

6934758: Expose the break down of clean up task time during safepoint.
Summary: Use -XX:+TraceSafepointCleanupTime to print out the details of each clean up tasks.
Reviewed-by: dholmes, ysr

! src/share/vm/runtime/globals.hpp
! src/share/vm/runtime/safepoint.cpp

Changeset: 21141e23627a
Author:    dcubed
Date:      2010-03-16 17:47 -0600
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/21141e23627a

6923488: 4/4 need minor tweaks to HotSpot build for Cygwin
Summary: Clean up get_msc_ver, build.bat and build.make in general and for Cygwin.
Reviewed-by: ohair

! make/windows/build.bat
! make/windows/build.make
! make/windows/get_msc_ver.sh

Changeset: cd20a6f46fec
Author:    dcubed
Date:      2010-03-16 20:47 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/cd20a6f46fec

Merge


Changeset: e392695de029
Author:    fparain
Date:      2010-03-17 11:01 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/e392695de029

6935224: Adding new DTrace probes to work with Palantir
Summary: Adding probes related to thread scheduling and class initialization
Reviewed-by: kamg, never

! src/os/solaris/dtrace/hotspot.d
! src/os/solaris/vm/attachListener_solaris.cpp
! src/share/vm/includeDB_core
! src/share/vm/oops/instanceKlass.cpp
! src/share/vm/prims/jvm.cpp
! src/share/vm/prims/unsafe.cpp
! src/share/vm/runtime/vmThread.cpp
! src/share/vm/services/dtraceAttacher.cpp
! src/share/vm/services/dtraceAttacher.hpp

Changeset: 98ba8ca25feb
Author:    coleenp
Date:      2010-03-18 16:47 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/98ba8ca25feb

6936168: Recent fix for unmapping stack guard pages doesn't close /proc/self/maps
Summary: Add close to returns (fix for 6929067 also contributed by aph)
Reviewed-by: aph, dcubed, andrew, acorn
Contributed-by: aph at redhat.com, andreas.kohn at fredhopper.com

! src/os/linux/vm/os_linux.cpp

Changeset: 4f7af0dc447b
Author:    dcubed
Date:      2010-03-23 14:37 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/4f7af0dc447b

6915365: 3/4 assert(false,"Unsupported VMGlobal Type") at management.cpp:1540
Summary: Remove assert to decouple JDK and HotSpot additions of known types.
Reviewed-by: mchung

! src/share/vm/services/management.cpp

Changeset: 5d393243d487
Author:    dcubed
Date:      2010-03-23 17:29 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/5d393243d487

Merge


Changeset: 39e409a664b3
Author:    dcubed
Date:      2010-03-25 16:27 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/39e409a664b3

6938185: 3/4 6923488 breaks Windows command shell builds
Summary: Fix build.bat so invoking command shell doesn't exit on error. Fix dirname assumptions. Document some MKS environment dependencies.
Reviewed-by: coleenp

! make/windows/build.bat
! make/windows/build.make
! make/windows/create.bat
! make/windows/get_msc_ver.sh

Changeset: 84043c7507b9
Author:    dcubed
Date:      2010-03-25 16:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/84043c7507b9

Merge


Changeset: 4a9cc99938e3
Author:    acorn
Date:      2010-03-26 11:10 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/4a9cc99938e3

Merge

! src/cpu/x86/vm/methodHandles_x86.cpp
! src/share/vm/includeDB_core
! src/share/vm/runtime/globals.hpp

Changeset: 7c358fbb6a84
Author:    acorn
Date:      2010-04-01 11:23 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/7c358fbb6a84

Merge


Changeset: 4b60f23c4223
Author:    acorn
Date:      2010-04-01 20:48 -0400
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/4b60f23c4223

Merge


Changeset: 3f12a94552cc
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/3f12a94552cc

Added tag jdk7-b88 for changeset 4b60f23c4223

! .hgtags

Changeset: 5b29c2368d93
Author:    lana
Date:      2010-04-08 15:28 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/5b29c2368d93

Merge

! src/share/vm/opto/type.cpp

Changeset: 15836273ac24
Author:    lana
Date:      2010-04-13 16:36 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/15836273ac24

Merge


Changeset: 765578777b6e
Author:    mikejwre
Date:      2010-04-15 13:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/hotspot/rev/765578777b6e

Added tag jdk7-b89 for changeset 15836273ac24

! .hgtags



From lana.steuck at sun.com  Sat Apr 17 22:19:17 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:19:17 +0000
Subject: hg: jdk7/tl/jaxp: 4 new changesets
Message-ID: <20100417221918.0B97F445EA@hg.openjdk.java.net>

Changeset: 8b493f1aa136
Author:    mikejwre
Date:      2010-03-18 13:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxp/rev/8b493f1aa136

Added tag jdk7-b86 for changeset 81c0f115bbe5

! .hgtags

Changeset: d8ebd1591003
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxp/rev/d8ebd1591003

Added tag jdk7-b87 for changeset 8b493f1aa136

! .hgtags

Changeset: d2818fd2b036
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxp/rev/d2818fd2b036

Added tag jdk7-b88 for changeset d8ebd1591003

! .hgtags

Changeset: c5d932ee326d
Author:    mikejwre
Date:      2010-04-15 13:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxp/rev/c5d932ee326d

Added tag jdk7-b89 for changeset d2818fd2b036

! .hgtags



From lana.steuck at sun.com  Sat Apr 17 22:19:23 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:19:23 +0000
Subject: hg: jdk7/tl/jaxws: 4 new changesets
Message-ID: <20100417221923.842D7445EB@hg.openjdk.java.net>

Changeset: 3febd6fab2ac
Author:    mikejwre
Date:      2010-03-18 13:52 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxws/rev/3febd6fab2ac

Added tag jdk7-b86 for changeset 512b0e924a5a

! .hgtags

Changeset: 8c666f8f3565
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxws/rev/8c666f8f3565

Added tag jdk7-b87 for changeset 3febd6fab2ac

! .hgtags

Changeset: bf3675aa7f20
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxws/rev/bf3675aa7f20

Added tag jdk7-b88 for changeset 8c666f8f3565

! .hgtags

Changeset: ead7c4566a00
Author:    mikejwre
Date:      2010-04-15 13:54 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jaxws/rev/ead7c4566a00

Added tag jdk7-b89 for changeset bf3675aa7f20

! .hgtags



From lana.steuck at sun.com  Sat Apr 17 22:20:50 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:20:50 +0000
Subject: hg: jdk7/tl/jdk: 21 new changesets
Message-ID: <20100417222718.A7AC2445EC@hg.openjdk.java.net>

Changeset: 2cafbbe9825e
Author:    mikejwre
Date:      2010-03-18 13:53 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/2cafbbe9825e

Added tag jdk7-b86 for changeset eae6e9ab2606

! .hgtags

Changeset: b3c69282f6d3
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/b3c69282f6d3

Added tag jdk7-b87 for changeset 2cafbbe9825e

! .hgtags

Changeset: b50cfd4479fa
Author:    mikejwre
Date:      2010-04-08 17:02 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/b50cfd4479fa

Added tag jdk7-b88 for changeset b3c69282f6d3

! .hgtags

Changeset: 0137b5857c63
Author:    okutsu
Date:      2010-03-10 14:32 +0900
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0137b5857c63

6932473: (tz) javazic produces incorrect SimpleTimeZone parameters with Sun<=7
Reviewed-by: peytoia

! make/tools/src/build/tools/javazic/RuleDay.java

Changeset: 0e0ce1aa1bba
Author:    peytoia
Date:      2010-03-11 11:54 +0900
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0e0ce1aa1bba

6933032: (tz) Support tzdata2010e
Reviewed-by: okutsu

! make/sun/javazic/tzdata/VERSION
! make/sun/javazic/tzdata/asia
! make/sun/javazic/tzdata/australasia
! make/sun/javazic/tzdata/southamerica

Changeset: ce3770eadf85
Author:    malenkov
Date:      2010-03-11 17:39 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ce3770eadf85

6707226: java.beans.Statement & java.beans.Expression miss one important usecase
Reviewed-by: rupashka

! src/share/classes/java/beans/Expression.java
! src/share/classes/java/beans/Statement.java
+ test/java/beans/Statement/Test6707226.java

Changeset: d86c053ca938
Author:    rupashka
Date:      2010-03-15 16:16 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/d86c053ca938

6931347: SynthTabbedPaneUI.paintTabArea() is not called when tabbed pane is painted
Reviewed-by: peterz

! src/share/classes/javax/swing/plaf/synth/SynthTabbedPaneUI.java

Changeset: 1224c1388e86
Author:    rupashka
Date:      2010-03-17 12:48 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/1224c1388e86

6933784: NIMBUS: ImageView getNoImageIcon and getLoadingImageIcon returns nulls instead of an icon
Reviewed-by: peterz

! src/share/classes/javax/swing/plaf/synth/SynthLookAndFeel.java
+ test/javax/swing/plaf/synth/Test6933784.java

Changeset: ac4c8e3bf93f
Author:    lana
Date:      2010-03-17 14:19 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ac4c8e3bf93f

Merge

- test/java/nio/file/WatchService/OverflowEventIsLoner.java

Changeset: 325823a26aac
Author:    peterz
Date:      2010-03-18 12:02 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/325823a26aac

6932524: NIMBUS: 3 constructors of JSplitPane creates new jsp with continuous layout - they should not.
Reviewed-by: alexp

! src/share/classes/javax/swing/JSplitPane.java
! src/share/classes/javax/swing/plaf/nimbus/skin.laf

Changeset: ef892cd084ec
Author:    rupashka
Date:      2010-03-24 15:14 +0300
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/ef892cd084ec

6922214: SynthTextPaneUI.installUI() doesn't install the default caret and highlighter
Reviewed-by: alexp

! src/share/classes/javax/swing/plaf/basic/BasicTextUI.java

Changeset: f799c62ad4f8
Author:    peytoia
Date:      2010-03-30 18:35 +0900
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/f799c62ad4f8

6939021: (tz) Support tzdata2010g
Reviewed-by: okutsu

! make/sun/javazic/tzdata/VERSION
! make/sun/javazic/tzdata/antarctica
! make/sun/javazic/tzdata/asia
! make/sun/javazic/tzdata/australasia
! make/sun/javazic/tzdata/europe
! make/sun/javazic/tzdata/zone.tab
! src/share/classes/sun/util/resources/TimeZoneNames.java

Changeset: dbde35ddda78
Author:    peytoia
Date:      2010-03-30 21:16 +0900
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/dbde35ddda78

6907881: Different undesired behavior for entering Asian characters in Windows IME starting with Java 6.0
Reviewed-by: okutsu

! src/windows/native/sun/windows/awt_Component.cpp

Changeset: eb39ccbd95f9
Author:    lana
Date:      2010-04-08 15:00 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/eb39ccbd95f9

Merge


Changeset: d23dcd3e3ce4
Author:    andrew
Date:      2010-03-12 01:09 +0000
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/d23dcd3e3ce4

6934327: Update linux fontconfigs for Ubuntu and Fedora.
Summary: Use fontconfigs suitable for recent Fedora, Ubuntu and Debian releases.
Reviewed-by: prr

! src/solaris/classes/sun/awt/fontconfigs/linux.fontconfig.Fedora.properties
! src/solaris/classes/sun/awt/fontconfigs/linux.fontconfig.Ubuntu.properties

Changeset: 45bd445f6250
Author:    lana
Date:      2010-03-19 18:49 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/45bd445f6250

Merge

- make/java/text/FILES_java.gmk
- test/java/nio/file/WatchService/OverflowEventIsLoner.java

Changeset: bf23bec88222
Author:    lana
Date:      2010-04-08 15:01 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/bf23bec88222

Merge


Changeset: 025f9e57566a
Author:    lana
Date:      2010-04-08 15:34 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/025f9e57566a

Merge

- make/tools/src/build/tools/charsetmapping/CharsetMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateDBCS.java
- make/tools/src/build/tools/charsetmapping/GenerateEUC_TW.java
- make/tools/src/build/tools/charsetmapping/GenerateMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateSBCS.java
! src/share/classes/java/beans/Statement.java
- src/share/classes/sun/io/ByteToCharHKSCS.java
- src/share/classes/sun/io/ByteToCharHKSCS_2001.java
- src/share/classes/sun/io/CharToByteHKSCS.java
- src/share/classes/sun/io/CharToByteHKSCS_2001.java
- src/share/classes/sun/nio/cs/ext/Big5.java
- src/share/classes/sun/nio/cs/ext/HKSCS_2001.java
- test/java/net/Socket/FDClose.java
- test/lib/security/cacerts/VerifyCACerts.java

Changeset: 4a6abb7e224c
Author:    lana
Date:      2010-04-13 16:41 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/4a6abb7e224c

Merge

- make/tools/src/build/tools/charsetmapping/CharsetMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateDBCS.java
- make/tools/src/build/tools/charsetmapping/GenerateEUC_TW.java
- make/tools/src/build/tools/charsetmapping/GenerateMapping.java
- make/tools/src/build/tools/charsetmapping/GenerateSBCS.java
- src/share/classes/sun/io/ByteToCharHKSCS.java
- src/share/classes/sun/io/ByteToCharHKSCS_2001.java
- src/share/classes/sun/io/CharToByteHKSCS.java
- src/share/classes/sun/io/CharToByteHKSCS_2001.java
- src/share/classes/sun/nio/cs/ext/Big5.java
- src/share/classes/sun/nio/cs/ext/HKSCS_2001.java
- test/java/net/Socket/FDClose.java
- test/lib/security/cacerts/VerifyCACerts.java

Changeset: 7f90d0b9dbb7
Author:    mikejwre
Date:      2010-04-15 13:55 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/7f90d0b9dbb7

Added tag jdk7-b89 for changeset 4a6abb7e224c

! .hgtags

Changeset: 7ecc023957d8
Author:    lana
Date:      2010-04-17 08:12 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/7ecc023957d8

Merge




From lana.steuck at sun.com  Sat Apr 17 22:32:39 2010
From: lana.steuck at sun.com (lana.steuck at sun.com)
Date: Sat, 17 Apr 2010 22:32:39 +0000
Subject: hg: jdk7/tl/langtools: 7 new changesets
Message-ID: <20100417223258.D9E59445EE@hg.openjdk.java.net>

Changeset: 409db93d19c0
Author:    mikejwre
Date:      2010-03-18 13:53 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/409db93d19c0

Added tag jdk7-b86 for changeset ef07347428f2

! .hgtags

Changeset: f9b5d4867a26
Author:    mikejwre
Date:      2010-03-25 15:05 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/f9b5d4867a26

Added tag jdk7-b87 for changeset 409db93d19c0

! .hgtags

Changeset: 737185f3300f
Author:    mikejwre
Date:      2010-04-08 17:03 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/737185f3300f

Added tag jdk7-b88 for changeset f9b5d4867a26

! .hgtags

Changeset: ad1bf317cc57
Author:    lana
Date:      2010-04-08 15:35 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/ad1bf317cc57

Merge

- test/tools/javap/T6305779.java

Changeset: 6cea9a143208
Author:    lana
Date:      2010-04-13 16:42 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/6cea9a143208

Merge

- test/tools/javap/T6305779.java

Changeset: 71c2c23a7c35
Author:    mikejwre
Date:      2010-04-15 13:55 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/71c2c23a7c35

Added tag jdk7-b89 for changeset 6cea9a143208

! .hgtags

Changeset: 37fa8cd046ab
Author:    lana
Date:      2010-04-17 08:12 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/37fa8cd046ab

Merge




From mnachev at gmail.com  Sun Apr 18 07:23:26 2010
From: mnachev at gmail.com (Miroslav Nachev)
Date: Sun, 18 Apr 2010 10:23:26 +0300
Subject: Support of USB, Serial (RS-232), Parallel (LPT) and WDT
Message-ID: <s2zf2db0d271004180023o2bb5e0cev9acb1d6c86460f0f@mail.gmail.com>

Hi,

I want to know whether there are plans JDK7 to support USB, Serial (RS-232),
Parallel (LPT) and WatchDog Timer (WDT)?


Regards,
Miro.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100418/79678302/attachment.html>

From uncle.alice at gmail.com  Mon Apr 19 10:23:48 2010
From: uncle.alice at gmail.com (Alan Moore)
Date: Mon, 19 Apr 2010 03:23:48 -0700
Subject: Wording improvements for String.indexOf, String.lastIndexOf
In-Reply-To: <p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>
References: <s2j1ccfd1c11003311606je42e5119z27f975960c2747f9@mail.gmail.com>
	<4BB5119E.7010208@oracle.com>
	<p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>
Message-ID: <w2zb5b21c661004190323zd4d15b20jded9041dac480b4e@mail.gmail.com>

On Thu, Apr 1, 2010 at 3:36 PM, Martin Buchholz <martinrb at google.com> wrote:
> On Thu, Apr 1, 2010 at 14:35, Xueming Shen <xueming.shen at oracle.com> wrote:
>> Seems like the "is <code>true</code>." line is missing from the new wording
>> (in
>> all cases), is it purposely? Not a native, but I would assume we still need
>> those 2 words to make the sentence complete.
>
> It is true that "is true"
> would make it a grammatically complete sentence,
> but I think it's best to not be quite that pedantic and lawyerly.
> In general, the JDK makes a pretty good compromise between
> precision and readability.
I think Sherman has a good point, except the wording should be
"returns true" instead of "is true" (with, of course, "true" formatted
to indicate that it's a boolean literal, not an English word).  That
would make it consistent with methods that return other-than-boolean
values, as well as being considerate of non-native English speakers
who see a method name like "startsWith" and _don't_ automatically read
it as the English phrase "starts with".


From Michael.McMahon at Sun.COM  Mon Apr 19 16:04:36 2010
From: Michael.McMahon at Sun.COM (Michael McMahon)
Date: Mon, 19 Apr 2010 17:04:36 +0100
Subject: UNIXProcess improvements
In-Reply-To: <r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
Message-ID: <4BCC7F14.6080600@sun.com>

Martin Buchholz wrote:
> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
>
>   
>> For now I suggest leaving old @author tags as-is.
>>     
>
> OK, done.
>
> Version 0.2 of the webrev is published.
>
> Martin
>   
Martin,

 From what I can see, you've cleaned up the code and the functional changes
are the use of a thread pool, and an explicit (8 k sized) stack.

Also, the threads created now belong to the root thread group rather than
the application's thread group. Is this so you can handle uncaught 
exceptions
as you mentioned before, and if so, I guess some other change is coming 
to complete
this work. Is that right?

Thanks,
Michael



From martinrb at google.com  Mon Apr 19 20:06:31 2010
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 19 Apr 2010 13:06:31 -0700
Subject: UNIXProcess improvements
In-Reply-To: <4BCC7F14.6080600@sun.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
Message-ID: <h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>

On Mon, Apr 19, 2010 at 09:04, Michael McMahon <Michael.McMahon at sun.com> wrote:
> Martin Buchholz wrote:
>>
>> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
>>
>>
>>>
>>> For now I suggest leaving old @author tags as-is.
>>>
>>
>> OK, done.
>>
>> Version 0.2 of the webrev is published.
>>
>> Martin
>>
>
> Martin,
>
> From what I can see, you've cleaned up the code and the functional changes
> are the use of a thread pool, and an explicit (8 k sized) stack.

Exceptions thrown in the "process reaper" thread,
which were not IOExceptions,
were not being caught, and would cause the user thread to hang.

> Also, the threads created now belong to the root thread group rather than
> the application's thread group.

Well, they have to belong to some thread group,
and they get reused, so in general the thread group will have
no relation to the thread group of the user thread,
so may as well use the root thread group.
I stole this technique from elsewhere in the JDK.

> Is this so you can handle uncaught
> exceptions
> as you mentioned before, and if so, I guess some other change is coming to
> complete
> this work. Is that right?

Yes.  This change by itself is a clear win,
except that because it is more memory efficient,
GC is less likely to get called,
which means file descriptors of defunct processes
are less likely to get cleaned up in the face of
lazy user code, which means it is more subject
to file descriptor exhaustion problems.
Which I would like to fix.

Martin

> Thanks,
> Michael
>
>


From martinrb at google.com  Mon Apr 19 22:56:22 2010
From: martinrb at google.com (Martin Buchholz)
Date: Mon, 19 Apr 2010 15:56:22 -0700
Subject: Wording improvements for String.indexOf, String.lastIndexOf
In-Reply-To: <w2zb5b21c661004190323zd4d15b20jded9041dac480b4e@mail.gmail.com>
References: <s2j1ccfd1c11003311606je42e5119z27f975960c2747f9@mail.gmail.com>
	<4BB5119E.7010208@oracle.com>
	<p2i1ccfd1c11004011536s7b11db73w5ff94b205e4373db@mail.gmail.com>
	<w2zb5b21c661004190323zd4d15b20jded9041dac480b4e@mail.gmail.com>
Message-ID: <i2s1ccfd1c11004191556o99661265jf51ddf8242b87fe9@mail.gmail.com>

Alan, thanks for the comments.

"returns true" works well for expressions
containing exactly one method call,
but e.g. in indexOf(String str, int fromIndex)
it is a complex expression, where Sherman's "is true"
would be more correct.  Because of this, I'm still
inclined not to change this.

http://cr.openjdk.java.net/~martin/webrevs/openjdk7/IndexOf-javadoc/

Martin

On Mon, Apr 19, 2010 at 03:23, Alan Moore <uncle.alice at gmail.com> wrote:
> On Thu, Apr 1, 2010 at 3:36 PM, Martin Buchholz <martinrb at google.com> wrote:
>> On Thu, Apr 1, 2010 at 14:35, Xueming Shen <xueming.shen at oracle.com> wrote:
>>> Seems like the "is <code>true</code>." line is missing from the new wording
>>> (in
>>> all cases), is it purposely? Not a native, but I would assume we still need
>>> those 2 words to make the sentence complete.
>>
>> It is true that "is true"
>> would make it a grammatically complete sentence,
>> but I think it's best to not be quite that pedantic and lawyerly.
>> In general, the JDK makes a pretty good compromise between
>> precision and readability.
> I think Sherman has a good point, except the wording should be
> "returns true" instead of "is true" (with, of course, "true" formatted
> to indicate that it's a boolean literal, not an English word). ?That
> would make it consistent with methods that return other-than-boolean
> values, as well as being considerate of non-native English speakers
> who see a method name like "startsWith" and _don't_ automatically read
> it as the English phrase "starts with".
>


From weijun.wang at sun.com  Tue Apr 20 11:31:47 2010
From: weijun.wang at sun.com (weijun.wang at sun.com)
Date: Tue, 20 Apr 2010 11:31:47 +0000
Subject: hg: jdk7/tl/jdk: 6944847: native gss lib names on linux
Message-ID: <20100420113239.C9A5044628@hg.openjdk.java.net>

Changeset: 97fb6f6d230a
Author:    weijun
Date:      2010-04-20 19:30 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/97fb6f6d230a

6944847: native gss lib names on linux
Reviewed-by: valeriep

! src/share/classes/sun/security/jgss/wrapper/SunNativeProvider.java
+ test/sun/security/krb5/Krb5NameEquals.java
+ test/sun/security/krb5/runNameEquals.sh



From Michael.McMahon at Sun.COM  Tue Apr 20 15:58:06 2010
From: Michael.McMahon at Sun.COM (Michael McMahon)
Date: Tue, 20 Apr 2010 16:58:06 +0100
Subject: UNIXProcess improvements
In-Reply-To: <h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
Message-ID: <4BCDCF0E.5090604@sun.com>

Martin,

Thanks for the answers. The changes look fine to me.

- Michael.

Martin Buchholz wrote:
> On Mon, Apr 19, 2010 at 09:04, Michael McMahon <Michael.McMahon at sun.com> wrote:
>   
>> Martin Buchholz wrote:
>>     
>>> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
>>>
>>>
>>>       
>>>> For now I suggest leaving old @author tags as-is.
>>>>
>>>>         
>>> OK, done.
>>>
>>> Version 0.2 of the webrev is published.
>>>
>>> Martin
>>>
>>>       
>> Martin,
>>
>> From what I can see, you've cleaned up the code and the functional changes
>> are the use of a thread pool, and an explicit (8 k sized) stack.
>>     
>
> Exceptions thrown in the "process reaper" thread,
> which were not IOExceptions,
> were not being caught, and would cause the user thread to hang.
>
>   
>> Also, the threads created now belong to the root thread group rather than
>> the application's thread group.
>>     
>
> Well, they have to belong to some thread group,
> and they get reused, so in general the thread group will have
> no relation to the thread group of the user thread,
> so may as well use the root thread group.
> I stole this technique from elsewhere in the JDK.
>
>   
>> Is this so you can handle uncaught
>> exceptions
>> as you mentioned before, and if so, I guess some other change is coming to
>> complete
>> this work. Is that right?
>>     
>
> Yes.  This change by itself is a clear win,
> except that because it is more memory efficient,
> GC is less likely to get called,
> which means file descriptors of defunct processes
> are less likely to get cleaned up in the face of
> lazy user code, which means it is more subject
> to file descriptor exhaustion problems.
> Which I would like to fix.
>
> Martin
>
>   
>> Thanks,
>> Michael
>>
>>
>>     



From bradford.wetmore at oracle.com  Tue Apr 20 21:25:32 2010
From: bradford.wetmore at oracle.com (bradford.wetmore at oracle.com)
Date: Tue, 20 Apr 2010 21:25:32 +0000
Subject: hg: jdk7/tl/jdk: 6945604: wrong error message in CardImpl.java
Message-ID: <20100420212638.5B2D14462F@hg.openjdk.java.net>

Changeset: d8ad2da3ecf3
Author:    wetmore
Date:      2010-04-20 14:24 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/d8ad2da3ecf3

6945604: wrong error message in CardImpl.java
Reviewed-by: mullan

! src/share/classes/sun/security/smartcardio/CardImpl.java



From maurizio.cimadamore at sun.com  Wed Apr 21 11:25:00 2010
From: maurizio.cimadamore at sun.com (maurizio.cimadamore at sun.com)
Date: Wed, 21 Apr 2010 11:25:00 +0000
Subject: hg: jdk7/tl/langtools: 6730476: invalid "unchecked generic array"
	warning
Message-ID: <20100421112508.4FF4C44640@hg.openjdk.java.net>

Changeset: 04cf82179fa7
Author:    mcimadamore
Date:      2010-04-21 12:24 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/04cf82179fa7

6730476: invalid "unchecked generic array" warning
Summary: Reifiable-ness of varargs element type should be checked after JLS3 15.12.2.8
Reviewed-by: jjg

! src/share/classes/com/sun/tools/javac/comp/Attr.java
! src/share/classes/com/sun/tools/javac/comp/Check.java
! src/share/classes/com/sun/tools/javac/comp/Infer.java
! src/share/classes/com/sun/tools/javac/comp/Resolve.java
+ test/tools/javac/varargs/6730476/T6730476a.java
+ test/tools/javac/varargs/6730476/T6730476b.java



From kevin.l.stern at gmail.com  Wed Apr 21 22:37:23 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Wed, 21 Apr 2010 17:37:23 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
Message-ID: <w2n1704b7a21004211537m9affedd6u1347f5f8750825de@mail.gmail.com>

Hi Benedict,

Please forgive my delayed response.  I've been working on a new
implementation to offer support for the deque interface per Martin's
suggestion.  I have this implementation here
https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfNGY4NWhzM2Q4&hl=enin
case you are interested in taking a look.

Regards,

Kevin

On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith
<lists at laerad.com>wrote:

> Hi Kevin,
>
> As it happens I might have something useful still to contribute. As an
> exercise in saving face I revisited the problem to see if I could achieve
> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
> must admit I still didn't fully appreciate how the algorithm in
> ChunkedArrayList worked until I tried to come up with an algorithm with
> similar properties. What I have ended up with is almost identical except
> adds I think a couple of incremental improvements, simply by redefining the
> arrayIndex() method. I should note that I have not yet implemented more than
> a prototype as it seems to me your implementation is excellent already, and
> if it is decided to include my modifications the changes should be modest.
>
> Firstly, (I hope that) what I have produced is a little more CPU pipe-line
> friendly; there is less dependency on immediately preceding calculations at
> each stage (i.e. so more operations should be able to proceed simultaneously
> in the pipeline), and consists exclusively of shifts, addition/subtraction
> and bit-wise (&)ands (except for the conditionals in
> Integer.numberOfLeadingZeros(i)), although the total number of instructions
> is approximately the same.
>
> Secondly, I have modified the algorithm so that a "seed" size can be
> specified (although I expect hard coding a suitable one will ultimately be
> best). Whereas ChunkedArrayList currently requires that the pattern of array
> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
> 16*s*(..*24)] etc. although when put in simple text like that it does
> appear to trivialise the change. The benefit of this, though, is two fold:
> 1) for small n the constant factor is reduced (both CPU and memory wise);
> and 2) the sqrt(n) bounds are reached more quickly also.
>
> As an illustration, consider setting *s* to 4, and assume the backing
> array is size two and doubles in size with each growth; with
> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as 4 we
> would instead resize at i=8,i=24,i=80,i=288; the cost at each would be some
> multiple of 2,4,8,16 respectively. As you can see the latter is much closer
> to the sqrt(n) cost - both approach it eventually, but my suggestion is to
> reach it more quickly. This is at the expense of more slowly reaching the
> sqrt(n) wasted memory condition, but given the high constant factor cost wrt
> to memory at this early stage, this seems a very sensible trade off. It
> seems likely this should also have a positive impact on cache performance
> for smaller lists as well.
>
> Finally, after playing with this idea in my head I am confident I can
> extend the core ideas of this data structure to hashing relatively easily,
> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
> wasted memory guarantees. I notice that this case hasn't been addressed yet,
> although I see from Martin's recent mail that this was raised before. Unless
> there are better suggestions for solving the hash table problem I will have
> a go at it as it seems an interesting problem - that is, assuming there are
> no objections?
>
> I'm interested to hear your thoughts. I hope this time I've been a bit more
> considered in what I've put forward, and hence less of a waste of time!
>
> Code snippet for calculation of array index and item offset:
>
> final int arraySizeShiftMinusSeed = ((31 -
> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3) -
> (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>  final int indexRemainder = index - ((1 << seed) <<
> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>
> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>
> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* +
> 1); seed only works for values for 2 or more at this moment, fyi
>
>
>
> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>> know if you have any other questions or if you have any related ideas or
>> concerns.
>>
>>
>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <lists at laerad.com
>> > wrote:
>>
>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>> help. that part of the discussion was clearly truncated by the time I had
>>> joined the list - I haven't been able to find the history in the archives
>>> either...
>>>
>>> I was just wondering about the worst case cost of add() as described by
>>> your javadoc; admittedly it is optimal with respect to unused memory, but
>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>> constant factor. I had been thinking that once n passed a threshold the cost
>>> of additions in this other structure would become a constant factor,
>>> offering nice algorithmic complexity guarantees for large n; however since
>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>> allocations would have to be unrealistically small (assuming linear cost for
>>> allocation) for this to be the case. It would still be nice to have a data
>>> structure that avoids needing to copy data with each grow, whilst still
>>> maintaining good memory performance.
>>>
>>> That *all* being said, I had been going by your javadoc and emails to
>>> ascertain the behaviour of this class, as I couldn't locate a free copy
>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>> than with what I assumed to be copying data between arraylets, and it seems
>>> this cost is pretty optimal. That will teach me to post to a list without
>>> getting my facts straight first. The interesting thing is simply that the
>>> constant factor for this implementation still seems to be quite high,
>>> although perhaps that is simply because I was not benchmarking sufficiently
>>> large values of n.
>>>
>>>
>>>
>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Hi Benedict,
>>>>
>>>> Unless I am misreading your post, this now has a very similar feel to
>>>> the first data structure that I posted to the list.  Martin Buchholz then
>>>> pointed out that we can incorporate the ideas from
>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>> addressed the problem that was originally being visited; only reduced the
>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>> power of 2), after which the array is increased in size linearly.
>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>> been exhaustive in the benchmarking:
>>>>>
>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>
>>>>> The nice thing about this is that the maximum amount of wasted memory
>>>>> is user configurable. Even with a low setting as above (65K) performance
>>>>> seems pretty consistent.
>>>>>
>>>>> Code for calculating index and array offset are pretty straight
>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>
>>>>> private final int indexFor(int a, int i) {
>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>> maxArraySizeShift : 1 << a) ;
>>>>>  }
>>>>> private final int arrayFor(int i) {
>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>  }
>>>>>
>>>>> Regarding the double list idea - yes, I agree, I certainly didn't think
>>>>> that one through fully!
>>>>>
>>>>>
>>>>>
>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Hi Benedict,
>>>>>>
>>>>>> Like you, I am relatively new to this mailing list; I am also trying
>>>>>> to tread lightly so as not to step on any toes.  That being said, I think
>>>>>> that I can offer a response to your inquiry.
>>>>>>
>>>>>> Regarding:  "The idea is to simply double the new array size each time
>>>>>> a new array needs to be allocated"
>>>>>>
>>>>>> It seems this would not address the desire of offering an alternative
>>>>>> to the allocation of a large backing array for ArrayList (your largest
>>>>>> backing array could still reach a size of 1/2 * Integer.MAX_VALUE) and would
>>>>>> not address the desire of wasting the (asymptotically) minimum amount of
>>>>>> memory in the worst case while maintaining O(1) amortized time bounds.  The
>>>>>> data structure described in [Brodnik99resizablearrays] has a maximum backing
>>>>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>>>>> over ArrayList do you see in your data structure?
>>>>>>
>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>> the other accepted inserts for near to the end."
>>>>>>
>>>>>> What happens with your structure when you add n elements and then
>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>> resize.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>
>>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>
>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>> impressive.
>>>>>>>
>>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>>>> statistic per list, and the second round results were used.
>>>>>>>
>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>
>>>>>>> and server JVM:
>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>
>>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>>> expected them to be fairly close.
>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at
>>>>>>> the beginning of large lists, which I haven't yet tried to understand.
>>>>>>> Insertion in the middle is similar.
>>>>>>>
>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>
>>>>>>> Finally, there are promising results for get() from the ExpArrayList
>>>>>>> as well (server JVM), again somehow beating ArrayList for larger lists:
>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>
>>>>>>>
>>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>>> source code and I will find somewhere to upload it.
>>>>>>>
>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>> reverseIterator() method.
>>>>>>>
>>>>>>>
>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>> and all of them for a backing array resize.
>>>>>>>>
>>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>
>>>>>>>> Hi Joe,
>>>>>>>>>
>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add does
>>>>>>>>> not amortize to constant time when the data structure employs the circular
>>>>>>>>> list trick to achieve deque behavior; ChunkedArrayList potentially resizes
>>>>>>>>> every n^(1/2) operations.
>>>>>>>>>
>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>
>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>
>>>>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>
>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>
>>>>>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>>>>>> implementation only moves array elements to the front on an array resize
>>>>>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>
>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Joe
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------------ CircularArrayList
>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>> 102          90         240         191         138
>>>>>>>>>>    100          19          67          70         166         138
>>>>>>>>>>          94         230         194         118
>>>>>>>>>>   1000          28          64          67         681         538
>>>>>>>>>>          91         324         382         119
>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>  69       32913         162         130         105
>>>>>>>>>>    100          20          67          70         129         104
>>>>>>>>>>       21944         169         134         135
>>>>>>>>>>   1000          29          63          67         651         506
>>>>>>>>>>        9602         364         333         526
>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>
>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>
>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>
>>>>>>>>>>>> It's interesting to note that the old circular list trick will
>>>>>>>>>>>> not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>> deque.
>>>>>>>>>>>>
>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>
>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>>>>> shape.  I have
>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>>>>>> (note that the
>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>> operation with
>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>>>>> indicates
>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>>>>> I've noticed
>>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>>> when I switch
>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>> performance
>>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>>> timed the process
>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>>> eliminate the
>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>>> achieving this by
>>>>>>>>>>>>> > way of a data structure that is both time and space optimal
>>>>>>>>>>>>> is a
>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees that
>>>>>>>>>>>>> no backing
>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to
>>>>>>>>>>>>> the multiple
>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>> > the additional cost of indexed access is partially mitigated
>>>>>>>>>>>>> by the
>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>>>>> ArrayList due
>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>> eventually
>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>>>>> originally
>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>> ).
>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet
>>>>>>>>>>>>> as it was
>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>>>>> more important
>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>> >> > popular collection, it's the primary "straight replacement
>>>>>>>>>>>>> for primitive
>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>>>>> updating the
>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.
>>>>>>>>>>>>> The code below
>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>>>> offset into the
>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>>>>>> this will
>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>>>>>> each time an
>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle
>>>>>>>>>>>>> the bits to see
>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b)
>>>>>>>>>>>>> + " " + e);
>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>>>> numbered
>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>>>> added (the
>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>> >> >> >> across an alternative characterization of superblock
>>>>>>>>>>>>> in the paper
>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>> example structure
>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>> following
>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>> consists of
>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>> implies the
>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i =
>>>>>>>>>>>>> 3:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>>>>>> "the number
>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only
>>>>>>>>>>>>> two data
>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>>>>> example above,
>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>> their data
>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>>>>>> that comes to
>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size)
>>>>>>>>>>>>> and to insert
>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>>>>>> size).  We
>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could
>>>>>>>>>>>>> you explain
>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>>>>>> shared by the
>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a
>>>>>>>>>>>>> way to turn
>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or
>>>>>>>>>>>>> be a better
>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>>>>> that you
>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>> operations -
>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100421/6c69cdff/attachment.html>

From martinrb at google.com  Wed Apr 21 23:15:43 2010
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 21 Apr 2010 16:15:43 -0700
Subject: UNIXProcess improvements
In-Reply-To: <4BCDCF0E.5090604@sun.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
	<4BCDCF0E.5090604@sun.com>
Message-ID: <o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>

I now have the second part of my planned improvements to
Linux process handling, and is now a serious proposal.
for Michael and others to review.

http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/
http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess2/

This is a huge performance and reliability improvement for
java programs that create many processes.
The program below (ManyProcesses) runs 4 times faster on my machine.

This doesn't try to address Solaris.

Martin

import java.util.*;
import java.util.concurrent.atomic.*;

public class ManyProcesses {
    public static void main(String[] args) throws Throwable {
        final int threadCount = 2;
        final int processCount = 2000;
        final AtomicLong count = new AtomicLong(0);
        Runnable r = new Runnable() {
          public void run() {
            try {
              for (int i = 0; i < processCount; i++) {
                Process p = new ProcessBuilder(new String[] {
"/bin/true" }).start();
                //if (p.waitFor() != 0) throw new Error();
//                 p.getInputStream().close();
//                 p.getOutputStream().close();
//                 p.getErrorStream().close();
                //count.getAndIncrement();
              }
            } catch (Throwable t) {
              throw new Error(t);
            }}};
        List<Thread> threads = new ArrayList<Thread>();
        for (int i = 0; i < threadCount; i++) {
          threads.add(new Thread(r));
        }

        for (Thread thread : threads) thread.start();
        //Thread.sleep(1000 * 10);
        //System.out.println(count.get());
        for (Thread thread : threads) thread.join();
        System.out.println(new Thread().getId());
    }
}

On Tue, Apr 20, 2010 at 08:58, Michael McMahon <Michael.McMahon at sun.com> wrote:
> Martin,
>
> Thanks for the answers. The changes look fine to me.
>
> - Michael.
>
> Martin Buchholz wrote:
>>
>> On Mon, Apr 19, 2010 at 09:04, Michael McMahon <Michael.McMahon at sun.com>
>> wrote:
>>
>>>
>>> Martin Buchholz wrote:
>>>
>>>>
>>>> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
>>>>
>>>>
>>>>
>>>>>
>>>>> For now I suggest leaving old @author tags as-is.
>>>>>
>>>>>
>>>>
>>>> OK, done.
>>>>
>>>> Version 0.2 of the webrev is published.
>>>>
>>>> Martin
>>>>
>>>>
>>>
>>> Martin,
>>>
>>> From what I can see, you've cleaned up the code and the functional
>>> changes
>>> are the use of a thread pool, and an explicit (8 k sized) stack.
>>>
>>
>> Exceptions thrown in the "process reaper" thread,
>> which were not IOExceptions,
>> were not being caught, and would cause the user thread to hang.
>>
>>
>>>
>>> Also, the threads created now belong to the root thread group rather than
>>> the application's thread group.
>>>
>>
>> Well, they have to belong to some thread group,
>> and they get reused, so in general the thread group will have
>> no relation to the thread group of the user thread,
>> so may as well use the root thread group.
>> I stole this technique from elsewhere in the JDK.
>>
>>
>>>
>>> Is this so you can handle uncaught
>>> exceptions
>>> as you mentioned before, and if so, I guess some other change is coming
>>> to
>>> complete
>>> this work. Is that right?
>>>
>>
>> Yes. ?This change by itself is a clear win,
>> except that because it is more memory efficient,
>> GC is less likely to get called,
>> which means file descriptors of defunct processes
>> are less likely to get cleaned up in the face of
>> lazy user code, which means it is more subject
>> to file descriptor exhaustion problems.
>> Which I would like to fix.
>>
>> Martin
>>
>>
>>>
>>> Thanks,
>>> Michael
>>>
>>>
>>>
>
>


From schlosna at gmail.com  Thu Apr 22 00:47:00 2010
From: schlosna at gmail.com (David Schlosnagle)
Date: Wed, 21 Apr 2010 20:47:00 -0400
Subject: UNIXProcess improvements
In-Reply-To: <o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
	<4BCDCF0E.5090604@sun.com>
	<o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>
Message-ID: <z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>

Martin,

In src/solaris/classes/java/lang/UNIXProcess.java.linux, are lines
177, 181, and 185 needed? If the assignment of these streams were in
the constructor, they could be final.

?176???? synchronized void processExited(int exitcode) {
?177???????? stdout = this.stdout;
?178???????? if (stdout instanceof ProcessPipeInputStream)
?179???????????? ((ProcessPipeInputStream) stdout).processExited();
?180
?181???????? stderr = this.stderr;
?182???????? if (stderr instanceof ProcessPipeInputStream)
?183???????????? ((ProcessPipeInputStream) stderr).processExited();
?184
?185???????? stdin = this.stdin;
?186???????? if (stdin instanceof ProcessPipeOutputStream)
?187???????????? ((ProcessPipeOutputStream) stdin).processExited();
?188
?189???????? this.exitcode = exitcode;
?190???????? hasExited = true;
?191???????? notifyAll();
?192???? }

Minor nit, the java.io.ByteArrayOutputStream import is not used.

Thanks,
Dave

On Wed, Apr 21, 2010 at 7:15 PM, Martin Buchholz <martinrb at google.com> wrote:
>
> I now have the second part of my planned improvements to
> Linux process handling, and is now a serious proposal.
> for Michael and others to review.
>
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/
> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess2/
>
> This is a huge performance and reliability improvement for
> java programs that create many processes.
> The program below (ManyProcesses) runs 4 times faster on my machine.
>
> This doesn't try to address Solaris.
>
> Martin
>
> import java.util.*;
> import java.util.concurrent.atomic.*;
>
> public class ManyProcesses {
> ? ?public static void main(String[] args) throws Throwable {
> ? ? ? ?final int threadCount = 2;
> ? ? ? ?final int processCount = 2000;
> ? ? ? ?final AtomicLong count = new AtomicLong(0);
> ? ? ? ?Runnable r = new Runnable() {
> ? ? ? ? ?public void run() {
> ? ? ? ? ? ?try {
> ? ? ? ? ? ? ?for (int i = 0; i < processCount; i++) {
> ? ? ? ? ? ? ? ?Process p = new ProcessBuilder(new String[] {
> "/bin/true" }).start();
> ? ? ? ? ? ? ? ?//if (p.waitFor() != 0) throw new Error();
> // ? ? ? ? ? ? ? ? p.getInputStream().close();
> // ? ? ? ? ? ? ? ? p.getOutputStream().close();
> // ? ? ? ? ? ? ? ? p.getErrorStream().close();
> ? ? ? ? ? ? ? ?//count.getAndIncrement();
> ? ? ? ? ? ? ?}
> ? ? ? ? ? ?} catch (Throwable t) {
> ? ? ? ? ? ? ?throw new Error(t);
> ? ? ? ? ? ?}}};
> ? ? ? ?List<Thread> threads = new ArrayList<Thread>();
> ? ? ? ?for (int i = 0; i < threadCount; i++) {
> ? ? ? ? ?threads.add(new Thread(r));
> ? ? ? ?}
>
> ? ? ? ?for (Thread thread : threads) thread.start();
> ? ? ? ?//Thread.sleep(1000 * 10);
> ? ? ? ?//System.out.println(count.get());
> ? ? ? ?for (Thread thread : threads) thread.join();
> ? ? ? ?System.out.println(new Thread().getId());
> ? ?}
> }
>
> On Tue, Apr 20, 2010 at 08:58, Michael McMahon <Michael.McMahon at sun.com> wrote:
> > Martin,
> >
> > Thanks for the answers. The changes look fine to me.
> >
> > - Michael.
> >
> > Martin Buchholz wrote:
> >>
> >> On Mon, Apr 19, 2010 at 09:04, Michael McMahon <Michael.McMahon at sun.com>
> >> wrote:
> >>
> >>>
> >>> Martin Buchholz wrote:
> >>>
> >>>>
> >>>> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>> For now I suggest leaving old @author tags as-is.
> >>>>>
> >>>>>
> >>>>
> >>>> OK, done.
> >>>>
> >>>> Version 0.2 of the webrev is published.
> >>>>
> >>>> Martin
> >>>>
> >>>>
> >>>
> >>> Martin,
> >>>
> >>> From what I can see, you've cleaned up the code and the functional
> >>> changes
> >>> are the use of a thread pool, and an explicit (8 k sized) stack.
> >>>
> >>
> >> Exceptions thrown in the "process reaper" thread,
> >> which were not IOExceptions,
> >> were not being caught, and would cause the user thread to hang.
> >>
> >>
> >>>
> >>> Also, the threads created now belong to the root thread group rather than
> >>> the application's thread group.
> >>>
> >>
> >> Well, they have to belong to some thread group,
> >> and they get reused, so in general the thread group will have
> >> no relation to the thread group of the user thread,
> >> so may as well use the root thread group.
> >> I stole this technique from elsewhere in the JDK.
> >>
> >>
> >>>
> >>> Is this so you can handle uncaught
> >>> exceptions
> >>> as you mentioned before, and if so, I guess some other change is coming
> >>> to
> >>> complete
> >>> this work. Is that right?
> >>>
> >>
> >> Yes. ?This change by itself is a clear win,
> >> except that because it is more memory efficient,
> >> GC is less likely to get called,
> >> which means file descriptors of defunct processes
> >> are less likely to get cleaned up in the face of
> >> lazy user code, which means it is more subject
> >> to file descriptor exhaustion problems.
> >> Which I would like to fix.
> >>
> >> Martin
> >>
> >>
> >>>
> >>> Thanks,
> >>> Michael
> >>>
> >>>
> >>>
> >
> >


From martinrb at google.com  Thu Apr 22 01:14:14 2010
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 21 Apr 2010 18:14:14 -0700
Subject: UNIXProcess improvements
In-Reply-To: <z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
	<4BCDCF0E.5090604@sun.com>
	<o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>
	<z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>
Message-ID: <z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>

Thanks for the careful review.

On Wed, Apr 21, 2010 at 17:47, David Schlosnagle <schlosna at gmail.com> wrote:
> Martin,
>
> In src/solaris/classes/java/lang/UNIXProcess.java.linux, are lines
> 177, 181, and 185 needed? If the assignment of these streams were in
> the constructor, they could be final.

Fixed.

Yes, the streams really should be final,
but it's not worth the boilerplate handsprings to make them so.

-    private OutputStream stdin_stream;
-    private InputStream  stdout_stream;
-    private InputStream  stderr_stream;
+    private /* final */ OutputStream stdin;
+    private /* final */ InputStream  stdout;
+    private /* final */ InputStream  stderr;


> ?176???? synchronized void processExited(int exitcode) {
> ?177???????? stdout = this.stdout;
> ?178???????? if (stdout instanceof ProcessPipeInputStream)
> ?179???????????? ((ProcessPipeInputStream) stdout).processExited();
> ?180
> ?181???????? stderr = this.stderr;
> ?182???????? if (stderr instanceof ProcessPipeInputStream)
> ?183???????????? ((ProcessPipeInputStream) stderr).processExited();
> ?184
> ?185???????? stdin = this.stdin;
> ?186???????? if (stdin instanceof ProcessPipeOutputStream)
> ?187???????????? ((ProcessPipeOutputStream) stdin).processExited();
> ?188
> ?189???????? this.exitcode = exitcode;
> ?190???????? hasExited = true;
> ?191???????? notifyAll();
> ?192???? }
>
> Minor nit, the java.io.ByteArrayOutputStream import is not used.

Fixed.  (do you have a tool to detect extra imports?)

Thanks,

Martin

> Thanks,
> Dave
>
> On Wed, Apr 21, 2010 at 7:15 PM, Martin Buchholz <martinrb at google.com> wrote:
>>
>> I now have the second part of my planned improvements to
>> Linux process handling, and is now a serious proposal.
>> for Michael and others to review.
>>
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess/
>> http://cr.openjdk.java.net/~martin/webrevs/openjdk7/UNIXProcess2/
>>
>> This is a huge performance and reliability improvement for
>> java programs that create many processes.
>> The program below (ManyProcesses) runs 4 times faster on my machine.
>>
>> This doesn't try to address Solaris.
>>
>> Martin
>>
>> import java.util.*;
>> import java.util.concurrent.atomic.*;
>>
>> public class ManyProcesses {
>> ? ?public static void main(String[] args) throws Throwable {
>> ? ? ? ?final int threadCount = 2;
>> ? ? ? ?final int processCount = 2000;
>> ? ? ? ?final AtomicLong count = new AtomicLong(0);
>> ? ? ? ?Runnable r = new Runnable() {
>> ? ? ? ? ?public void run() {
>> ? ? ? ? ? ?try {
>> ? ? ? ? ? ? ?for (int i = 0; i < processCount; i++) {
>> ? ? ? ? ? ? ? ?Process p = new ProcessBuilder(new String[] {
>> "/bin/true" }).start();
>> ? ? ? ? ? ? ? ?//if (p.waitFor() != 0) throw new Error();
>> // ? ? ? ? ? ? ? ? p.getInputStream().close();
>> // ? ? ? ? ? ? ? ? p.getOutputStream().close();
>> // ? ? ? ? ? ? ? ? p.getErrorStream().close();
>> ? ? ? ? ? ? ? ?//count.getAndIncrement();
>> ? ? ? ? ? ? ?}
>> ? ? ? ? ? ?} catch (Throwable t) {
>> ? ? ? ? ? ? ?throw new Error(t);
>> ? ? ? ? ? ?}}};
>> ? ? ? ?List<Thread> threads = new ArrayList<Thread>();
>> ? ? ? ?for (int i = 0; i < threadCount; i++) {
>> ? ? ? ? ?threads.add(new Thread(r));
>> ? ? ? ?}
>>
>> ? ? ? ?for (Thread thread : threads) thread.start();
>> ? ? ? ?//Thread.sleep(1000 * 10);
>> ? ? ? ?//System.out.println(count.get());
>> ? ? ? ?for (Thread thread : threads) thread.join();
>> ? ? ? ?System.out.println(new Thread().getId());
>> ? ?}
>> }
>>
>> On Tue, Apr 20, 2010 at 08:58, Michael McMahon <Michael.McMahon at sun.com> wrote:
>> > Martin,
>> >
>> > Thanks for the answers. The changes look fine to me.
>> >
>> > - Michael.
>> >
>> > Martin Buchholz wrote:
>> >>
>> >> On Mon, Apr 19, 2010 at 09:04, Michael McMahon <Michael.McMahon at sun.com>
>> >> wrote:
>> >>
>> >>>
>> >>> Martin Buchholz wrote:
>> >>>
>> >>>>
>> >>>> On Fri, Apr 16, 2010 at 09:18, Mark Reinhold <mr at sun.com> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> For now I suggest leaving old @author tags as-is.
>> >>>>>
>> >>>>>
>> >>>>
>> >>>> OK, done.
>> >>>>
>> >>>> Version 0.2 of the webrev is published.
>> >>>>
>> >>>> Martin
>> >>>>
>> >>>>
>> >>>
>> >>> Martin,
>> >>>
>> >>> From what I can see, you've cleaned up the code and the functional
>> >>> changes
>> >>> are the use of a thread pool, and an explicit (8 k sized) stack.
>> >>>
>> >>
>> >> Exceptions thrown in the "process reaper" thread,
>> >> which were not IOExceptions,
>> >> were not being caught, and would cause the user thread to hang.
>> >>
>> >>
>> >>>
>> >>> Also, the threads created now belong to the root thread group rather than
>> >>> the application's thread group.
>> >>>
>> >>
>> >> Well, they have to belong to some thread group,
>> >> and they get reused, so in general the thread group will have
>> >> no relation to the thread group of the user thread,
>> >> so may as well use the root thread group.
>> >> I stole this technique from elsewhere in the JDK.
>> >>
>> >>
>> >>>
>> >>> Is this so you can handle uncaught
>> >>> exceptions
>> >>> as you mentioned before, and if so, I guess some other change is coming
>> >>> to
>> >>> complete
>> >>> this work. Is that right?
>> >>>
>> >>
>> >> Yes. ?This change by itself is a clear win,
>> >> except that because it is more memory efficient,
>> >> GC is less likely to get called,
>> >> which means file descriptors of defunct processes
>> >> are less likely to get cleaned up in the face of
>> >> lazy user code, which means it is more subject
>> >> to file descriptor exhaustion problems.
>> >> Which I would like to fix.
>> >>
>> >> Martin
>> >>
>> >>
>> >>>
>> >>> Thanks,
>> >>> Michael
>> >>>
>> >>>
>> >>>
>> >
>> >
>


From schlosna at gmail.com  Thu Apr 22 03:15:43 2010
From: schlosna at gmail.com (David Schlosnagle)
Date: Wed, 21 Apr 2010 23:15:43 -0400
Subject: UNIXProcess improvements
In-Reply-To: <z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
	<4BCDCF0E.5090604@sun.com>
	<o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>
	<z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>
	<z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>
Message-ID: <w2w9146341c1004212015wfcc94822l26cd56a6554e95a1@mail.gmail.com>

On Wed, Apr 21, 2010 at 9:14 PM, Martin Buchholz <martinrb at google.com> wrote:
> Thanks for the careful review.

No problem, I wanted test this patch out on OS X since the current
UNIXProcess.java.bsd is virtually identical to UNIXProcess.java.linux.
I slightly tweaked the ManyProcesses test to exec /usr/bin/true and
log the elapsed time within main, as well as using `time` for overall
process time.

Running ManyProcesses with Apple's version of 1.6.0_17-b04-248,
averages around 45 seconds and 4010 started threads.
Running ManyProcesses with the bsd-port of OpenJDK7 on OS X without
these changes, averages around 16 seconds and 4010 started threads.
Running ManyProcesses with the bsd-port of OpenJDK7 on OS X with these
changes, averages around 14 seconds and around 14 started threads.

One definite benefit is that the total number of started threads drops
from 4010 to around 14. Compared to the Apple version of Java 6, I see
the multiple factor speedup of 3x. The speedup on my machine for
OpenJDK7 bsd-port with this patch versus without is much smaller, of
course that could be due differences in OS and hardware (I'm running
on an older Intel Core 2 Duo).

> Yes, the streams really should be final,
> but it's not worth the boilerplate handsprings to make them so.

I'm not sure I quite follow, I was thinking something along the lines of:

    private final OutputStream stdin;
    private final InputStream  stdout;
    private final InputStream  stderr;

// ...snip...

    UNIXProcess(final byte[] prog,
                final byte[] argBlock, final int argc,
                final byte[] envBlock, final int envc,
                final byte[] dir,
                final int[] fds,
                final boolean redirectErrorStream)
            throws IOException {

        pid = forkAndExec(prog,
                          argBlock, argc,
                          envBlock, envc,
                          dir,
                          fds,
                          redirectErrorStream);

        stdin = (fds[0] == -1) ?
                ProcessBuilder.NullOutputStream.INSTANCE :
                    new ProcessPipeOutputStream(fds[0]);

        stdout = (fds[1] == -1) ?
                ProcessBuilder.NullInputStream.INSTANCE :
                    new ProcessPipeInputStream(fds[1]);

        stderr = (fds[2] == -1) ?
                ProcessBuilder.NullInputStream.INSTANCE :
                    new ProcessPipeInputStream(fds[2]);

        startReaper();
    }

    private void startReaper() throws IOException {
        try {
            AccessController.doPrivileged
            (new PrivilegedExceptionAction<Void>() {
                public Void run() throws IOException {
                    processReaperExecutor.execute(new Runnable() {
                        public void run() {
                            int exitcode = waitForProcessExit(pid);
                            UNIXProcess.this.processExited(exitcode);
                        }});
                    return null;
                }});
        } catch (PrivilegedActionException ex) {
            throw (IOException) ex.getException();
        }
    }


>> Minor nit, the java.io.ByteArrayOutputStream import is not used.
>
> Fixed. ?(do you have a tool to detect extra imports?)

I had just dropped your changes into Eclipse to quickly rebuild the
classes to test out on my OS X build, and noticed that it was unused.
It really shouldn't matter, javac won't include it in the class file
anyway.

Thanks,
Dave


From java.net at freetocreate.org  Thu Apr 22 03:15:44 2010
From: java.net at freetocreate.org (Rob Leland)
Date: Wed, 21 Apr 2010 23:15:44 -0400
Subject: UNIXProcess improvements
In-Reply-To: <z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>	<20100416161831.C21CE511@eggemoggin.niobe.net>	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>	<4BCC7F14.6080600@sun.com>	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>	<4BCDCF0E.5090604@sun.com>	<o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>	<z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>
	<z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>
Message-ID: <4BCFBF60.8020009@freetocreate.org>

.
> Fixed.  (do you have a tool to detect extra imports?)
>    
There are a number of tools:

IntelliJ: Free community version 
:http://download.jetbrains.com/idea/ideaIC-9.0.2.exe
PMD:  http://pmd.sourceforge.net/rules/imports.html
NetBeans: http://netbeans.org/kb/67/java/editor-tips.html

etc...




From weijun.wang at sun.com  Thu Apr 22 04:46:57 2010
From: weijun.wang at sun.com (weijun.wang at sun.com)
Date: Thu, 22 Apr 2010 04:46:57 +0000
Subject: hg: jdk7/tl/jdk: 6856069: PrincipalName.clone() does not invoke
	super.clone()
Message-ID: <20100422044716.459B244657@hg.openjdk.java.net>

Changeset: edde2f60415b
Author:    weijun
Date:      2010-04-22 12:45 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/edde2f60415b

6856069: PrincipalName.clone() does not invoke super.clone()
Reviewed-by: chegar

! src/share/classes/sun/security/krb5/PrincipalName.java
+ test/sun/security/krb5/ServiceNameClone.java



From martinrb at google.com  Thu Apr 22 05:14:29 2010
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 21 Apr 2010 22:14:29 -0700
Subject: UNIXProcess improvements
In-Reply-To: <w2w9146341c1004212015wfcc94822l26cd56a6554e95a1@mail.gmail.com>
References: <v2s1ccfd1c11004160909of9a8670fk2d3e4f1e35dded57@mail.gmail.com>
	<20100416161831.C21CE511@eggemoggin.niobe.net>
	<r2h1ccfd1c11004161001pb96fbf88ldadfa508c6ef1cf5@mail.gmail.com>
	<4BCC7F14.6080600@sun.com>
	<h2s1ccfd1c11004191306md954ee8iaf57aade9591d965@mail.gmail.com>
	<4BCDCF0E.5090604@sun.com>
	<o2h1ccfd1c11004211615y1dd3ce8bmfa0bf163078c8bce@mail.gmail.com>
	<z2r9146341c1004211747m74ace105g1edcb3b3c9861b1c@mail.gmail.com>
	<z2v1ccfd1c11004211814h77fd9d5ftbe339761a63584a7@mail.gmail.com>
	<w2w9146341c1004212015wfcc94822l26cd56a6554e95a1@mail.gmail.com>
Message-ID: <q2j1ccfd1c11004212214iaa8adf35j145b78b3962f9681@mail.gmail.com>

David,

On Wed, Apr 21, 2010 at 20:15, David Schlosnagle <schlosna at gmail.com> wrote:
> On Wed, Apr 21, 2010 at 9:14 PM, Martin Buchholz <martinrb at google.com> wrote:

> Running ManyProcesses with Apple's version of 1.6.0_17-b04-248,
> averages around 45 seconds and 4010 started threads.
> Running ManyProcesses with the bsd-port of OpenJDK7 on OS X without
> these changes, averages around 16 seconds and 4010 started threads.
> Running ManyProcesses with the bsd-port of OpenJDK7 on OS X with these
> changes, averages around 14 seconds and around 14 started threads.
>
> One definite benefit is that the total number of started threads drops
> from 4010 to around 14. Compared to the Apple version of Java 6, I see
> the multiple factor speedup of 3x. The speedup on my machine for
> OpenJDK7 bsd-port with this patch versus without is much smaller, of
> course that could be due differences in OS and hardware (I'm running
> on an older Intel Core 2 Duo).

I did some more benchmarking myself and can confirm that
the win is less compared to stock openjdk7.  It is all highly
dependent on how often GC is run to reclaim file
descriptors, and how much the performance of fork+exec
is dependent on the number of unreclaimed file descriptors.

>> Yes, the streams really should be final,
>> but it's not worth the boilerplate handsprings to make them so.
>
> I'm not sure I quite follow, I was thinking something along the lines of:
>
> ? ?private final OutputStream stdin;
> ? ?private final InputStream ?stdout;
> ? ?private final InputStream ?stderr;
>
> // ...snip...
>
> ? ?UNIXProcess(final byte[] prog,
> ? ? ? ? ? ? ? ?final byte[] argBlock, final int argc,
> ? ? ? ? ? ? ? ?final byte[] envBlock, final int envc,
> ? ? ? ? ? ? ? ?final byte[] dir,
> ? ? ? ? ? ? ? ?final int[] fds,
> ? ? ? ? ? ? ? ?final boolean redirectErrorStream)
> ? ? ? ? ? ?throws IOException {
>
> ? ? ? ?pid = forkAndExec(prog,
> ? ? ? ? ? ? ? ? ? ? ? ? ?argBlock, argc,
> ? ? ? ? ? ? ? ? ? ? ? ? ?envBlock, envc,
> ? ? ? ? ? ? ? ? ? ? ? ? ?dir,
> ? ? ? ? ? ? ? ? ? ? ? ? ?fds,
> ? ? ? ? ? ? ? ? ? ? ? ? ?redirectErrorStream);
>
> ? ? ? ?stdin = (fds[0] == -1) ?
> ? ? ? ? ? ? ? ?ProcessBuilder.NullOutputStream.INSTANCE :
> ? ? ? ? ? ? ? ? ? ?new ProcessPipeOutputStream(fds[0]);

the creation of the file streams needs to be
within a privileged block, else you might get
a SecurityException.

Martin


From xueming.shen at oracle.com  Thu Apr 22 08:01:20 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Thu, 22 Apr 2010 01:01:20 -0700
Subject: Unicode script support in Regex and Character class
Message-ID: <4BD00250.3020206@oracle.com>

Hi,

Here is the webrev of the proposal to add Unicode script support in 
regex and j.l.Character.

http://cr.openjdk.java.net/~sherman/script/webrev

and the corresponding blenderrev

http://cr.openjdk.java.net/~sherman/script/blenderrev.html

Please comment on the APIs before I submit the CCC, especially

(1) to use enum for the j.l.Character.UnicodeScript (compared to the 
traditional j.l.c.Subset)
(2) the piggyback method j.l.c.getName() :-)
(3) the syntax for script constructs. In addition to the "normal"
     \p{InScriptName} and \P{InScriptName} for the script support
     I'm also adding
    \p{script=ScriptName} \P{script=ScriptName}  for the new script support
    \p{block=BlockName} \P{block=BlockName}  for the "existing" block 
support
    \p{general_category=CategoryName} \P{general_category=CategoryName} 
for the "existing" gc
    Perl recently also started to accept this  \p{propName=propValue} 
Unicode style.
    It opens the door for future "expanding", for example \p{name=XYZ} :-)
(4)and of course, the wording.

Thanks,
Sherman




From Ulf.Zibis at gmx.de  Thu Apr 22 13:50:30 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Thu, 22 Apr 2010 15:50:30 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD00250.3020206@oracle.com>
References: <4BD00250.3020206@oracle.com>
Message-ID: <4BD05426.7030309@gmx.de>

Am 22.04.2010 10:01, schrieb Xueming Shen:
> Hi,
>
> Here is the webrev of the proposal to add Unicode script support in 
> regex and j.l.Character.
>
> http://cr.openjdk.java.net/~sherman/script/webrev
>
> and the corresponding blenderrev
>
> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
>
> Please comment on the APIs before I submit the CCC, especially

- I like the idea, saving the data in a compressed binary file, instead 
classfile static data.
- wouldn't PreHashMaps be faster initialized as a normal HashMaps in 
j.l.Character.UnicodeScript and j.l.CharacterName?
- As alternative to lookup in a hash table, I guess retrieving the 
pointers from a memory saving sorted array via binary search would be 
fast enough.
- j.l.CharacterName:
-- You could instantiate the HashMap with capacity=cpLeng
-- Is it faster, first copying the whole date in a byte[], and then 
using ByteBuffer.getInt etc. against directly using DataInputStream methods?
-- You could create a very long String with the whole data and then use 
subString for the individual strings which could share the same backing 
char[].
-- I don't think, it's a good idea, holding the whole data in memory, 
especiallly as String objects; Additionally the backing char[]'s occupy 
twice the space than a byte[]
-- the big new byte[total] and later the huge amount of String objects 
could result in OOM error on small VM heap.
-- as compromise, you could put the cp->nameOff pointers in a separate 
not-compressed data file, only hold this in memory, or access it via 
DirectByteBuffer, and read the string data from separate file only on 
request from Character.getName(int codePoint). As option, a PreHashMap 
could cache individual loaded strings.
-- Anyway, having DirectByteBuffer access on deflated data would be a 
performace/footprint gain.
- enum j.l.Character.UnicodeScript:
-- IIRC, enums internally are handled as int constants, so retrieving an 
element via name would need a name->int lookup
-- So UnicodeScript.forName would have to lookup 2 times
--- alias->fullName (name of enum element)
--- fullName->internal int constant
-- I suggest to add the full names to the aliasses map.
-- Why don't you use Arrays.binarySearch in UnicodeScript.of(int 
codePoint) ?


>
> (1) to use enum for the j.l.Character.UnicodeScript (compared to the 
> traditional j.l.c.Subset)

- enum j.l.Character.UnicodeScript:
-- IIRC, enums internally are handled as int constants, so retrieving an 
element via name would need a name->int lookup
-- So UnicodeScript.forName would have to lookup 2 times
--- alias->fullName (name of enum element)
--- fullName->internal int constant
-- I suggest to add the full names to the aliasses map and only lookup once.
-- Why don't you use Arrays.binarySearch in UnicodeScript.of(int 
codePoint) ?


> (2) the piggyback method j.l.c.getName() :-)
> (3) the syntax for script constructs. In addition to the "normal"
>     \p{InScriptName} and \P{InScriptName} for the script support
>     I'm also adding
>    \p{script=ScriptName} \P{script=ScriptName}  for the new script 
> support
>    \p{block=BlockName} \P{block=BlockName}  for the "existing" block 
> support
>    \p{general_category=CategoryName} \P{general_category=CategoryName} 
> for the "existing" gc
>    Perl recently also started to accept this  \p{propName=propValue} 
> Unicode style.
>    It opens the door for future "expanding", for example \p{name=XYZ} :-)

I'm missing \p{InScriptName} in Pattern javadoc.

-Ulf


> (4)and of course, the wording.
>
> Thanks,
> Sherman
>
>
>



From xueming.shen at oracle.com  Thu Apr 22 22:38:49 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Thu, 22 Apr 2010 15:38:49 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD05426.7030309@gmx.de>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
Message-ID: <4BD0CFF9.7080706@oracle.com>

Ulf Zibis wrote:
>> (3) the syntax for script constructs. In addition to the "normal"
>>     \p{InScriptName} and \P{InScriptName} for the script support
>>     I'm also adding
>>    \p{script=ScriptName} \P{script=ScriptName}  for the new script 
>> support
>>    \p{block=BlockName} \P{block=BlockName}  for the "existing" block 
>> support
>>    \p{general_category=CategoryName} 
>> \P{general_category=CategoryName} for the "existing" gc
>>    Perl recently also started to accept this  \p{propName=propValue} 
>> Unicode style.
>>    It opens the door for future "expanding", for example \p{name=XYZ} 
>> :-)
> (2) the piggyback method j.l.c.getName() :-)
>
> I'm missing \p{InScriptName} in Pattern javadoc.
>

I meant to say

\p{IsScriptName} and \P{IsScriptName}

So the "recommended" usage would be

Script:
\p{IsScriptName} and \P{IsScriptName} or \p{script=ScriptName} 
\P{script=ScriptName}

Block
\p{InBlockName} \P{InBlockName} or \p{block=BlockName} \P{block=BlockName}

Category
\p{CategoryName} \P{CategoryName} or \p{general_category=CategoryName} 
\P{general_category=CategoryName}

For compatibility reason, we also take \p{IsCategoryName} \P{IsCategoryName}
It appears there is no conflict between the category name and script 
name, yet.

My apology for the inconvenience.

Sherman


From xueming.shen at oracle.com  Thu Apr 22 22:44:57 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Thu, 22 Apr 2010 15:44:57 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD01C1C.3070707@Sun.com>
References: <4BD00250.3020206@oracle.com> <4BD01C1C.3070707@Sun.com>
Message-ID: <4BD0D169.5020402@oracle.com>

Yuri Gaevsky wrote:
> Hi Sherman,
>
> A couple of minor comments:
>   - There is a typo (Uniocde) in Character.UnicodeScript.forName(java.lang.String):
>         "Returns the UnicodeScript with the given Uniocde script name or the script
>          name alias. "
>   - Shouldn't the method be more specific in respect of inner spaces, underscores
>     and so on (as [1] does)?
>
> Regards,
> -Yuri
>
> [1] http://java.sun.com/javase/6/docs/api/java/lang/Character.UnicodeBlock.html#forName(java.lang.String)
>
>
>   

Thanks Yuri.

Typo has been fixed and webrev has been updated.

The difference of block name and script name is that the block name 
defined by Unicode in Blocks.txt uses space
character and hyphen as the separator (instead of the underscore)  for 
example, the "Latin-1 Supplement", which
makes it impossible to use the name as a identifier in Java directly. 
The UnicodeBlock.forName() then has too accept
both the original/canonical  block name and the "text representation" of 
the UnicoeBlock identifer.

For the script name, while the tr24[2] states that " the presence of 
hyphen or underscore is optional", the Scripts.txt[3]
strictly only uses underscore for the script name. I was considering if 
I should also allow "loose-match" for the script
name to accept those names that use space or hyphen in place of "_",  
but decided to stick with the canonical name
(actually there are only several few names that need this).  Well, I'm 
still open on this one, if people think the
"loose-match" is important.

I added "The en_US locale's case mapping rules are used to provide 
case-insensitive string comparisons for script
name validation", as suggested.

I also replaced the "Character.UnicodeScript object/instance" with 
"constant" in several places to be consistent with
the inherited methods valueOf(0 and values()

The rfe ids are
4860714: Make Unicode scripts available for use in regular expressions
6945564: Unicode script support in Character class

-Sherman


[1] http://www.unicode.org/Public/5.2.0/ucd/Blocks.txt
[2] http://www.unicode.org/reports/tr24/
[3] http://www.unicode.org/Public/UNIDATA/Scripts.txt


From kevin.l.stern at gmail.com  Fri Apr 23 10:00:18 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Fri, 23 Apr 2010 05:00:18 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2s1704b7a21004140404l419711b4u7466378af4f13a9@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
Message-ID: <q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>

Hi Benedict,

Have you had a chance to get your index decomposition procedure to work with
seed values less than two?

Kevin

On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith
<lists at laerad.com>wrote:

> Hi Kevin,
>
> As it happens I might have something useful still to contribute. As an
> exercise in saving face I revisited the problem to see if I could achieve
> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
> must admit I still didn't fully appreciate how the algorithm in
> ChunkedArrayList worked until I tried to come up with an algorithm with
> similar properties. What I have ended up with is almost identical except
> adds I think a couple of incremental improvements, simply by redefining the
> arrayIndex() method. I should note that I have not yet implemented more than
> a prototype as it seems to me your implementation is excellent already, and
> if it is decided to include my modifications the changes should be modest.
>
> Firstly, (I hope that) what I have produced is a little more CPU pipe-line
> friendly; there is less dependency on immediately preceding calculations at
> each stage (i.e. so more operations should be able to proceed simultaneously
> in the pipeline), and consists exclusively of shifts, addition/subtraction
> and bit-wise (&)ands (except for the conditionals in
> Integer.numberOfLeadingZeros(i)), although the total number of instructions
> is approximately the same.
>
> Secondly, I have modified the algorithm so that a "seed" size can be
> specified (although I expect hard coding a suitable one will ultimately be
> best). Whereas ChunkedArrayList currently requires that the pattern of array
> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
> 16*s*(..*24)] etc. although when put in simple text like that it does
> appear to trivialise the change. The benefit of this, though, is two fold:
> 1) for small n the constant factor is reduced (both CPU and memory wise);
> and 2) the sqrt(n) bounds are reached more quickly also.
>
> As an illustration, consider setting *s* to 4, and assume the backing
> array is size two and doubles in size with each growth; with
> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as 4 we
> would instead resize at i=8,i=24,i=80,i=288; the cost at each would be some
> multiple of 2,4,8,16 respectively. As you can see the latter is much closer
> to the sqrt(n) cost - both approach it eventually, but my suggestion is to
> reach it more quickly. This is at the expense of more slowly reaching the
> sqrt(n) wasted memory condition, but given the high constant factor cost wrt
> to memory at this early stage, this seems a very sensible trade off. It
> seems likely this should also have a positive impact on cache performance
> for smaller lists as well.
>
> Finally, after playing with this idea in my head I am confident I can
> extend the core ideas of this data structure to hashing relatively easily,
> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
> wasted memory guarantees. I notice that this case hasn't been addressed yet,
> although I see from Martin's recent mail that this was raised before. Unless
> there are better suggestions for solving the hash table problem I will have
> a go at it as it seems an interesting problem - that is, assuming there are
> no objections?
>
> I'm interested to hear your thoughts. I hope this time I've been a bit more
> considered in what I've put forward, and hence less of a waste of time!
>
> Code snippet for calculation of array index and item offset:
>
> final int arraySizeShiftMinusSeed = ((31 -
> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3) -
> (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>  final int indexRemainder = index - ((1 << seed) <<
> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>
> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>
> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* +
> 1); seed only works for values for 2 or more at this moment, fyi
>
>
>
> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>> know if you have any other questions or if you have any related ideas or
>> concerns.
>>
>>
>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <lists at laerad.com
>> > wrote:
>>
>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>> help. that part of the discussion was clearly truncated by the time I had
>>> joined the list - I haven't been able to find the history in the archives
>>> either...
>>>
>>> I was just wondering about the worst case cost of add() as described by
>>> your javadoc; admittedly it is optimal with respect to unused memory, but
>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>> constant factor. I had been thinking that once n passed a threshold the cost
>>> of additions in this other structure would become a constant factor,
>>> offering nice algorithmic complexity guarantees for large n; however since
>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>> allocations would have to be unrealistically small (assuming linear cost for
>>> allocation) for this to be the case. It would still be nice to have a data
>>> structure that avoids needing to copy data with each grow, whilst still
>>> maintaining good memory performance.
>>>
>>> That *all* being said, I had been going by your javadoc and emails to
>>> ascertain the behaviour of this class, as I couldn't locate a free copy
>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>> than with what I assumed to be copying data between arraylets, and it seems
>>> this cost is pretty optimal. That will teach me to post to a list without
>>> getting my facts straight first. The interesting thing is simply that the
>>> constant factor for this implementation still seems to be quite high,
>>> although perhaps that is simply because I was not benchmarking sufficiently
>>> large values of n.
>>>
>>>
>>>
>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Hi Benedict,
>>>>
>>>> Unless I am misreading your post, this now has a very similar feel to
>>>> the first data structure that I posted to the list.  Martin Buchholz then
>>>> pointed out that we can incorporate the ideas from
>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>
>>>> Regards,
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>> addressed the problem that was originally being visited; only reduced the
>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>> power of 2), after which the array is increased in size linearly.
>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>> been exhaustive in the benchmarking:
>>>>>
>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>
>>>>> The nice thing about this is that the maximum amount of wasted memory
>>>>> is user configurable. Even with a low setting as above (65K) performance
>>>>> seems pretty consistent.
>>>>>
>>>>> Code for calculating index and array offset are pretty straight
>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>
>>>>> private final int indexFor(int a, int i) {
>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>> maxArraySizeShift : 1 << a) ;
>>>>>  }
>>>>> private final int arrayFor(int i) {
>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>  }
>>>>>
>>>>> Regarding the double list idea - yes, I agree, I certainly didn't think
>>>>> that one through fully!
>>>>>
>>>>>
>>>>>
>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Hi Benedict,
>>>>>>
>>>>>> Like you, I am relatively new to this mailing list; I am also trying
>>>>>> to tread lightly so as not to step on any toes.  That being said, I think
>>>>>> that I can offer a response to your inquiry.
>>>>>>
>>>>>> Regarding:  "The idea is to simply double the new array size each time
>>>>>> a new array needs to be allocated"
>>>>>>
>>>>>> It seems this would not address the desire of offering an alternative
>>>>>> to the allocation of a large backing array for ArrayList (your largest
>>>>>> backing array could still reach a size of 1/2 * Integer.MAX_VALUE) and would
>>>>>> not address the desire of wasting the (asymptotically) minimum amount of
>>>>>> memory in the worst case while maintaining O(1) amortized time bounds.  The
>>>>>> data structure described in [Brodnik99resizablearrays] has a maximum backing
>>>>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>>>>> over ArrayList do you see in your data structure?
>>>>>>
>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>> the other accepted inserts for near to the end."
>>>>>>
>>>>>> What happens with your structure when you add n elements and then
>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>> resize.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>
>>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>
>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>> impressive.
>>>>>>>
>>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>>>> statistic per list, and the second round results were used.
>>>>>>>
>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>
>>>>>>> and server JVM:
>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>
>>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>>> expected them to be fairly close.
>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion at
>>>>>>> the beginning of large lists, which I haven't yet tried to understand.
>>>>>>> Insertion in the middle is similar.
>>>>>>>
>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>
>>>>>>> Finally, there are promising results for get() from the ExpArrayList
>>>>>>> as well (server JVM), again somehow beating ArrayList for larger lists:
>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>
>>>>>>>
>>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>>> source code and I will find somewhere to upload it.
>>>>>>>
>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>> reverseIterator() method.
>>>>>>>
>>>>>>>
>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>> and all of them for a backing array resize.
>>>>>>>>
>>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Joe
>>>>>>>>
>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>
>>>>>>>> Hi Joe,
>>>>>>>>>
>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add does
>>>>>>>>> not amortize to constant time when the data structure employs the circular
>>>>>>>>> list trick to achieve deque behavior; ChunkedArrayList potentially resizes
>>>>>>>>> every n^(1/2) operations.
>>>>>>>>>
>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>
>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>
>>>>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>
>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>
>>>>>>>>>> Kevin, I don't fully understand your point about not amortizing to
>>>>>>>>>> O(1). Certainly that's true for insert not at head or tail. Otherwise this
>>>>>>>>>> implementation only moves array elements to the front on an array resize
>>>>>>>>>> operation which happens every O(ln n) operations at most, if we do lots of
>>>>>>>>>> adds, maybe a little more if we add array shrinking too.  This is the same
>>>>>>>>>> as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>
>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Joe
>>>>>>>>>>
>>>>>>>>>> ------------------------------------------------ CircularArrayList
>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>> 102          90         240         191         138
>>>>>>>>>>    100          19          67          70         166         138
>>>>>>>>>>          94         230         194         118
>>>>>>>>>>   1000          28          64          67         681         538
>>>>>>>>>>          91         324         382         119
>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>   size         add         get         set   iterAdd/3   iterAdd/2
>>>>>>>>>>   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>  69       32913         162         130         105
>>>>>>>>>>    100          20          67          70         129         104
>>>>>>>>>>       21944         169         134         135
>>>>>>>>>>   1000          29          63          67         651         506
>>>>>>>>>>        9602         364         333         526
>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>
>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>
>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>
>>>>>>>>>>>> It's interesting to note that the old circular list trick will
>>>>>>>>>>>> not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>> deque.
>>>>>>>>>>>>
>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>
>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>>>>> shape.  I have
>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included below
>>>>>>>>>>>>> (note that the
>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>> operation with
>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>>>>> indicates
>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>>>>> I've noticed
>>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>>> when I switch
>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>> performance
>>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>>> timed the process
>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>>> eliminate the
>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>>> achieving this by
>>>>>>>>>>>>> > way of a data structure that is both time and space optimal
>>>>>>>>>>>>> is a
>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees that
>>>>>>>>>>>>> no backing
>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due to
>>>>>>>>>>>>> the multiple
>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>> > the additional cost of indexed access is partially mitigated
>>>>>>>>>>>>> by the
>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>> > the index decomposition procedure, and the additional cost of
>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact that
>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>>>>> ArrayList due
>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>> eventually
>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>> >>
>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>>>>> originally
>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>> ).
>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as yet
>>>>>>>>>>>>> as it was
>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>>>>> more important
>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>> >> > popular collection, it's the primary "straight replacement
>>>>>>>>>>>>> for primitive
>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>>>>> updating the
>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.
>>>>>>>>>>>>> The code below
>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>>>> offset into the
>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned that
>>>>>>>>>>>>> this will
>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing this
>>>>>>>>>>>>> each time an
>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle
>>>>>>>>>>>>> the bits to see
>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 - 1);
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b)
>>>>>>>>>>>>> + " " + e);
>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>>>> numbered
>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>>>> added (the
>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>> >> >> >> across an alternative characterization of superblock
>>>>>>>>>>>>> in the paper
>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>> example structure
>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>> following
>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>> consists of
>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>> implies the
>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i =
>>>>>>>>>>>>> 3:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p represents
>>>>>>>>>>>>> "the number
>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are only
>>>>>>>>>>>>> two data
>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>>>>> example above,
>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>> their data
>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first approach
>>>>>>>>>>>>> that comes to
>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size)
>>>>>>>>>>>>> and to insert
>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo the
>>>>>>>>>>>>> size).  We
>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.  Could
>>>>>>>>>>>>> you explain
>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that is
>>>>>>>>>>>>> shared by the
>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's a
>>>>>>>>>>>>> way to turn
>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or
>>>>>>>>>>>>> be a better
>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>>>>> that you
>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>> operations -
>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100423/d1542e69/attachment.html>

From lists at laerad.com  Fri Apr 23 19:26:09 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Fri, 23 Apr 2010 20:26:09 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<x2jbb01431e1004140425w2bc4a9fcg11914b5d22872d3a@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
Message-ID: <v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>

Hi Kevin,

Unfortunately this week has been pretty hectic, and I haven't had much time
to much more than theorise on this topic - and this weekend the weather
looks set to be much too nice to stay in doors! It looks like you've made
really good progress on the ChunkedArrayDeque - I haven't fully digested it
yet, but it looks pretty impressive. I'm not sure (since I misunderstood
your list implementation prior to reading it in detail) but I think I may
have been toying with a similar growth strategy for a hash map variant since
the copies are necessary there anyway.

I have taken another look at the index algorithm and have tidied it up as
below, and it now supports a seed size of 1; however I am not sure that
using a value this small is advisable, given that the overhead for the first
array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
utilise less than 20% of the allocated memory for data, the first 24 less
than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
advisable.

As an aside, it is worth noting that the indexOffset calculation below can
be replaced with index & (_backingArray[arrayIndex].length - 1) , although I
am not certain what effect this will have on performance, given that this
would be another memory operation to retrieve the length from the array; but
it would make the separation of the calculation into functions more straight
forward.

     final int arraySizeShiftMinusSeed = (31 -
Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
     final int arraySizeShift = arraySizeShiftMinusSeed + seed ;

     final int firstArrayOfThisSize =
      (1 << arraySizeShiftMinusSeed + 2)
     - (1 << arraySizeShiftMinusSeed)
     - 1 - (arraySizeShift >>> 31) ;
     final int indexRemainder = index - (1 << arraySizeShift <<
arraySizeShiftMinusSeed) ;

     final int arrayOffset = indexRemainder >>> arraySizeShift ;
     final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
Integer.MAX_VALUE ;

     final int itemIndex = index & ((1 << arraySizeShift) - 1) ;



On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> Have you had a chance to get your index decomposition procedure to work
> with seed values less than two?
>
> Kevin
>
>
> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <lists at laerad.com
> > wrote:
>
>> Hi Kevin,
>>
>> As it happens I might have something useful still to contribute. As an
>> exercise in saving face I revisited the problem to see if I could achieve
>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>> must admit I still didn't fully appreciate how the algorithm in
>> ChunkedArrayList worked until I tried to come up with an algorithm with
>> similar properties. What I have ended up with is almost identical except
>> adds I think a couple of incremental improvements, simply by redefining the
>> arrayIndex() method. I should note that I have not yet implemented more than
>> a prototype as it seems to me your implementation is excellent already, and
>> if it is decided to include my modifications the changes should be modest.
>>
>> Firstly, (I hope that) what I have produced is a little more CPU pipe-line
>> friendly; there is less dependency on immediately preceding calculations at
>> each stage (i.e. so more operations should be able to proceed simultaneously
>> in the pipeline), and consists exclusively of shifts, addition/subtraction
>> and bit-wise (&)ands (except for the conditionals in
>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>> is approximately the same.
>>
>> Secondly, I have modified the algorithm so that a "seed" size can be
>> specified (although I expect hard coding a suitable one will ultimately be
>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>> 16*s*(..*24)] etc. although when put in simple text like that it does
>> appear to trivialise the change. The benefit of this, though, is two fold:
>> 1) for small n the constant factor is reduced (both CPU and memory wise);
>> and 2) the sqrt(n) bounds are reached more quickly also.
>>
>> As an illustration, consider setting *s* to 4, and assume the backing
>> array is size two and doubles in size with each growth; with
>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as 4
>> we would instead resize at i=8,i=24,i=80,i=288; the cost at each would be
>> some multiple of 2,4,8,16 respectively. As you can see the latter is much
>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>> is to reach it more quickly. This is at the expense of more slowly reaching
>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>> wrt to memory at this early stage, this seems a very sensible trade off. It
>> seems likely this should also have a positive impact on cache performance
>> for smaller lists as well.
>>
>> Finally, after playing with this idea in my head I am confident I can
>> extend the core ideas of this data structure to hashing relatively easily,
>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>> although I see from Martin's recent mail that this was raised before. Unless
>> there are better suggestions for solving the hash table problem I will have
>> a go at it as it seems an interesting problem - that is, assuming there are
>> no objections?
>>
>> I'm interested to hear your thoughts. I hope this time I've been a bit
>> more considered in what I've put forward, and hence less of a waste of time!
>>
>> Code snippet for calculation of array index and item offset:
>>
>> final int arraySizeShiftMinusSeed = ((31 -
>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3) -
>> (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>  final int indexRemainder = index - ((1 << seed) <<
>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>
>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>
>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* +
>> 1); seed only works for values for 2 or more at this moment, fyi
>>
>>
>>
>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>>> know if you have any other questions or if you have any related ideas or
>>> concerns.
>>>
>>>
>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>>> help. that part of the discussion was clearly truncated by the time I had
>>>> joined the list - I haven't been able to find the history in the archives
>>>> either...
>>>>
>>>> I was just wondering about the worst case cost of add() as described by
>>>> your javadoc; admittedly it is optimal with respect to unused memory, but
>>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>>> constant factor. I had been thinking that once n passed a threshold the cost
>>>> of additions in this other structure would become a constant factor,
>>>> offering nice algorithmic complexity guarantees for large n; however since
>>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>>> allocations would have to be unrealistically small (assuming linear cost for
>>>> allocation) for this to be the case. It would still be nice to have a data
>>>> structure that avoids needing to copy data with each grow, whilst still
>>>> maintaining good memory performance.
>>>>
>>>> That *all* being said, I had been going by your javadoc and emails to
>>>> ascertain the behaviour of this class, as I couldn't locate a free copy
>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>> getting my facts straight first. The interesting thing is simply that the
>>>> constant factor for this implementation still seems to be quite high,
>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>> large values of n.
>>>>
>>>>
>>>>
>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Hi Benedict,
>>>>>
>>>>> Unless I am misreading your post, this now has a very similar feel to
>>>>> the first data structure that I posted to the list.  Martin Buchholz then
>>>>> pointed out that we can incorporate the ideas from
>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>> lists at laerad.com> wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>> been exhaustive in the benchmarking:
>>>>>>
>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>
>>>>>> The nice thing about this is that the maximum amount of wasted memory
>>>>>> is user configurable. Even with a low setting as above (65K) performance
>>>>>> seems pretty consistent.
>>>>>>
>>>>>> Code for calculating index and array offset are pretty straight
>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>
>>>>>> private final int indexFor(int a, int i) {
>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>  }
>>>>>> private final int arrayFor(int i) {
>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>  }
>>>>>>
>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>> think that one through fully!
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>
>>>>>>> Hi Benedict,
>>>>>>>
>>>>>>> Like you, I am relatively new to this mailing list; I am also trying
>>>>>>> to tread lightly so as not to step on any toes.  That being said, I think
>>>>>>> that I can offer a response to your inquiry.
>>>>>>>
>>>>>>> Regarding:  "The idea is to simply double the new array size each
>>>>>>> time a new array needs to be allocated"
>>>>>>>
>>>>>>> It seems this would not address the desire of offering an alternative
>>>>>>> to the allocation of a large backing array for ArrayList (your largest
>>>>>>> backing array could still reach a size of 1/2 * Integer.MAX_VALUE) and would
>>>>>>> not address the desire of wasting the (asymptotically) minimum amount of
>>>>>>> memory in the worst case while maintaining O(1) amortized time bounds.  The
>>>>>>> data structure described in [Brodnik99resizablearrays] has a maximum backing
>>>>>>> array size of sqrt(n) and caps wasted memory at sqrt(n).  What advantage
>>>>>>> over ArrayList do you see in your data structure?
>>>>>>>
>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>> the other accepted inserts for near to the end."
>>>>>>>
>>>>>>> What happens with your structure when you add n elements and then
>>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>> resize.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>> lists at laerad.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>
>>>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>
>>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>>> impressive.
>>>>>>>>
>>>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>>>>> statistic per list, and the second round results were used.
>>>>>>>>
>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>
>>>>>>>> and server JVM:
>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>
>>>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>>>> expected them to be fairly close.
>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion
>>>>>>>> at the beginning of large lists, which I haven't yet tried to understand.
>>>>>>>> Insertion in the middle is similar.
>>>>>>>>
>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>
>>>>>>>> Finally, there are promising results for get() from the ExpArrayList
>>>>>>>> as well (server JVM), again somehow beating ArrayList for larger lists:
>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>>>> source code and I will find somewhere to upload it.
>>>>>>>>
>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>> reverseIterator() method.
>>>>>>>>
>>>>>>>>
>>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>>
>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>
>>>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Joe
>>>>>>>>>
>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>
>>>>>>>>> Hi Joe,
>>>>>>>>>>
>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add
>>>>>>>>>> does not amortize to constant time when the data structure employs the
>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>
>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>
>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>
>>>>>>>>>>> I'd be interested if you have any comments in the context of this
>>>>>>>>>>> discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>
>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>
>>>>>>>>>>> Kevin, I don't fully understand your point about not amortizing
>>>>>>>>>>> to O(1). Certainly that's true for insert not at head or tail. Otherwise
>>>>>>>>>>> this implementation only moves array elements to the front on an array
>>>>>>>>>>> resize operation which happens every O(ln n) operations at most, if we do
>>>>>>>>>>> lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>
>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Joe
>>>>>>>>>>>
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>> 102          90         240         191         138
>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>> 138          94         230         194         118
>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>> 538          91         324         382         119
>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>  69       32913         162         130         105
>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>> 104       21944         169         134         135
>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>> 506        9602         364         333         526
>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>>
>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>
>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's interesting to note that the old circular list trick will
>>>>>>>>>>>>> not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>
>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> > implementation; I believe the latest version to be in decent
>>>>>>>>>>>>>> shape.  I have
>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>>> operation with
>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so 1.00
>>>>>>>>>>>>>> indicates
>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less costly).
>>>>>>>>>>>>>> I've noticed
>>>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>>>> when I switch
>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>>>> timed the process
>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>>>> eliminate the
>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>>>> achieving this by
>>>>>>>>>>>>>> > way of a data structure that is both time and space optimal
>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees that
>>>>>>>>>>>>>> no backing
>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due
>>>>>>>>>>>>>> to the multiple
>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>>> > the additional cost of indexed access is partially mitigated
>>>>>>>>>>>>>> by the
>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>> > the index decomposition procedure, and the additional cost
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation on
>>>>>>>>>>>>>> ArrayList due
>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>> >> > The data structure is available at the second link that I
>>>>>>>>>>>>>> originally
>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as
>>>>>>>>>>>>>> yet as it was
>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a much
>>>>>>>>>>>>>> more important
>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll be
>>>>>>>>>>>>>> updating the
>>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is incorrect.
>>>>>>>>>>>>>> The code below
>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>>>>> offset into the
>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned
>>>>>>>>>>>>>> that this will
>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing
>>>>>>>>>>>>>> this each time an
>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle
>>>>>>>>>>>>>> the bits to see
>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 -
>>>>>>>>>>>>>> 1);
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p + b)
>>>>>>>>>>>>>> + " " + e);
>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>>>>> numbered
>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>>>>> added (the
>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>> >> >> >> across an alternative characterization of superblock
>>>>>>>>>>>>>> in the paper
>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>>> following
>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>>> consists of
>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i
>>>>>>>>>>>>>> = 3:
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are
>>>>>>>>>>>>>> only two data
>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in my
>>>>>>>>>>>>>> example above,
>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>>> their data
>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size)
>>>>>>>>>>>>>> and to insert
>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo
>>>>>>>>>>>>>> the size).  We
>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that
>>>>>>>>>>>>>> is shared by the
>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's
>>>>>>>>>>>>>> a way to turn
>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the ArrayDeque
>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or
>>>>>>>>>>>>>> be a better
>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the paper
>>>>>>>>>>>>>> that you
>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100423/bdcc69d1/attachment.html>

From xueming.shen at oracle.com  Fri Apr 23 23:09:23 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Fri, 23 Apr 2010 16:09:23 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD05426.7030309@gmx.de>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
Message-ID: <4BD228A3.7010103@oracle.com>

Ulf Zibis wrote:
>
> - I like the idea, saving the data in a compressed binary file, 
> instead classfile static data.
> - wouldn't PreHashMaps be faster initialized as a normal HashMaps in 
> j.l.Character.UnicodeScript and j.l.CharacterName?
I don't think so. The key for these 2 cases is the whole unicode range. 
But you can always try. Current
binary-search for script definitely is not a perfect solution.

> - As alternative to lookup in a hash table, I guess retrieving the 
> pointers from a memory saving sorted array via binary search would be 
> fast enough.
> - j.l.CharacterName:
> -- You could instantiate the HashMap with capacity=cpLeng
I changed the data file "format" a bit, so now the overal uniName.dat is 
less than 88k (last version is 122+k), but
the I can no long use cpLen as the capacity for the hashmap. I'm now 
using a hardcoded 20000 for 5.2.

> -- Is it faster, first copying the whole date in a byte[], and then 
> using ByteBuffer.getInt etc. against directly using DataInputStream 
> methods?
> -- You could create a very long String with the whole data and then 
> use subString for the individual strings which could share the same 
> backing char[].
> -- I don't think, it's a good idea, holding the whole data in memory, 
> especiallly as String objects; Additionally the backing char[]'s 
> occupy twice the space than a byte[]
> -- the big new byte[total] and later the huge amount of String objects 
> could result in OOM error on small VM heap.
> -- as compromise, you could put the cp->nameOff pointers in a separate 
> not-compressed data file, only hold this in memory, or access it via 
> DirectByteBuffer, and read the string data from separate file only on 
> request from Character.getName(int codePoint). As option, a PreHashMap 
> could cache individual loaded strings.
> -- Anyway, having DirectByteBuffer access on deflated data would be a 
> performace/footprint gain.
>
Sorry, I don't think I fully understand your points here.

I believe you would NOT see any meaningful performance boost from using 
DirectByteBuffer, given the
size of the data file, 88k. It probably will slow it down a little.

If you take a look at the last version
http://cr.openjdk.java.net/~sherman/script/webrev/src/share/classes/java/lang/CharacterName.java.html
You probably will not consider to use DataInputStream class. I no longer 
store the code point value for
most entries, one the length of the name, in which 1 byte is definitely 
big enough.

Yes, the final table takes about 500k, we might consider to use a 
weakref or something, if memory really
a concern. But the table will get initialized only if you invoke 
Character.getName(), I would expect most
of the application would never get down there.

>
>>
>> (1) to use enum for the j.l.Character.UnicodeScript (compared to the 
>> traditional j.l.c.Subset)
>
> - enum j.l.Character.UnicodeScript:
> -- IIRC, enums internally are handled as int constants, so retrieving 
> an element via name would need a name->int lookup
> -- So UnicodeScript.forName would have to lookup 2 times
> --- alias->fullName (name of enum element)
> --- fullName->internal int constant
> -- I suggest to add the full names to the aliasses map and only lookup 
> once.
Not really. It's not alias->fullName, it's alias->UnicodeScript costant. 
So if the passed in is an alias, then
we don't do the second lookup. That said, it's always a trade-off of 
memory use and speed. To put all
full name in aliases map definitely will reduce the second lookup if the 
passed in is a canonical name, with
the price of having name entries in both alias map and enum's internal 
hashmap. I really don't know which
one is a better choice. I did it this way with the assumption the lookup 
for script name is not critical. I
might be wrong.


> -- Why don't you use Arrays.binarySearch in UnicodeScript.of(int 
> codePoint) ?
>
>

why? I don't know:-) Maybe the copy/paste from UnicodeBlock lookup is 
more convenient than using
the Arrays.binarySearch. Not a big deal.

Thanks,
-Sherman


From kevin.l.stern at gmail.com  Sat Apr 24 00:31:14 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Fri, 23 Apr 2010 19:31:14 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<o2r51a29d951004141442j2b76149re47079ee4735b4a2@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
Message-ID: <x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>

Hi Benedict,

I took a look at your index decomposition routine; it was not working for
seed = 1 until I made index the query index plus one (similar to my r
variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
changes the routine was actually slower than my version (although not by
much).  Let me know if you have a better way to bring your routine in line
with the arraylet structure.

Kevin

On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <lists at laerad.com>wrote:

> Hi Kevin,
>
> Unfortunately this week has been pretty hectic, and I haven't had much time
> to much more than theorise on this topic - and this weekend the weather
> looks set to be much too nice to stay in doors! It looks like you've made
> really good progress on the ChunkedArrayDeque - I haven't fully digested it
> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
> your list implementation prior to reading it in detail) but I think I may
> have been toying with a similar growth strategy for a hash map variant since
> the copies are necessary there anyway.
>
> I have taken another look at the index algorithm and have tidied it up as
> below, and it now supports a seed size of 1; however I am not sure that
> using a value this small is advisable, given that the overhead for the first
> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
> utilise less than 20% of the allocated memory for data, the first 24 less
> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
> advisable.
>
> As an aside, it is worth noting that the indexOffset calculation below can
> be replaced with index & (_backingArray[arrayIndex].length - 1) , although I
> am not certain what effect this will have on performance, given that this
> would be another memory operation to retrieve the length from the array; but
> it would make the separation of the calculation into functions more straight
> forward.
>
>      final int arraySizeShiftMinusSeed = (31 -
> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>
>      final int firstArrayOfThisSize =
>       (1 << arraySizeShiftMinusSeed + 2)
>      - (1 << arraySizeShiftMinusSeed)
>      - 1 - (arraySizeShift >>> 31) ;
>      final int indexRemainder = index - (1 << arraySizeShift <<
> arraySizeShiftMinusSeed) ;
>
>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
> Integer.MAX_VALUE ;
>
>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>
>
>
> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Hi Benedict,
>>
>> Have you had a chance to get your index decomposition procedure to work
>> with seed values less than two?
>>
>> Kevin
>>
>>
>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>> lists at laerad.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> As it happens I might have something useful still to contribute. As an
>>> exercise in saving face I revisited the problem to see if I could achieve
>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>> must admit I still didn't fully appreciate how the algorithm in
>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>> similar properties. What I have ended up with is almost identical except
>>> adds I think a couple of incremental improvements, simply by redefining the
>>> arrayIndex() method. I should note that I have not yet implemented more than
>>> a prototype as it seems to me your implementation is excellent already, and
>>> if it is decided to include my modifications the changes should be modest.
>>>
>>> Firstly, (I hope that) what I have produced is a little more CPU
>>> pipe-line friendly; there is less dependency on immediately preceding
>>> calculations at each stage (i.e. so more operations should be able to
>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>> is approximately the same.
>>>
>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>> specified (although I expect hard coding a suitable one will ultimately be
>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>> 16*s*(..*24)] etc. although when put in simple text like that it does
>>> appear to trivialise the change. The benefit of this, though, is two fold:
>>> 1) for small n the constant factor is reduced (both CPU and memory wise);
>>> and 2) the sqrt(n) bounds are reached more quickly also.
>>>
>>> As an illustration, consider setting *s* to 4, and assume the backing
>>> array is size two and doubles in size with each growth; with
>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as 4
>>> we would instead resize at i=8,i=24,i=80,i=288; the cost at each would be
>>> some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>> seems likely this should also have a positive impact on cache performance
>>> for smaller lists as well.
>>>
>>> Finally, after playing with this idea in my head I am confident I can
>>> extend the core ideas of this data structure to hashing relatively easily,
>>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>>> although I see from Martin's recent mail that this was raised before. Unless
>>> there are better suggestions for solving the hash table problem I will have
>>> a go at it as it seems an interesting problem - that is, assuming there are
>>> no objections?
>>>
>>> I'm interested to hear your thoughts. I hope this time I've been a bit
>>> more considered in what I've put forward, and hence less of a waste of time!
>>>
>>> Code snippet for calculation of array index and item offset:
>>>
>>> final int arraySizeShiftMinusSeed = ((31 -
>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3)
>>> - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>  final int indexRemainder = index - ((1 << seed) <<
>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>
>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>
>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* +
>>> 1); seed only works for values for 2 or more at this moment, fyi
>>>
>>>
>>>
>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>>>> know if you have any other questions or if you have any related ideas or
>>>> concerns.
>>>>
>>>>
>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>>>> help. that part of the discussion was clearly truncated by the time I had
>>>>> joined the list - I haven't been able to find the history in the archives
>>>>> either...
>>>>>
>>>>> I was just wondering about the worst case cost of add() as described by
>>>>> your javadoc; admittedly it is optimal with respect to unused memory, but
>>>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>>>> constant factor. I had been thinking that once n passed a threshold the cost
>>>>> of additions in this other structure would become a constant factor,
>>>>> offering nice algorithmic complexity guarantees for large n; however since
>>>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>>>> allocations would have to be unrealistically small (assuming linear cost for
>>>>> allocation) for this to be the case. It would still be nice to have a data
>>>>> structure that avoids needing to copy data with each grow, whilst still
>>>>> maintaining good memory performance.
>>>>>
>>>>> That *all* being said, I had been going by your javadoc and emails to
>>>>> ascertain the behaviour of this class, as I couldn't locate a free copy
>>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>> constant factor for this implementation still seems to be quite high,
>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>> large values of n.
>>>>>
>>>>>
>>>>>
>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Hi Benedict,
>>>>>>
>>>>>> Unless I am misreading your post, this now has a very similar feel to
>>>>>> the first data structure that I posted to the list.  Martin Buchholz then
>>>>>> pointed out that we can incorporate the ideas from
>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>> been exhaustive in the benchmarking:
>>>>>>>
>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>
>>>>>>> The nice thing about this is that the maximum amount of wasted memory
>>>>>>> is user configurable. Even with a low setting as above (65K) performance
>>>>>>> seems pretty consistent.
>>>>>>>
>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>
>>>>>>> private final int indexFor(int a, int i) {
>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>  }
>>>>>>> private final int arrayFor(int i) {
>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>  }
>>>>>>>
>>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>>> think that one through fully!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi Benedict,
>>>>>>>>
>>>>>>>> Like you, I am relatively new to this mailing list; I am also trying
>>>>>>>> to tread lightly so as not to step on any toes.  That being said, I think
>>>>>>>> that I can offer a response to your inquiry.
>>>>>>>>
>>>>>>>> Regarding:  "The idea is to simply double the new array size each
>>>>>>>> time a new array needs to be allocated"
>>>>>>>>
>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>
>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>
>>>>>>>> What happens with your structure when you add n elements and then
>>>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>> resize.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>
>>>>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>
>>>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>>>> impressive.
>>>>>>>>>
>>>>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>>>>> All statistics were run in two rounds with ~1000 runs per round per
>>>>>>>>> statistic per list, and the second round results were used.
>>>>>>>>>
>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>
>>>>>>>>> and server JVM:
>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>
>>>>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>>>>> expected them to be fairly close.
>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion
>>>>>>>>> at the beginning of large lists, which I haven't yet tried to understand.
>>>>>>>>> Insertion in the middle is similar.
>>>>>>>>>
>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>
>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>> larger lists:
>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>>>>> source code and I will find somewhere to upload it.
>>>>>>>>>
>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>> reverseIterator() method.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin,
>>>>>>>>>>
>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>
>>>>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Joe
>>>>>>>>>>
>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>
>>>>>>>>>> Hi Joe,
>>>>>>>>>>>
>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add
>>>>>>>>>>> does not amortize to constant time when the data structure employs the
>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>
>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>
>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>
>>>>>>>>>>>> I'd be interested if you have any comments in the context of
>>>>>>>>>>>> this discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>
>>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin, I don't fully understand your point about not amortizing
>>>>>>>>>>>> to O(1). Certainly that's true for insert not at head or tail. Otherwise
>>>>>>>>>>>> this implementation only moves array elements to the front on an array
>>>>>>>>>>>> resize operation which happens every O(ln n) operations at most, if we do
>>>>>>>>>>>> lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>
>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Joe
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>> 102          90         240         191         138
>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>> 138          94         230         194         118
>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>> 538          91         324         382         119
>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>  69       32913         162         130         105
>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>> 104       21944         169         134         135
>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>> 506        9602         364         333         526
>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's interesting to note that the old circular list trick will
>>>>>>>>>>>>>> not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>>>> operation with
>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so
>>>>>>>>>>>>>>> 1.00 indicates
>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>>>>> when I switch
>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>>>>> timed the process
>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>>>>> eliminate the
>>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>>>>> achieving this by
>>>>>>>>>>>>>>> > way of a data structure that is both time and space optimal
>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees
>>>>>>>>>>>>>>> that no backing
>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize than
>>>>>>>>>>>>>>> ArrayList.  Of
>>>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due
>>>>>>>>>>>>>>> to the multiple
>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional cost
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation
>>>>>>>>>>>>>>> on ArrayList due
>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>> >> > The data structure is available at the second link that
>>>>>>>>>>>>>>> I originally
>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as
>>>>>>>>>>>>>>> yet as it was
>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a
>>>>>>>>>>>>>>> much more important
>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll
>>>>>>>>>>>>>>> be updating the
>>>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and the
>>>>>>>>>>>>>>> offset into the
>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned
>>>>>>>>>>>>>>> that this will
>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing
>>>>>>>>>>>>>>> this each time an
>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to twiddle
>>>>>>>>>>>>>>> the bits to see
>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 -
>>>>>>>>>>>>>>> 1);
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p +
>>>>>>>>>>>>>>> b) + " " + e);
>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for even
>>>>>>>>>>>>>>> numbered
>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional term
>>>>>>>>>>>>>>> added (the
>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of superblock
>>>>>>>>>>>>>>> in the paper
>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>>>> consists of
>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with i
>>>>>>>>>>>>>>> = 3:
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are
>>>>>>>>>>>>>>> only two data
>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in
>>>>>>>>>>>>>>> my example above,
>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>>>> their data
>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the size)
>>>>>>>>>>>>>>> and to insert
>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo
>>>>>>>>>>>>>>> the size).  We
>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that
>>>>>>>>>>>>>>> is shared by the
>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if there's
>>>>>>>>>>>>>>> a way to turn
>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help and/or
>>>>>>>>>>>>>>> be a better
>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100423/e0d2070e/attachment.html>

From Ulf.Zibis at gmx.de  Sat Apr 24 02:36:31 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Sat, 24 Apr 2010 04:36:31 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD228A3.7010103@oracle.com>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com>
Message-ID: <4BD2592F.2010300@gmx.de>

Am 24.04.2010 01:09, schrieb Xueming Shen:
> Ulf Zibis wrote:
>>
>> - I like the idea, saving the data in a compressed binary file, 
>> instead classfile static data.
>> - wouldn't PreHashMaps be faster initialized as a normal HashMaps in 
>> j.l.Character.UnicodeScript and j.l.CharacterName?
> I don't think so. The key for these 2 cases is the whole unicode range.

At least the aliases map has string keys.

> But you can always try. Current
> binary-search for script definitely is not a perfect solution.

In most cases you don't have an exact match from the HashMap of 
CharacterName, so then you anyway have to do the binary search.

>
>> - As alternative to lookup in a hash table, I guess retrieving the 
>> pointers from a memory saving sorted array via binary search would be 
>> fast enough.
>> - j.l.CharacterName:
>> -- You could instantiate the HashMap with capacity=cpLeng
> I changed the data file "format" a bit, so now the overal uniName.dat 
> is less than 88k (last version is 122+k), but

Is this compressed size or un-compressed ?

> the I can no long use cpLen as the capacity for the hashmap. I'm now 
> using a hardcoded 20000 for 5.2.

You could pre-calculate the actual value by help of 
generatecharacter/CharacterName.java

>
> I believe you would NOT see any meaningful performance boost from 
> using DirectByteBuffer, given the
> size of the data file, 88k. It probably will slow it down a little.

If you read the whole file, yes, but retrieving a single data from a 
distinct position ?


>
> If you take a look at the last version
> http://cr.openjdk.java.net/~sherman/script/webrev/src/share/classes/java/lang/CharacterName.java.html 
>
> You probably will not consider to use DataInputStream class. I no 
> longer store the code point value for
> most entries, one the length of the name, in which 1 byte is 
> definitely big enough.

You could save one more byte:

   66             do {
   67                 int len = ba[off++]&  0xff;
   68                 if (len<  0x11) {
   69                     // always big-endian
   70                     cp = (len<<  16) |
   71                          ((ba[off++]&  0xff)<<   8) |
   72                          ((ba[off++]&  0xff));
   73                     len = ba[off++]&  0xff;
   74
   75                 }  else {
   76                     len -= 0x11;
   77                     cp++;
   78                 }

>
> Yes, the final table takes about 500k, we might consider to use a 
> weakref or something, if memory really
> a concern. But the table will get initialized only if you invoke 
> Character.getName(),

Yes, retrieving one single Character.getName() would cause the whole map 
to initialize. Is that economic?

> I would expect most
> of the application would never get down there.
>
>>
>>>
>>> (1) to use enum for the j.l.Character.UnicodeScript (compared to the 
>>> traditional j.l.c.Subset)
>>
>> - enum j.l.Character.UnicodeScript:
>> -- IIRC, enums internally are handled as int constants, so retrieving 
>> an element via name would need a name->int lookup
>> -- So UnicodeScript.forName would have to lookup 2 times
>> --- alias->fullName (name of enum element)
>> --- fullName->internal int constant
>> -- I suggest to add the full names to the aliasses map and only 
>> lookup once.
> Not really. It's not alias->fullName, it's alias->UnicodeScript 
> costant. So if the passed in is an alias, then
> we don't do the second lookup.
This I wanted to say, sorry about not being more detailed.

> That said, it's always a trade-off of memory use and speed. To put all
> full name in aliases map definitely will reduce the second lookup if 
> the passed in is a canonical name, with
> the price of having name entries in both alias map and enum's internal 
> hashmap.

~100 * (4 + 4) bytes against the above 500.000 bytes, does that matter ?

> I really don't know which
> one is a better choice. I did it this way with the assumption the 
> lookup for script name is not critical. I
> might be wrong.
>
>
>> -- Why don't you use Arrays.binarySearch in UnicodeScript.of(int 
>> codePoint) ?
>>
>>
>
> why? I don't know:-) Maybe the copy/paste from UnicodeBlock lookup is 
> more convenient than using
> the Arrays.binarySearch. Not a big deal.

So both could use Arrays.binarySearch ;-)

-Ulf





From lists at laerad.com  Sat Apr 24 07:24:45 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sat, 24 Apr 2010 08:24:45 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<i2s1704b7a21004141844n68f3c762je104d36a079ade6d@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
Message-ID: <g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>

Hi Kevin,

It looks like this is because ChunkedArrayList creates only one initial
array of size s, whereas my algorithm expects two . Apologies for not
spotting this - the pattern of allocations is identical after this point.
I'll see if it can be modified to support your pattern of allocation.

On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> I took a look at your index decomposition routine; it was not working for
> seed = 1 until I made index the query index plus one (similar to my r
> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
> changes the routine was actually slower than my version (although not by
> much).  Let me know if you have a better way to bring your routine in line
> with the arraylet structure.
>
> Kevin
>
>
> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <lists at laerad.com>wrote:
>
>> Hi Kevin,
>>
>> Unfortunately this week has been pretty hectic, and I haven't had much
>> time to much more than theorise on this topic - and this weekend the weather
>> looks set to be much too nice to stay in doors! It looks like you've made
>> really good progress on the ChunkedArrayDeque - I haven't fully digested it
>> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
>> your list implementation prior to reading it in detail) but I think I may
>> have been toying with a similar growth strategy for a hash map variant since
>> the copies are necessary there anyway.
>>
>> I have taken another look at the index algorithm and have tidied it up as
>> below, and it now supports a seed size of 1; however I am not sure that
>> using a value this small is advisable, given that the overhead for the first
>> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
>> utilise less than 20% of the allocated memory for data, the first 24 less
>> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
>> advisable.
>>
>> As an aside, it is worth noting that the indexOffset calculation below can
>> be replaced with index & (_backingArray[arrayIndex].length - 1) , although I
>> am not certain what effect this will have on performance, given that this
>> would be another memory operation to retrieve the length from the array; but
>> it would make the separation of the calculation into functions more straight
>> forward.
>>
>>      final int arraySizeShiftMinusSeed = (31 -
>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>
>>      final int firstArrayOfThisSize =
>>       (1 << arraySizeShiftMinusSeed + 2)
>>      - (1 << arraySizeShiftMinusSeed)
>>      - 1 - (arraySizeShift >>> 31) ;
>>      final int indexRemainder = index - (1 << arraySizeShift <<
>> arraySizeShiftMinusSeed) ;
>>
>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>> Integer.MAX_VALUE ;
>>
>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>
>>
>>
>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Hi Benedict,
>>>
>>> Have you had a chance to get your index decomposition procedure to work
>>> with seed values less than two?
>>>
>>> Kevin
>>>
>>>
>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> As it happens I might have something useful still to contribute. As an
>>>> exercise in saving face I revisited the problem to see if I could achieve
>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>> must admit I still didn't fully appreciate how the algorithm in
>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>> similar properties. What I have ended up with is almost identical except
>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>> a prototype as it seems to me your implementation is excellent already, and
>>>> if it is decided to include my modifications the changes should be modest.
>>>>
>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>> calculations at each stage (i.e. so more operations should be able to
>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>> is approximately the same.
>>>>
>>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>>> specified (although I expect hard coding a suitable one will ultimately be
>>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>>> 16*s*(..*24)] etc. although when put in simple text like that it does
>>>> appear to trivialise the change. The benefit of this, though, is two fold:
>>>> 1) for small n the constant factor is reduced (both CPU and memory wise);
>>>> and 2) the sqrt(n) bounds are reached more quickly also.
>>>>
>>>> As an illustration, consider setting *s* to 4, and assume the backing
>>>> array is size two and doubles in size with each growth; with
>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as 4
>>>> we would instead resize at i=8,i=24,i=80,i=288; the cost at each would be
>>>> some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>> seems likely this should also have a positive impact on cache performance
>>>> for smaller lists as well.
>>>>
>>>> Finally, after playing with this idea in my head I am confident I can
>>>> extend the core ideas of this data structure to hashing relatively easily,
>>>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>>>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>>>> although I see from Martin's recent mail that this was raised before. Unless
>>>> there are better suggestions for solving the hash table problem I will have
>>>> a go at it as it seems an interesting problem - that is, assuming there are
>>>> no objections?
>>>>
>>>> I'm interested to hear your thoughts. I hope this time I've been a bit
>>>> more considered in what I've put forward, and hence less of a waste of time!
>>>>
>>>> Code snippet for calculation of array index and item offset:
>>>>
>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed + 3)
>>>> - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>
>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>
>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s* +
>>>> 1); seed only works for values for 2 or more at this moment, fyi
>>>>
>>>>
>>>>
>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>>>>> know if you have any other questions or if you have any related ideas or
>>>>> concerns.
>>>>>
>>>>>
>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>> lists at laerad.com> wrote:
>>>>>
>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>>>>> help. that part of the discussion was clearly truncated by the time I had
>>>>>> joined the list - I haven't been able to find the history in the archives
>>>>>> either...
>>>>>>
>>>>>> I was just wondering about the worst case cost of add() as described
>>>>>> by your javadoc; admittedly it is optimal with respect to unused memory, but
>>>>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>>>>> constant factor. I had been thinking that once n passed a threshold the cost
>>>>>> of additions in this other structure would become a constant factor,
>>>>>> offering nice algorithmic complexity guarantees for large n; however since
>>>>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>>>>> allocations would have to be unrealistically small (assuming linear cost for
>>>>>> allocation) for this to be the case. It would still be nice to have a data
>>>>>> structure that avoids needing to copy data with each grow, whilst still
>>>>>> maintaining good memory performance.
>>>>>>
>>>>>> That *all* being said, I had been going by your javadoc and emails to
>>>>>> ascertain the behaviour of this class, as I couldn't locate a free copy
>>>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>> large values of n.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>
>>>>>>> Hi Benedict,
>>>>>>>
>>>>>>> Unless I am misreading your post, this now has a very similar feel to
>>>>>>> the first data structure that I posted to the list.  Martin Buchholz then
>>>>>>> pointed out that we can incorporate the ideas from
>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>> lists at laerad.com> wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>>> been exhaustive in the benchmarking:
>>>>>>>>
>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>
>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>> performance seems pretty consistent.
>>>>>>>>
>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>
>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>  }
>>>>>>>> private final int arrayFor(int i) {
>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>  }
>>>>>>>>
>>>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>>>> think that one through fully!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Hi Benedict,
>>>>>>>>>
>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>
>>>>>>>>> Regarding:  "The idea is to simply double the new array size each
>>>>>>>>> time a new array needs to be allocated"
>>>>>>>>>
>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>
>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>
>>>>>>>>> What happens with your structure when you add n elements and then
>>>>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>> resize.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>
>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion, but I
>>>>>>>>>> have tinkered with a "chunked" array list which seems to offer better time
>>>>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>
>>>>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>>>>> impressive.
>>>>>>>>>>
>>>>>>>>>> Some random statistics for addition only on the client JVM (I have
>>>>>>>>>> quickly dubbed my implementation ExpArrayList)
>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per round
>>>>>>>>>> per statistic per list, and the second round results were used.
>>>>>>>>>>
>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>
>>>>>>>>>> and server JVM:
>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>
>>>>>>>>>> Interestingly insertion at the beginning of the list appears to be
>>>>>>>>>> quicker with ExpArrayList, at least on the server JVM, whereas I would have
>>>>>>>>>> expected them to be fairly close.
>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for insertion
>>>>>>>>>> at the beginning of large lists, which I haven't yet tried to understand.
>>>>>>>>>> Insertion in the middle is similar.
>>>>>>>>>>
>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>
>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>> larger lists:
>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I'm willing to explore this further but I'm not sure how desirable
>>>>>>>>>> that is, given that Kevin's data structure appears to perform pretty well
>>>>>>>>>> already wrt to CPU time, and better wrt to memory utilisation, and in effect
>>>>>>>>>> this mostly changes only the function to determine which array to use, not
>>>>>>>>>> the body of the implementation. Let me know if you would like a copy of the
>>>>>>>>>> source code and I will find somewhere to upload it.
>>>>>>>>>>
>>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <joe.j.kearney at googlemail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>
>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>
>>>>>>>>>>> The idea is to get a replacement for arraylist that performs like
>>>>>>>>>>> arraydeque on remove(0). As a side effect, we should be able to get better
>>>>>>>>>>> performance on other operations by requiring fewer elements to be moved.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Joe
>>>>>>>>>>>
>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>
>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add
>>>>>>>>>>>> does not amortize to constant time when the data structure employs the
>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd be interested if you have any comments in the context of
>>>>>>>>>>>>> this discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin, I don't fully understand your point about not amortizing
>>>>>>>>>>>>> to O(1). Certainly that's true for insert not at head or tail. Otherwise
>>>>>>>>>>>>> this implementation only moves array elements to the front on an array
>>>>>>>>>>>>> resize operation which happens every O(ln n) operations at most, if we do
>>>>>>>>>>>>> lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>   102          90         240         191         138
>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>> 138          94         230         194         118
>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>> 538          91         324         382         119
>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>    69       32913         162         130         105
>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>> 104       21944         169         134         135
>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>> 506        9602         364         333         526
>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's interesting to note that the old circular list trick
>>>>>>>>>>>>>>> will not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>>>>> operation with
>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so
>>>>>>>>>>>>>>>> 1.00 indicates
>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>> > relatively significant variability in a few of the numbers
>>>>>>>>>>>>>>>> when I switch
>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and then
>>>>>>>>>>>>>>>> timed the process
>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view to
>>>>>>>>>>>>>>>> eliminate the
>>>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion of
>>>>>>>>>>>>>>>> achieving this by
>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees
>>>>>>>>>>>>>>>> that no backing
>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize
>>>>>>>>>>>>>>>> than ArrayList.  Of
>>>>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly, due
>>>>>>>>>>>>>>>> to the multiple
>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional cost
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation
>>>>>>>>>>>>>>>> on ArrayList due
>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>> >> > The data structure is available at the second link that
>>>>>>>>>>>>>>>> I originally
>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as
>>>>>>>>>>>>>>>> yet as it was
>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a
>>>>>>>>>>>>>>>> much more important
>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that role.
>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll
>>>>>>>>>>>>>>>> be updating the
>>>>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and
>>>>>>>>>>>>>>>> the offset into the
>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned
>>>>>>>>>>>>>>>> that this will
>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple bit
>>>>>>>>>>>>>>>> shift and a bit
>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing
>>>>>>>>>>>>>>>> this each time an
>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 -
>>>>>>>>>>>>>>>> 1);
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p +
>>>>>>>>>>>>>>>> b) + " " + e);
>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for
>>>>>>>>>>>>>>>> even numbered
>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional
>>>>>>>>>>>>>>>> term added (the
>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>>>>> consists of
>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with
>>>>>>>>>>>>>>>> i = 3:
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are
>>>>>>>>>>>>>>>> only two data
>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in
>>>>>>>>>>>>>>>> my example above,
>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>>>>> their data
>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo
>>>>>>>>>>>>>>>> the size).  We
>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet that
>>>>>>>>>>>>>>>> is shared by the
>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/e6eaec4c/attachment.html>

From lists at laerad.com  Sat Apr 24 08:34:56 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sat, 24 Apr 2010 09:34:56 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<p2q51a29d951004150207v8d3c4150pefacfe6bda5d730c@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
Message-ID: <i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>

Hi Kevin,

If you are willing to change the pattern of allocation just slightly, I have
come up with an alternate algorithm (optimised for the seed = 1 case)
that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
for random(ish) calls to get() on a list of size 1 << 24. The change is to
simply drop your first array allocation (so that there are no arrays of size
1, but that all remaining allocations follow the existing pattern), as this
allows simplifying the algorithm noticeably (several ops in the previous
algorithm were unnecessary for any but the first two arrays).

My get(index) method is defined as:

if ((index < 0) | (index >= _size))
throw new IllegalArgumentException() ;
index += 2 ;
final Object[] array = _backingArray[arrayFor(index)] ;
return (T) array[index & (array.length - 1)] ;

and arrayFor(index) is defined as:

private static final int arrayFor(int index) {
     final int arraySizeShiftMinusSeed = (31 -
Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed) ;
     final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
final int b2 = index - b1 ;
     final int a2 = (1 << arraySizeShiftMinusSeed) ;
final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
final int av = a1 - a2 ;
final int bv = b4 - 3 ;
return av + bv ;
}

I have deliberately interleaved the calculations here to make sure the
pipeline is being used (just in case javac+hotspot are not re-ordering the
intermediate calculations for us)





On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com> wrote:

> Hi Kevin,
>
> It looks like this is because ChunkedArrayList creates only one initial
> array of size s, whereas my algorithm expects two . Apologies for not
> spotting this - the pattern of allocations is identical after this point.
> I'll see if it can be modified to support your pattern of allocation.
>
>
> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Hi Benedict,
>>
>> I took a look at your index decomposition routine; it was not working for
>> seed = 1 until I made index the query index plus one (similar to my r
>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>> changes the routine was actually slower than my version (although not by
>> much).  Let me know if you have a better way to bring your routine in line
>> with the arraylet structure.
>>
>> Kevin
>>
>>
>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <lists at laerad.com
>> > wrote:
>>
>>> Hi Kevin,
>>>
>>> Unfortunately this week has been pretty hectic, and I haven't had much
>>> time to much more than theorise on this topic - and this weekend the weather
>>> looks set to be much too nice to stay in doors! It looks like you've made
>>> really good progress on the ChunkedArrayDeque - I haven't fully digested it
>>> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
>>> your list implementation prior to reading it in detail) but I think I may
>>> have been toying with a similar growth strategy for a hash map variant since
>>> the copies are necessary there anyway.
>>>
>>> I have taken another look at the index algorithm and have tidied it up as
>>> below, and it now supports a seed size of 1; however I am not sure that
>>> using a value this small is advisable, given that the overhead for the first
>>> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
>>> utilise less than 20% of the allocated memory for data, the first 24 less
>>> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
>>> advisable.
>>>
>>> As an aside, it is worth noting that the indexOffset calculation below
>>> can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>> although I am not certain what effect this will have on performance, given
>>> that this would be another memory operation to retrieve the length from the
>>> array; but it would make the separation of the calculation into functions
>>> more straight forward.
>>>
>>>      final int arraySizeShiftMinusSeed = (31 -
>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>
>>>      final int firstArrayOfThisSize =
>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>      - (1 << arraySizeShiftMinusSeed)
>>>      - 1 - (arraySizeShift >>> 31) ;
>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>> arraySizeShiftMinusSeed) ;
>>>
>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>> Integer.MAX_VALUE ;
>>>
>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>
>>>
>>>
>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Hi Benedict,
>>>>
>>>> Have you had a chance to get your index decomposition procedure to work
>>>> with seed values less than two?
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> As it happens I might have something useful still to contribute. As an
>>>>> exercise in saving face I revisited the problem to see if I could achieve
>>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>>> must admit I still didn't fully appreciate how the algorithm in
>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>> similar properties. What I have ended up with is almost identical except
>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>
>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>> is approximately the same.
>>>>>
>>>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>>>> specified (although I expect hard coding a suitable one will ultimately be
>>>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>>>> 16*s*(..*24)] etc. although when put in simple text like that it does
>>>>> appear to trivialise the change. The benefit of this, though, is two fold:
>>>>> 1) for small n the constant factor is reduced (both CPU and memory wise);
>>>>> and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>
>>>>> As an illustration, consider setting *s* to 4, and assume the backing
>>>>> array is size two and doubles in size with each growth; with
>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as
>>>>> 4 we would instead resize at i=8,i=24,i=80,i=288; the cost at each would be
>>>>> some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>>> seems likely this should also have a positive impact on cache performance
>>>>> for smaller lists as well.
>>>>>
>>>>> Finally, after playing with this idea in my head I am confident I can
>>>>> extend the core ideas of this data structure to hashing relatively easily,
>>>>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>>>>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>>>>> although I see from Martin's recent mail that this was raised before. Unless
>>>>> there are better suggestions for solving the hash table problem I will have
>>>>> a go at it as it seems an interesting problem - that is, assuming there are
>>>>> no objections?
>>>>>
>>>>> I'm interested to hear your thoughts. I hope this time I've been a bit
>>>>> more considered in what I've put forward, and hence less of a waste of time!
>>>>>
>>>>> Code snippet for calculation of array index and item offset:
>>>>>
>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed +
>>>>> 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>
>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>
>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s*+ 1); seed only works for values for 2 or more at this moment, fyi
>>>>>
>>>>>
>>>>>
>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let me
>>>>>> know if you have any other questions or if you have any related ideas or
>>>>>> concerns.
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>>>>>> help. that part of the discussion was clearly truncated by the time I had
>>>>>>> joined the list - I haven't been able to find the history in the archives
>>>>>>> either...
>>>>>>>
>>>>>>> I was just wondering about the worst case cost of add() as described
>>>>>>> by your javadoc; admittedly it is optimal with respect to unused memory, but
>>>>>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>>>>>> constant factor. I had been thinking that once n passed a threshold the cost
>>>>>>> of additions in this other structure would become a constant factor,
>>>>>>> offering nice algorithmic complexity guarantees for large n; however since
>>>>>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>>>>>> allocations would have to be unrealistically small (assuming linear cost for
>>>>>>> allocation) for this to be the case. It would still be nice to have a data
>>>>>>> structure that avoids needing to copy data with each grow, whilst still
>>>>>>> maintaining good memory performance.
>>>>>>>
>>>>>>> That *all* being said, I had been going by your javadoc and emails
>>>>>>> to ascertain the behaviour of this class, as I couldn't locate a free copy
>>>>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>> large values of n.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi Benedict,
>>>>>>>>
>>>>>>>> Unless I am misreading your post, this now has a very similar feel
>>>>>>>> to the first data structure that I posted to the list.  Martin Buchholz then
>>>>>>>> pointed out that we can incorporate the ideas from
>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>>
>>>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>>>> been exhaustive in the benchmarking:
>>>>>>>>>
>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>
>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>
>>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>>
>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>  }
>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>>  }
>>>>>>>>>
>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>>>>> think that one through fully!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Benedict,
>>>>>>>>>>
>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>
>>>>>>>>>> Regarding:  "The idea is to simply double the new array size each
>>>>>>>>>> time a new array needs to be allocated"
>>>>>>>>>>
>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>
>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it seems
>>>>>>>>>> that the simplest solution would be to simply have two lists, with one
>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>
>>>>>>>>>> What happens with your structure when you add n elements and then
>>>>>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>> resize.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>
>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion, but
>>>>>>>>>>> I have tinkered with a "chunked" array list which seems to offer better time
>>>>>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>
>>>>>>>>>>> I have prototyped the data structure this evening and benchmarked
>>>>>>>>>>> additions at the end of the list, for which the performance is pretty
>>>>>>>>>>> impressive.
>>>>>>>>>>>
>>>>>>>>>>> Some random statistics for addition only on the client JVM (I
>>>>>>>>>>> have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per round
>>>>>>>>>>> per statistic per list, and the second round results were used.
>>>>>>>>>>>
>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>
>>>>>>>>>>> and server JVM:
>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>
>>>>>>>>>>> Interestingly insertion at the beginning of the list appears to
>>>>>>>>>>> be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>
>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>
>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>> larger lists:
>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>
>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>
>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>
>>>>>>>>>>>> The idea is to get a replacement for arraylist that performs
>>>>>>>>>>>> like arraydeque on remove(0). As a side effect, we should be able to get
>>>>>>>>>>>> better performance on other operations by requiring fewer elements to be
>>>>>>>>>>>> moved.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Joe
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add
>>>>>>>>>>>>> does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd be interested if you have any comments in the context of
>>>>>>>>>>>>>> this discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>   102          90         240         191         138
>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>> 138          94         230         194         118
>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>> 538          91         324         382         119
>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>  4425          94        1296        2330         124
>>>>>>>>>>>>>> ---------------------------------------------------- ArrayList
>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>    69       32913         162         130         105
>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>> 104       21944         169         134         135
>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>> 506        9602         364         333         526
>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>  4414        9947        2312        2280        4437
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's interesting to note that the old circular list trick
>>>>>>>>>>>>>>>> will not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik D.
>>>>>>>>>>>>>>>> Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs
>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the specified
>>>>>>>>>>>>>>>>> operation with
>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so
>>>>>>>>>>>>>>>>> 1.00 indicates
>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and
>>>>>>>>>>>>>>>>> then timed the process
>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally I
>>>>>>>>>>>>>>>>> used the iterator
>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away the
>>>>>>>>>>>>>>>>> timing for the
>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view
>>>>>>>>>>>>>>>>> to eliminate the
>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion
>>>>>>>>>>>>>>>>> of achieving this by
>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees
>>>>>>>>>>>>>>>>> that no backing
>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead than
>>>>>>>>>>>>>>>>> ArrayList, and
>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize
>>>>>>>>>>>>>>>>> than ArrayList.  Of
>>>>>>>>>>>>>>>>> > course, this data structure does not do everything better
>>>>>>>>>>>>>>>>> than ArrayList; in
>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly,
>>>>>>>>>>>>>>>>> due to the multiple
>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That being
>>>>>>>>>>>>>>>>> said, I think that
>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional
>>>>>>>>>>>>>>>>> cost of
>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable operation
>>>>>>>>>>>>>>>>> on ArrayList due
>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>> >> > The data structure is available at the second link
>>>>>>>>>>>>>>>>> that I originally
>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front as
>>>>>>>>>>>>>>>>> yet as it was
>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support efficient
>>>>>>>>>>>>>>>>> add-at-front or
>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a
>>>>>>>>>>>>>>>>> much more important
>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so I'll
>>>>>>>>>>>>>>>>> be updating the
>>>>>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the bug.
>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and
>>>>>>>>>>>>>>>>> the offset into the
>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned
>>>>>>>>>>>>>>>>> that this will
>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple
>>>>>>>>>>>>>>>>> bit shift and a bit
>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing
>>>>>>>>>>>>>>>>> this each time an
>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2 -
>>>>>>>>>>>>>>>>> 1);
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p +
>>>>>>>>>>>>>>>>> b) + " " + e);
>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for
>>>>>>>>>>>>>>>>> even numbered
>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional
>>>>>>>>>>>>>>>>> term added (the
>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated, it
>>>>>>>>>>>>>>>>> consists of
>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i), with
>>>>>>>>>>>>>>>>> i = 3:
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are
>>>>>>>>>>>>>>>>> only two data
>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better in
>>>>>>>>>>>>>>>>> my example above,
>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation of
>>>>>>>>>>>>>>>>> their data
>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer (modulo
>>>>>>>>>>>>>>>>> the size).  We
>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet
>>>>>>>>>>>>>>>>> that is shared by the
>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/48fdc614/attachment.html>

From alan.bateman at sun.com  Sat Apr 24 16:40:28 2010
From: alan.bateman at sun.com (alan.bateman at sun.com)
Date: Sat, 24 Apr 2010 16:40:28 +0000
Subject: hg: jdk7/tl/corba: 6939646: Remove obsolete
	com.sun.corba.se.internal.io package
Message-ID: <20100424164029.55B224416A@hg.openjdk.java.net>

Changeset: cefae6b4a590
Author:    alanb
Date:      2010-04-24 17:09 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/corba/rev/cefae6b4a590

6939646: Remove obsolete com.sun.corba.se.internal.io package
Reviewed-by: ohair

! make/sun/corba/Makefile
- make/sun/corba/core/Makefile
- make/sun/corba/core/mapfile-vers
- src/share/classes/com/sun/corba/se/internal/io/IIOPInputStream.java
- src/share/classes/com/sun/corba/se/internal/io/IIOPOutputStream.java
- src/share/classes/com/sun/corba/se/internal/io/LibraryManager.java
- src/share/classes/com/sun/corba/se/internal/io/ObjectStreamClass.java
- src/share/native/com/sun/corba/se/internal/io/ioser.c



From lists at laerad.com  Sat Apr 24 17:30:29 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sat, 24 Apr 2010 18:30:29 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<g2w1704b7a21004150412idd1d5fc4v59369e3078e4e6d8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
	<i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
Message-ID: <w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>

If you want a drop in replacement with minimal fuss to test it out, try
below (although it's a quick hack of including the
Integer.numberOfLeadingZeros() call to avoid its now unnecessary non-zero
test, and instead utilise this op to return the zero index). Just delete the
existing get() method and replace it with this code. Actually this version
seems to improve throughput further - on my laptop regular ChunkedArrayList
is currently averaging around 790ms per run of my hacky speed test, whereas
with the modifications below it averages around 610ms, making it almost 25%
faster. I've included my performance benchmark as well, for reference. I'm
sure with some thought a better one could be put together.

    private static final int arrayIndex2(int index) {
        if (index == 1)
            return 0 ;
        int i = index >>> 1 ;
        int n = 1;
        if (i >>> 16 == 0) { n += 16; i <<= 16; }
        if (i >>> 24 == 0) { n +=  8; i <<=  8; }
        if (i >>> 28 == 0) { n +=  4; i <<=  4; }
        if (i >>> 30 == 0) { n +=  2; i <<=  2; }
        n -= i >>> 31;
     final int arraySizeShiftMinusSeed = (31 - n) >>> 1 ;
 final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed) ;
     final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
final int b2 = index - b1 ;
     final int a2 = (1 << arraySizeShiftMinusSeed) ;
final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
 final int av = a1 - a2 ;
final int bv = b4 - 2 ;
 return av + bv ;
    }

    @Override
    public T get(int index) {
        if ((0 > index) | (index >= _size))
            throw new IndexOutOfBoundsException();
        index += 1 ;
        final Object[] array = _backingArray[arrayIndex2(index)] ;
        return (T) array[index & (array.length - 1)] ;
    }


//// benchmarked by

static final int count = 1 << 24 ;
static final int mask = count - 1 ;
public static void main(String[] args) {
double sum = 0 ;
final List<Integer> list = new ChunkedArrayList<Integer>() ;
for (int i = 0 ; i != count ; i++)
list.add(i) ;
for (int r = 0 ; r != 100 ; r++) {
final long start = System.currentTimeMillis() ;
int o = 0 ;
for (int j = 0 ; j != count ; j++) {
list.get((j + o) & mask) ;
o += 1 ;
}
final long end = System.currentTimeMillis() ;
sum += (end - start) ;
                        System.out.println(String.format("run %d: %dms;
avg=%.0fms", r, (end - start), sum / (r + 1))) ;
}
}


On 24 April 2010 09:34, Benedict Elliott Smith <lists at laerad.com> wrote:

> Hi Kevin,
>
> If you are willing to change the pattern of allocation just slightly, I
> have come up with an alternate algorithm (optimised for the seed = 1 case)
> that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
> for random(ish) calls to get() on a list of size 1 << 24. The change is to
> simply drop your first array allocation (so that there are no arrays of size
> 1, but that all remaining allocations follow the existing pattern), as this
> allows simplifying the algorithm noticeably (several ops in the previous
> algorithm were unnecessary for any but the first two arrays).
>
> My get(index) method is defined as:
>
> if ((index < 0) | (index >= _size))
> throw new IllegalArgumentException() ;
>  index += 2 ;
> final Object[] array = _backingArray[arrayFor(index)] ;
>  return (T) array[index & (array.length - 1)] ;
>
> and arrayFor(index) is defined as:
>
> private static final int arrayFor(int index) {
>      final int arraySizeShiftMinusSeed = (31 -
> Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
> final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed) ;
>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
> final int b2 = index - b1 ;
>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>  final int av = a1 - a2 ;
> final int bv = b4 - 3 ;
> return av + bv ;
>  }
>
> I have deliberately interleaved the calculations here to make sure the
> pipeline is being used (just in case javac+hotspot are not re-ordering the
> intermediate calculations for us)
>
>
>
>
>
> On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com> wrote:
>
>> Hi Kevin,
>>
>> It looks like this is because ChunkedArrayList creates only one initial
>> array of size s, whereas my algorithm expects two . Apologies for not
>> spotting this - the pattern of allocations is identical after this point.
>> I'll see if it can be modified to support your pattern of allocation.
>>
>>
>> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Hi Benedict,
>>>
>>> I took a look at your index decomposition routine; it was not working for
>>> seed = 1 until I made index the query index plus one (similar to my r
>>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>>> changes the routine was actually slower than my version (although not by
>>> much).  Let me know if you have a better way to bring your routine in line
>>> with the arraylet structure.
>>>
>>> Kevin
>>>
>>>
>>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> Unfortunately this week has been pretty hectic, and I haven't had much
>>>> time to much more than theorise on this topic - and this weekend the weather
>>>> looks set to be much too nice to stay in doors! It looks like you've made
>>>> really good progress on the ChunkedArrayDeque - I haven't fully digested it
>>>> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
>>>> your list implementation prior to reading it in detail) but I think I may
>>>> have been toying with a similar growth strategy for a hash map variant since
>>>> the copies are necessary there anyway.
>>>>
>>>> I have taken another look at the index algorithm and have tidied it up
>>>> as below, and it now supports a seed size of 1; however I am not sure that
>>>> using a value this small is advisable, given that the overhead for the first
>>>> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
>>>> utilise less than 20% of the allocated memory for data, the first 24 less
>>>> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
>>>> advisable.
>>>>
>>>> As an aside, it is worth noting that the indexOffset calculation below
>>>> can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>>> although I am not certain what effect this will have on performance, given
>>>> that this would be another memory operation to retrieve the length from the
>>>> array; but it would make the separation of the calculation into functions
>>>> more straight forward.
>>>>
>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>
>>>>      final int firstArrayOfThisSize =
>>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>>      - (1 << arraySizeShiftMinusSeed)
>>>>      - 1 - (arraySizeShift >>> 31) ;
>>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>>> arraySizeShiftMinusSeed) ;
>>>>
>>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>>> Integer.MAX_VALUE ;
>>>>
>>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>
>>>>
>>>>
>>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Hi Benedict,
>>>>>
>>>>> Have you had a chance to get your index decomposition procedure to work
>>>>> with seed values less than two?
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>>> lists at laerad.com> wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> As it happens I might have something useful still to contribute. As an
>>>>>> exercise in saving face I revisited the problem to see if I could achieve
>>>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>>>> must admit I still didn't fully appreciate how the algorithm in
>>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>>> similar properties. What I have ended up with is almost identical except
>>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>>
>>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>>> is approximately the same.
>>>>>>
>>>>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>>>>> specified (although I expect hard coding a suitable one will ultimately be
>>>>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>>>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>>>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>>>>> 16*s*(..*24)] etc. although when put in simple text like that it does
>>>>>> appear to trivialise the change. The benefit of this, though, is two fold:
>>>>>> 1) for small n the constant factor is reduced (both CPU and memory wise);
>>>>>> and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>>
>>>>>> As an illustration, consider setting *s* to 4, and assume the backing
>>>>>> array is size two and doubles in size with each growth; with
>>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s* as
>>>>>> 4 we would instead resize at i=8,i=24,i=80,i=288; the cost at each would be
>>>>>> some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>>>> seems likely this should also have a positive impact on cache performance
>>>>>> for smaller lists as well.
>>>>>>
>>>>>> Finally, after playing with this idea in my head I am confident I can
>>>>>> extend the core ideas of this data structure to hashing relatively easily,
>>>>>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>>>>>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>>>>>> although I see from Martin's recent mail that this was raised before. Unless
>>>>>> there are better suggestions for solving the hash table problem I will have
>>>>>> a go at it as it seems an interesting problem - that is, assuming there are
>>>>>> no objections?
>>>>>>
>>>>>> I'm interested to hear your thoughts. I hope this time I've been a bit
>>>>>> more considered in what I've put forward, and hence less of a waste of time!
>>>>>>
>>>>>> Code snippet for calculation of array index and item offset:
>>>>>>
>>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed +
>>>>>> 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>
>>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>
>>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s*+ 1); seed only works for values for 2 or more at this moment, fyi
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>
>>>>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let
>>>>>>> me know if you have any other questions or if you have any related ideas or
>>>>>>> concerns.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>>> lists at laerad.com> wrote:
>>>>>>>
>>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance than
>>>>>>>> help. that part of the discussion was clearly truncated by the time I had
>>>>>>>> joined the list - I haven't been able to find the history in the archives
>>>>>>>> either...
>>>>>>>>
>>>>>>>> I was just wondering about the worst case cost of add() as described
>>>>>>>> by your javadoc; admittedly it is optimal with respect to unused memory, but
>>>>>>>> the worst case cost of an add is still sqrt(n), with a relatively high
>>>>>>>> constant factor. I had been thinking that once n passed a threshold the cost
>>>>>>>> of additions in this other structure would become a constant factor,
>>>>>>>> offering nice algorithmic complexity guarantees for large n; however since
>>>>>>>> sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new array
>>>>>>>> allocations would have to be unrealistically small (assuming linear cost for
>>>>>>>> allocation) for this to be the case. It would still be nice to have a data
>>>>>>>> structure that avoids needing to copy data with each grow, whilst still
>>>>>>>> maintaining good memory performance.
>>>>>>>>
>>>>>>>> That *all* being said, I had been going by your javadoc and emails
>>>>>>>> to ascertain the behaviour of this class, as I couldn't locate a free copy
>>>>>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>>> large values of n.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Hi Benedict,
>>>>>>>>>
>>>>>>>>> Unless I am misreading your post, this now has a very similar feel
>>>>>>>>> to the first data structure that I posted to the list.  Martin Buchholz then
>>>>>>>>> pointed out that we can incorporate the ideas from
>>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin,
>>>>>>>>>>
>>>>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>>>>> been exhaustive in the benchmarking:
>>>>>>>>>>
>>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>>
>>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>>
>>>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>>>
>>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>>  }
>>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>>>  }
>>>>>>>>>>
>>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>>>>>> think that one through fully!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>
>>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>>
>>>>>>>>>>> Regarding:  "The idea is to simply double the new array size each
>>>>>>>>>>> time a new array needs to be allocated"
>>>>>>>>>>>
>>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>>
>>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it
>>>>>>>>>>> seems that the simplest solution would be to simply have two lists, with one
>>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>>
>>>>>>>>>>> What happens with your structure when you add n elements and then
>>>>>>>>>>> remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>>> resize.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion, but
>>>>>>>>>>>> I have tinkered with a "chunked" array list which seems to offer better time
>>>>>>>>>>>> performance in general at the cost of greater (worst case) memory
>>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>>
>>>>>>>>>>>> I have prototyped the data structure this evening and
>>>>>>>>>>>> benchmarked additions at the end of the list, for which the performance is
>>>>>>>>>>>> pretty impressive.
>>>>>>>>>>>>
>>>>>>>>>>>> Some random statistics for addition only on the client JVM (I
>>>>>>>>>>>> have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per round
>>>>>>>>>>>> per statistic per list, and the second round results were used.
>>>>>>>>>>>>
>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>>
>>>>>>>>>>>> and server JVM:
>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>>
>>>>>>>>>>>> Interestingly insertion at the beginning of the list appears to
>>>>>>>>>>>> be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>>
>>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>>
>>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>>> larger lists:
>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining in...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The idea is to get a replacement for arraylist that performs
>>>>>>>>>>>>> like arraydeque on remove(0). As a side effect, we should be able to get
>>>>>>>>>>>>> better performance on other operations by requiring fewer elements to be
>>>>>>>>>>>>> moved.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that add
>>>>>>>>>>>>>> does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd be interested if you have any comments in the context of
>>>>>>>>>>>>>>> this discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>>     102          90         240         191         138
>>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>>>   138          94         230         194         118
>>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>>>   538          91         324         382         119
>>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>>    4425          94        1296        2330         124
>>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>> ArrayList ----------------------------------------------------
>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>>      69       32913         162         130         105
>>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>>>   104       21944         169         134         135
>>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>>>   506        9602         364         333         526
>>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>>    4414        9947        2312        2280        4437
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's interesting to note that the old circular list trick
>>>>>>>>>>>>>>>>> will not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik
>>>>>>>>>>>>>>>>> D. Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs
>>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the
>>>>>>>>>>>>>>>>>> specified operation with
>>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so
>>>>>>>>>>>>>>>>>> 1.00 indicates
>>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and
>>>>>>>>>>>>>>>>>> then timed the process
>>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally
>>>>>>>>>>>>>>>>>> I used the iterator
>>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away
>>>>>>>>>>>>>>>>>> the timing for the
>>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs in
>>>>>>>>>>>>>>>>>> java.util, I would
>>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view
>>>>>>>>>>>>>>>>>> to eliminate the
>>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion
>>>>>>>>>>>>>>>>>> of achieving this by
>>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees
>>>>>>>>>>>>>>>>>> that no backing
>>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead
>>>>>>>>>>>>>>>>>> than ArrayList, and
>>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize
>>>>>>>>>>>>>>>>>> than ArrayList.  Of
>>>>>>>>>>>>>>>>>> > course, this data structure does not do everything
>>>>>>>>>>>>>>>>>> better than ArrayList; in
>>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and the
>>>>>>>>>>>>>>>>>> additional memory
>>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly,
>>>>>>>>>>>>>>>>>> due to the multiple
>>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That
>>>>>>>>>>>>>>>>>> being said, I think that
>>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional
>>>>>>>>>>>>>>>>>> cost of
>>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the fact
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable
>>>>>>>>>>>>>>>>>> operation on ArrayList due
>>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the surprising
>>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>>> >> > The data structure is available at the second link
>>>>>>>>>>>>>>>>>> that I originally
>>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front
>>>>>>>>>>>>>>>>>> as yet as it was
>>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by multiple
>>>>>>>>>>>>>>>>>> small arrays
>>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support
>>>>>>>>>>>>>>>>>> efficient add-at-front or
>>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a
>>>>>>>>>>>>>>>>>> much more important
>>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so
>>>>>>>>>>>>>>>>>> I'll be updating the
>>>>>>>>>>>>>>>>>> >> > document at the provided link as I find improvements.
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising that
>>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the
>>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and
>>>>>>>>>>>>>>>>>> the offset into the
>>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I mentioned
>>>>>>>>>>>>>>>>>> that this will
>>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple
>>>>>>>>>>>>>>>>>> bit shift and a bit
>>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be doing
>>>>>>>>>>>>>>>>>> this each time an
>>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2
>>>>>>>>>>>>>>>>>> - 1);
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p
>>>>>>>>>>>>>>>>>> + b) + " " + e);
>>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for
>>>>>>>>>>>>>>>>>> even numbered
>>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional
>>>>>>>>>>>>>>>>>> term added (the
>>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of (1)
>>>>>>>>>>>>>>>>>> below.  I must be
>>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays in
>>>>>>>>>>>>>>>>>> optimal time and
>>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with the
>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated,
>>>>>>>>>>>>>>>>>> it consists of
>>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i),
>>>>>>>>>>>>>>>>>> with i = 3:
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There are
>>>>>>>>>>>>>>>>>> only two data
>>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given (1)
>>>>>>>>>>>>>>>>>> above, unless I'm
>>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks in
>>>>>>>>>>>>>>>>>> superblocks prior
>>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better
>>>>>>>>>>>>>>>>>> in my example above,
>>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation
>>>>>>>>>>>>>>>>>> of their data
>>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin Buchholz
>>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer - to
>>>>>>>>>>>>>>>>>> insert at the
>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer
>>>>>>>>>>>>>>>>>> (modulo the size).  We
>>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet
>>>>>>>>>>>>>>>>>> that is shared by the
>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and space",
>>>>>>>>>>>>>>>>>> gives a deque so
>>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/e08d0e36/attachment.html>

From kevin.l.stern at gmail.com  Sat Apr 24 18:14:37 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Sat, 24 Apr 2010 13:14:37 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<s2u51a29d951004150600ue27e6f10m81bd831a96cb278e@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
	<i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
	<w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>
Message-ID: <r2h1704b7a21004241114wfca78ae7lb3e9571316ca5b3f@mail.gmail.com>

Hi Benedict,

Thanks, I'll definitely give this a try.  I certainly don't see any issue
with skipping the size 1 array in order to speed up general operation.

By the way, I'm currently focused a bit more on the deque - I'm not sure
that anything is going to come of this ChunkedArrayList on its own.  With
the deque, index decomposition is much faster than any of these
implementations as it lends itself to bit operations without the requirement
of computing the logarithm.

Kevin

On Sat, Apr 24, 2010 at 12:30 PM, Benedict Elliott Smith
<lists at laerad.com>wrote:

> If you want a drop in replacement with minimal fuss to test it out, try
> below (although it's a quick hack of including the
> Integer.numberOfLeadingZeros() call to avoid its now unnecessary non-zero
> test, and instead utilise this op to return the zero index). Just delete the
> existing get() method and replace it with this code. Actually this version
> seems to improve throughput further - on my laptop regular ChunkedArrayList
> is currently averaging around 790ms per run of my hacky speed test, whereas
> with the modifications below it averages around 610ms, making it almost 25%
> faster. I've included my performance benchmark as well, for reference. I'm
> sure with some thought a better one could be put together.
>
>     private static final int arrayIndex2(int index) {
>         if (index == 1)
>             return 0 ;
>         int i = index >>> 1 ;
>         int n = 1;
>         if (i >>> 16 == 0) { n += 16; i <<= 16; }
>         if (i >>> 24 == 0) { n +=  8; i <<=  8; }
>         if (i >>> 28 == 0) { n +=  4; i <<=  4; }
>         if (i >>> 30 == 0) { n +=  2; i <<=  2; }
>         n -= i >>> 31;
>      final int arraySizeShiftMinusSeed = (31 - n) >>> 1 ;
>  final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed)
> ;
>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
> final int b2 = index - b1 ;
>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>  final int av = a1 - a2 ;
> final int bv = b4 - 2 ;
>  return av + bv ;
>     }
>
>     @Override
>     public T get(int index) {
>         if ((0 > index) | (index >= _size))
>             throw new IndexOutOfBoundsException();
>         index += 1 ;
>         final Object[] array = _backingArray[arrayIndex2(index)] ;
>         return (T) array[index & (array.length - 1)] ;
>     }
>
>
> //// benchmarked by
>
> static final int count = 1 << 24 ;
> static final int mask = count - 1 ;
>  public static void main(String[] args) {
> double sum = 0 ;
> final List<Integer> list = new ChunkedArrayList<Integer>() ;
>  for (int i = 0 ; i != count ; i++)
> list.add(i) ;
> for (int r = 0 ; r != 100 ; r++) {
>  final long start = System.currentTimeMillis() ;
> int o = 0 ;
> for (int j = 0 ; j != count ; j++) {
>  list.get((j + o) & mask) ;
> o += 1 ;
> }
>  final long end = System.currentTimeMillis() ;
> sum += (end - start) ;
>                         System.out.println(String.format("run %d: %dms;
> avg=%.0fms", r, (end - start), sum / (r + 1))) ;
>  }
> }
>
>
> On 24 April 2010 09:34, Benedict Elliott Smith <lists at laerad.com> wrote:
>
>> Hi Kevin,
>>
>> If you are willing to change the pattern of allocation just slightly, I
>> have come up with an alternate algorithm (optimised for the seed = 1 case)
>> that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
>> for random(ish) calls to get() on a list of size 1 << 24. The change is to
>> simply drop your first array allocation (so that there are no arrays of size
>> 1, but that all remaining allocations follow the existing pattern), as this
>> allows simplifying the algorithm noticeably (several ops in the previous
>> algorithm were unnecessary for any but the first two arrays).
>>
>> My get(index) method is defined as:
>>
>> if ((index < 0) | (index >= _size))
>> throw new IllegalArgumentException() ;
>>  index += 2 ;
>> final Object[] array = _backingArray[arrayFor(index)] ;
>>  return (T) array[index & (array.length - 1)] ;
>>
>> and arrayFor(index) is defined as:
>>
>> private static final int arrayFor(int index) {
>>      final int arraySizeShiftMinusSeed = (31 -
>> Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
>> final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed) ;
>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>> final int b2 = index - b1 ;
>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>  final int av = a1 - a2 ;
>> final int bv = b4 - 3 ;
>> return av + bv ;
>>  }
>>
>> I have deliberately interleaved the calculations here to make sure the
>> pipeline is being used (just in case javac+hotspot are not re-ordering the
>> intermediate calculations for us)
>>
>>
>>
>>
>>
>> On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> It looks like this is because ChunkedArrayList creates only one initial
>>> array of size s, whereas my algorithm expects two . Apologies for not
>>> spotting this - the pattern of allocations is identical after this point.
>>> I'll see if it can be modified to support your pattern of allocation.
>>>
>>>
>>> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>
>>>> Hi Benedict,
>>>>
>>>> I took a look at your index decomposition routine; it was not working
>>>> for seed = 1 until I made index the query index plus one (similar to my r
>>>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>>>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>>>> changes the routine was actually slower than my version (although not by
>>>> much).  Let me know if you have a better way to bring your routine in line
>>>> with the arraylet structure.
>>>>
>>>> Kevin
>>>>
>>>>
>>>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <
>>>> lists at laerad.com> wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> Unfortunately this week has been pretty hectic, and I haven't had much
>>>>> time to much more than theorise on this topic - and this weekend the weather
>>>>> looks set to be much too nice to stay in doors! It looks like you've made
>>>>> really good progress on the ChunkedArrayDeque - I haven't fully digested it
>>>>> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
>>>>> your list implementation prior to reading it in detail) but I think I may
>>>>> have been toying with a similar growth strategy for a hash map variant since
>>>>> the copies are necessary there anyway.
>>>>>
>>>>> I have taken another look at the index algorithm and have tidied it up
>>>>> as below, and it now supports a seed size of 1; however I am not sure that
>>>>> using a value this small is advisable, given that the overhead for the first
>>>>> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
>>>>> utilise less than 20% of the allocated memory for data, the first 24 less
>>>>> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
>>>>> advisable.
>>>>>
>>>>> As an aside, it is worth noting that the indexOffset calculation below
>>>>> can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>>>> although I am not certain what effect this will have on performance, given
>>>>> that this would be another memory operation to retrieve the length from the
>>>>> array; but it would make the separation of the calculation into functions
>>>>> more straight forward.
>>>>>
>>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>
>>>>>      final int firstArrayOfThisSize =
>>>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>>>      - (1 << arraySizeShiftMinusSeed)
>>>>>      - 1 - (arraySizeShift >>> 31) ;
>>>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>>>> arraySizeShiftMinusSeed) ;
>>>>>
>>>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>>>> Integer.MAX_VALUE ;
>>>>>
>>>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>
>>>>>
>>>>>
>>>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Hi Benedict,
>>>>>>
>>>>>> Have you had a chance to get your index decomposition procedure to
>>>>>> work with seed values less than two?
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> As it happens I might have something useful still to contribute. As
>>>>>>> an exercise in saving face I revisited the problem to see if I could achieve
>>>>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>>>>> must admit I still didn't fully appreciate how the algorithm in
>>>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>>>> similar properties. What I have ended up with is almost identical except
>>>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>>>
>>>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>>>> is approximately the same.
>>>>>>>
>>>>>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>>>>>> specified (although I expect hard coding a suitable one will ultimately be
>>>>>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>>>>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>>>>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>>>>>> 16*s*(..*24)] etc. although when put in simple text like that it
>>>>>>> does appear to trivialise the change. The benefit of this, though, is two
>>>>>>> fold: 1) for small n the constant factor is reduced (both CPU and memory
>>>>>>> wise); and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>>>
>>>>>>> As an illustration, consider setting *s* to 4, and assume the
>>>>>>> backing array is size two and doubles in size with each growth; with
>>>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s*as 4 we would instead resize at i=8,i=24,i=80,i=288; the cost at each would
>>>>>>> be some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>>>>> seems likely this should also have a positive impact on cache performance
>>>>>>> for smaller lists as well.
>>>>>>>
>>>>>>> Finally, after playing with this idea in my head I am confident I can
>>>>>>> extend the core ideas of this data structure to hashing relatively easily,
>>>>>>> getting the the same worst case O(sqrt(n)) insertion cost, and O(sqrt(n))
>>>>>>> wasted memory guarantees. I notice that this case hasn't been addressed yet,
>>>>>>> although I see from Martin's recent mail that this was raised before. Unless
>>>>>>> there are better suggestions for solving the hash table problem I will have
>>>>>>> a go at it as it seems an interesting problem - that is, assuming there are
>>>>>>> no objections?
>>>>>>>
>>>>>>> I'm interested to hear your thoughts. I hope this time I've been a
>>>>>>> bit more considered in what I've put forward, and hence less of a waste of
>>>>>>> time!
>>>>>>>
>>>>>>> Code snippet for calculation of array index and item offset:
>>>>>>>
>>>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed +
>>>>>>> 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>
>>>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>
>>>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *s
>>>>>>> * + 1); seed only works for values for 2 or more at this moment, fyi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>
>>>>>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let
>>>>>>>> me know if you have any other questions or if you have any related ideas or
>>>>>>>> concerns.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance
>>>>>>>>> than help. that part of the discussion was clearly truncated by the time I
>>>>>>>>> had joined the list - I haven't been able to find the history in the
>>>>>>>>> archives either...
>>>>>>>>>
>>>>>>>>> I was just wondering about the worst case cost of add() as
>>>>>>>>> described by your javadoc; admittedly it is optimal with respect to unused
>>>>>>>>> memory, but the worst case cost of an add is still sqrt(n), with a
>>>>>>>>> relatively high constant factor. I had been thinking that once n passed a
>>>>>>>>> threshold the cost of additions in this other structure would become a
>>>>>>>>> constant factor, offering nice algorithmic complexity guarantees for large
>>>>>>>>> n; however since sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new
>>>>>>>>> array allocations would have to be unrealistically small (assuming linear
>>>>>>>>> cost for allocation) for this to be the case. It would still be nice to have
>>>>>>>>> a data structure that avoids needing to copy data with each grow, whilst
>>>>>>>>> still maintaining good memory performance.
>>>>>>>>>
>>>>>>>>> That *all* being said, I had been going by your javadoc and emails
>>>>>>>>> to ascertain the behaviour of this class, as I couldn't locate a free copy
>>>>>>>>> of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>>>> large values of n.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Hi Benedict,
>>>>>>>>>>
>>>>>>>>>> Unless I am misreading your post, this now has a very similar feel
>>>>>>>>>> to the first data structure that I posted to the list.  Martin Buchholz then
>>>>>>>>>> pointed out that we can incorporate the ideas from
>>>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>
>>>>>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>>>>>> been exhaustive in the benchmarking:
>>>>>>>>>>>
>>>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82,
>>>>>>>>>>> ExpArray=0.75
>>>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>>>
>>>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>>>
>>>>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>>>>
>>>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>>>  }
>>>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift) +
>>>>>>>>>>> maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>>>>  }
>>>>>>>>>>>
>>>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly didn't
>>>>>>>>>>> think that one through fully!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>
>>>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding:  "The idea is to simply double the new array size
>>>>>>>>>>>> each time a new array needs to be allocated"
>>>>>>>>>>>>
>>>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it
>>>>>>>>>>>> seems that the simplest solution would be to simply have two lists, with one
>>>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>>>
>>>>>>>>>>>> What happens with your structure when you add n elements and
>>>>>>>>>>>> then remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>>>> resize.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion,
>>>>>>>>>>>>> but I have tinkered with a "chunked" array list which seems to offer better
>>>>>>>>>>>>> time performance in general at the cost of greater (worst case) memory
>>>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have prototyped the data structure this evening and
>>>>>>>>>>>>> benchmarked additions at the end of the list, for which the performance is
>>>>>>>>>>>>> pretty impressive.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Some random statistics for addition only on the client JVM (I
>>>>>>>>>>>>> have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per round
>>>>>>>>>>>>> per statistic per list, and the second round results were used.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>>>
>>>>>>>>>>>>> and server JVM:
>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>>>
>>>>>>>>>>>>> Interestingly insertion at the beginning of the list appears to
>>>>>>>>>>>>> be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>>>
>>>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>>>> larger lists:
>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining
>>>>>>>>>>>>> in...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The idea is to get a replacement for arraylist that performs
>>>>>>>>>>>>>> like arraydeque on remove(0). As a side effect, we should be able to get
>>>>>>>>>>>>>> better performance on other operations by requiring fewer elements to be
>>>>>>>>>>>>>> moved.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that
>>>>>>>>>>>>>>> add does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd be interested if you have any comments in the context of
>>>>>>>>>>>>>>>> this discussion. The code is not entirely ready yet, a couple of tests fail
>>>>>>>>>>>>>>>> (6/789) because of a corner case I haven't nailed yet, but the idea is there
>>>>>>>>>>>>>>>> at least. I'd like to add array shrinking later, when the size dips below
>>>>>>>>>>>>>>>> capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the O(1)
>>>>>>>>>>>>>>>> operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>>>     102          90         240         191         138
>>>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>>>>   138          94         230         194         118
>>>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>>>>   538          91         324         382         119
>>>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>>>    4425          94        1296        2330         124
>>>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>>> ArrayList ----------------------------------------------------
>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>>>      69       32913         162         130         105
>>>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>>>>   104       21944         169         134         135
>>>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>>>>   506        9602         364         333         526
>>>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>>>    4414        9947        2312        2280        4437
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It's interesting to note that the old circular list trick
>>>>>>>>>>>>>>>>>> will not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik
>>>>>>>>>>>>>>>>>> D. Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out bugs
>>>>>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've included
>>>>>>>>>>>>>>>>>>> below (note that the
>>>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the
>>>>>>>>>>>>>>>>>>> specified operation with
>>>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList, so
>>>>>>>>>>>>>>>>>>> 1.00 indicates
>>>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent rough
>>>>>>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and
>>>>>>>>>>>>>>>>>>> then timed the process
>>>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and finally
>>>>>>>>>>>>>>>>>>> I used the iterator
>>>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element (of
>>>>>>>>>>>>>>>>>>> course, I performed
>>>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away
>>>>>>>>>>>>>>>>>>> the timing for the
>>>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs
>>>>>>>>>>>>>>>>>>> in java.util, I would
>>>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of view
>>>>>>>>>>>>>>>>>>> to eliminate the
>>>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your suggestion
>>>>>>>>>>>>>>>>>>> of achieving this by
>>>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only guarantees
>>>>>>>>>>>>>>>>>>> that no backing
>>>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead
>>>>>>>>>>>>>>>>>>> than ArrayList, and
>>>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize
>>>>>>>>>>>>>>>>>>> than ArrayList.  Of
>>>>>>>>>>>>>>>>>>> > course, this data structure does not do everything
>>>>>>>>>>>>>>>>>>> better than ArrayList; in
>>>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and
>>>>>>>>>>>>>>>>>>> the additional memory
>>>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly,
>>>>>>>>>>>>>>>>>>> due to the multiple
>>>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That
>>>>>>>>>>>>>>>>>>> being said, I think that
>>>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional
>>>>>>>>>>>>>>>>>>> cost of
>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the
>>>>>>>>>>>>>>>>>>> fact that
>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable
>>>>>>>>>>>>>>>>>>> operation on ArrayList due
>>>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the
>>>>>>>>>>>>>>>>>>> surprising
>>>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in java.util?
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>>>> >> > The data structure is available at the second link
>>>>>>>>>>>>>>>>>>> that I originally
>>>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front
>>>>>>>>>>>>>>>>>>> as yet as it was
>>>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by
>>>>>>>>>>>>>>>>>>> multiple small arrays
>>>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support
>>>>>>>>>>>>>>>>>>> efficient add-at-front or
>>>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still a
>>>>>>>>>>>>>>>>>>> much more important
>>>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so
>>>>>>>>>>>>>>>>>>> I'll be updating the
>>>>>>>>>>>>>>>>>>> >> > document at the provided link as I find
>>>>>>>>>>>>>>>>>>> improvements.
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the
>>>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array and
>>>>>>>>>>>>>>>>>>> the offset into the
>>>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I
>>>>>>>>>>>>>>>>>>> mentioned that this will
>>>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple
>>>>>>>>>>>>>>>>>>> bit shift and a bit
>>>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be
>>>>>>>>>>>>>>>>>>> doing this each time an
>>>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) & (powFloorKO2
>>>>>>>>>>>>>>>>>>> - 1);
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " + (p
>>>>>>>>>>>>>>>>>>> + b) + " " + e);
>>>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for
>>>>>>>>>>>>>>>>>>> even numbered
>>>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional
>>>>>>>>>>>>>>>>>>> term added (the
>>>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of
>>>>>>>>>>>>>>>>>>> (1) below.  I must be
>>>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays
>>>>>>>>>>>>>>>>>>> in optimal time and
>>>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with
>>>>>>>>>>>>>>>>>>> the following
>>>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated,
>>>>>>>>>>>>>>>>>>> it consists of
>>>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed this
>>>>>>>>>>>>>>>>>>> implies the
>>>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i),
>>>>>>>>>>>>>>>>>>> with i = 3:
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There
>>>>>>>>>>>>>>>>>>> are only two data
>>>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given
>>>>>>>>>>>>>>>>>>> (1) above, unless I'm
>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks
>>>>>>>>>>>>>>>>>>> in superblocks prior
>>>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better
>>>>>>>>>>>>>>>>>>> in my example above,
>>>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation
>>>>>>>>>>>>>>>>>>> of their data
>>>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is their
>>>>>>>>>>>>>>>>>>> mistake rather than
>>>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin
>>>>>>>>>>>>>>>>>>> Buchholz
>>>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer -
>>>>>>>>>>>>>>>>>>> to insert at the
>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer
>>>>>>>>>>>>>>>>>>> (modulo the size).  We
>>>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each other.
>>>>>>>>>>>>>>>>>>> Could you explain
>>>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet
>>>>>>>>>>>>>>>>>>> that is shared by the
>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single array.
>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and
>>>>>>>>>>>>>>>>>>> space", gives a deque so
>>>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the Deque
>>>>>>>>>>>>>>>>>>> operations -
>>>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/f654a449/attachment.html>

From martinrb at google.com  Sat Apr 24 18:21:20 2010
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 24 Apr 2010 11:21:20 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD00250.3020206@oracle.com>
References: <4BD00250.3020206@oracle.com>
Message-ID: <o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>

Providing script support is obvious and non-controversial,
because other regex programming environments provide it.
Check that the behavior and syntax of the extension is
consistent with e.g. ICU, python, and especially perl
(5.12 just released!)

http://perldoc.perl.org/perlunicode.html

I would add some documentation to the three special script values;
their meaning is not obvious.

For implementation, the character matching problem is in general
equivalent to the problem of compiling a switch statement, which is
known to be non-trivial.  Guava contains a CharMatcher class that
tries to solve related problems.

http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/CharMatcher.html

I'm thinking scripts and blocks should know about which ranges they contain.
In particular, \p{BlockName} should not need binary search at
regex compile time or runtime.

---
There is one place you need to change
key word => keyword
---
InMongolian => {@code InMongolian}
---
I notice current Unicode block support in JDK is not updated to the
latest standard.
E.g. Samaritan is missing.

Martin

On Thu, Apr 22, 2010 at 01:01, Xueming Shen <xueming.shen at oracle.com> wrote:
> Hi,
>
> Here is the webrev of the proposal to add Unicode script support in regex
> and j.l.Character.
>
> http://cr.openjdk.java.net/~sherman/script/webrev
>
> and the corresponding blenderrev
>
> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
>
> Please comment on the APIs before I submit the CCC, especially
>
> (1) to use enum for the j.l.Character.UnicodeScript (compared to the
> traditional j.l.c.Subset)
> (2) the piggyback method j.l.c.getName() :-)
> (3) the syntax for script constructs. In addition to the "normal"
> ? ?\p{InScriptName} and \P{InScriptName} for the script support
> ? ?I'm also adding
> ? \p{script=ScriptName} \P{script=ScriptName} ?for the new script support
> ? \p{block=BlockName} \P{block=BlockName} ?for the "existing" block support
> ? \p{general_category=CategoryName} \P{general_category=CategoryName} for
> the "existing" gc
> ? Perl recently also started to accept this ?\p{propName=propValue} Unicode
> style.
> ? It opens the door for future "expanding", for example \p{name=XYZ} :-)
> (4)and of course, the wording.
>
> Thanks,
> Sherman
>
>
>


From lists at laerad.com  Sat Apr 24 18:27:56 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sat, 24 Apr 2010 19:27:56 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <r2h1704b7a21004241114wfca78ae7lb3e9571316ca5b3f@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<n2t1704b7a21004151618y453956egacfbbd7d3997872f@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
	<i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
	<w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>
	<r2h1704b7a21004241114wfca78ae7lb3e9571316ca5b3f@mail.gmail.com>
Message-ID: <z2x51a29d951004241127l120c8d6ek986f2ce93a7550d7@mail.gmail.com>

Yes, I had spotted that benefit - but (please correct me if I am misreading
this as it is quite possible) in order to maintain your array allocation
invariant, array copies are needed (rather than straight allocations) - and
so the cost of growth is likely to be noticeably larger. That said, if
random access is considerably quicker this might be a sensible trade off,
but perhaps there is a place for both data structures.


On 24 April 2010 19:14, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> Thanks, I'll definitely give this a try.  I certainly don't see any issue
> with skipping the size 1 array in order to speed up general operation.
>
> By the way, I'm currently focused a bit more on the deque - I'm not sure
> that anything is going to come of this ChunkedArrayList on its own.  With
> the deque, index decomposition is much faster than any of these
> implementations as it lends itself to bit operations without the requirement
> of computing the logarithm.
>
> Kevin
>
>
> On Sat, Apr 24, 2010 at 12:30 PM, Benedict Elliott Smith <lists at laerad.com
> > wrote:
>
>> If you want a drop in replacement with minimal fuss to test it out, try
>> below (although it's a quick hack of including the
>> Integer.numberOfLeadingZeros() call to avoid its now unnecessary non-zero
>> test, and instead utilise this op to return the zero index). Just delete the
>> existing get() method and replace it with this code. Actually this version
>> seems to improve throughput further - on my laptop regular ChunkedArrayList
>> is currently averaging around 790ms per run of my hacky speed test, whereas
>> with the modifications below it averages around 610ms, making it almost 25%
>> faster. I've included my performance benchmark as well, for reference. I'm
>> sure with some thought a better one could be put together.
>>
>>     private static final int arrayIndex2(int index) {
>>         if (index == 1)
>>             return 0 ;
>>         int i = index >>> 1 ;
>>         int n = 1;
>>         if (i >>> 16 == 0) { n += 16; i <<= 16; }
>>         if (i >>> 24 == 0) { n +=  8; i <<=  8; }
>>         if (i >>> 28 == 0) { n +=  4; i <<=  4; }
>>         if (i >>> 30 == 0) { n +=  2; i <<=  2; }
>>         n -= i >>> 31;
>>      final int arraySizeShiftMinusSeed = (31 - n) >>> 1 ;
>>  final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed)
>> ;
>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>> final int b2 = index - b1 ;
>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>  final int av = a1 - a2 ;
>> final int bv = b4 - 2 ;
>>  return av + bv ;
>>     }
>>
>>     @Override
>>     public T get(int index) {
>>         if ((0 > index) | (index >= _size))
>>             throw new IndexOutOfBoundsException();
>>         index += 1 ;
>>         final Object[] array = _backingArray[arrayIndex2(index)] ;
>>         return (T) array[index & (array.length - 1)] ;
>>     }
>>
>>
>> //// benchmarked by
>>
>> static final int count = 1 << 24 ;
>> static final int mask = count - 1 ;
>>  public static void main(String[] args) {
>> double sum = 0 ;
>> final List<Integer> list = new ChunkedArrayList<Integer>() ;
>>  for (int i = 0 ; i != count ; i++)
>> list.add(i) ;
>> for (int r = 0 ; r != 100 ; r++) {
>>  final long start = System.currentTimeMillis() ;
>> int o = 0 ;
>> for (int j = 0 ; j != count ; j++) {
>>  list.get((j + o) & mask) ;
>> o += 1 ;
>> }
>>  final long end = System.currentTimeMillis() ;
>> sum += (end - start) ;
>>                         System.out.println(String.format("run %d: %dms;
>> avg=%.0fms", r, (end - start), sum / (r + 1))) ;
>>  }
>> }
>>
>>
>> On 24 April 2010 09:34, Benedict Elliott Smith <lists at laerad.com> wrote:
>>
>>> Hi Kevin,
>>>
>>> If you are willing to change the pattern of allocation just slightly, I
>>> have come up with an alternate algorithm (optimised for the seed = 1 case)
>>> that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
>>> for random(ish) calls to get() on a list of size 1 << 24. The change is to
>>> simply drop your first array allocation (so that there are no arrays of size
>>> 1, but that all remaining allocations follow the existing pattern), as this
>>> allows simplifying the algorithm noticeably (several ops in the previous
>>> algorithm were unnecessary for any but the first two arrays).
>>>
>>> My get(index) method is defined as:
>>>
>>> if ((index < 0) | (index >= _size))
>>> throw new IllegalArgumentException() ;
>>>  index += 2 ;
>>> final Object[] array = _backingArray[arrayFor(index)] ;
>>>  return (T) array[index & (array.length - 1)] ;
>>>
>>> and arrayFor(index) is defined as:
>>>
>>> private static final int arrayFor(int index) {
>>>      final int arraySizeShiftMinusSeed = (31 -
>>> Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
>>> final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed)
>>> ;
>>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>>> final int b2 = index - b1 ;
>>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>>  final int av = a1 - a2 ;
>>> final int bv = b4 - 3 ;
>>> return av + bv ;
>>>  }
>>>
>>> I have deliberately interleaved the calculations here to make sure the
>>> pipeline is being used (just in case javac+hotspot are not re-ordering the
>>> intermediate calculations for us)
>>>
>>>
>>>
>>>
>>>
>>> On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> It looks like this is because ChunkedArrayList creates only one initial
>>>> array of size s, whereas my algorithm expects two . Apologies for not
>>>> spotting this - the pattern of allocations is identical after this point.
>>>> I'll see if it can be modified to support your pattern of allocation.
>>>>
>>>>
>>>> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>>>
>>>>> Hi Benedict,
>>>>>
>>>>> I took a look at your index decomposition routine; it was not working
>>>>> for seed = 1 until I made index the query index plus one (similar to my r
>>>>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>>>>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>>>>> changes the routine was actually slower than my version (although not by
>>>>> much).  Let me know if you have a better way to bring your routine in line
>>>>> with the arraylet structure.
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <
>>>>> lists at laerad.com> wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> Unfortunately this week has been pretty hectic, and I haven't had much
>>>>>> time to much more than theorise on this topic - and this weekend the weather
>>>>>> looks set to be much too nice to stay in doors! It looks like you've made
>>>>>> really good progress on the ChunkedArrayDeque - I haven't fully digested it
>>>>>> yet, but it looks pretty impressive. I'm not sure (since I misunderstood
>>>>>> your list implementation prior to reading it in detail) but I think I may
>>>>>> have been toying with a similar growth strategy for a hash map variant since
>>>>>> the copies are necessary there anyway.
>>>>>>
>>>>>> I have taken another look at the index algorithm and have tidied it up
>>>>>> as below, and it now supports a seed size of 1; however I am not sure that
>>>>>> using a value this small is advisable, given that the overhead for the first
>>>>>> array is at least 60 bytes; with a seed size of 1 the first 8 indexes would
>>>>>> utilise less than 20% of the allocated memory for data, the first 24 less
>>>>>> than 25%. I would have thought a seed of at least 2 and perhaps 3 would be
>>>>>> advisable.
>>>>>>
>>>>>> As an aside, it is worth noting that the indexOffset calculation below
>>>>>> can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>>>>> although I am not certain what effect this will have on performance, given
>>>>>> that this would be another memory operation to retrieve the length from the
>>>>>> array; but it would make the separation of the calculation into functions
>>>>>> more straight forward.
>>>>>>
>>>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>
>>>>>>      final int firstArrayOfThisSize =
>>>>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>>>>      - (1 << arraySizeShiftMinusSeed)
>>>>>>      - 1 - (arraySizeShift >>> 31) ;
>>>>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>>>>> arraySizeShiftMinusSeed) ;
>>>>>>
>>>>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>>>>> Integer.MAX_VALUE ;
>>>>>>
>>>>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>
>>>>>>> Hi Benedict,
>>>>>>>
>>>>>>> Have you had a chance to get your index decomposition procedure to
>>>>>>> work with seed values less than two?
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>>>>> lists at laerad.com> wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> As it happens I might have something useful still to contribute. As
>>>>>>>> an exercise in saving face I revisited the problem to see if I could achieve
>>>>>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>>>>>> must admit I still didn't fully appreciate how the algorithm in
>>>>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>>>>> similar properties. What I have ended up with is almost identical except
>>>>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>>>>
>>>>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>>>>> is approximately the same.
>>>>>>>>
>>>>>>>> Secondly, I have modified the algorithm so that a "seed" size can be
>>>>>>>> specified (although I expect hard coding a suitable one will ultimately be
>>>>>>>> best). Whereas ChunkedArrayList currently requires that the pattern of array
>>>>>>>> allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we can now
>>>>>>>> support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6), 8*s*(..*12),
>>>>>>>> 16*s*(..*24)] etc. although when put in simple text like that it
>>>>>>>> does appear to trivialise the change. The benefit of this, though, is two
>>>>>>>> fold: 1) for small n the constant factor is reduced (both CPU and memory
>>>>>>>> wise); and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>>>>
>>>>>>>> As an illustration, consider setting *s* to 4, and assume the
>>>>>>>> backing array is size two and doubles in size with each growth; with
>>>>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s*as 4 we would instead resize at i=8,i=24,i=80,i=288; the cost at each would
>>>>>>>> be some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>>>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>>>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>>>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>>>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>>>>>> seems likely this should also have a positive impact on cache performance
>>>>>>>> for smaller lists as well.
>>>>>>>>
>>>>>>>> Finally, after playing with this idea in my head I am confident I
>>>>>>>> can extend the core ideas of this data structure to hashing relatively
>>>>>>>> easily, getting the the same worst case O(sqrt(n)) insertion cost, and
>>>>>>>> O(sqrt(n)) wasted memory guarantees. I notice that this case hasn't been
>>>>>>>> addressed yet, although I see from Martin's recent mail that this was raised
>>>>>>>> before. Unless there are better suggestions for solving the hash table
>>>>>>>> problem I will have a go at it as it seems an interesting problem - that is,
>>>>>>>> assuming there are no objections?
>>>>>>>>
>>>>>>>> I'm interested to hear your thoughts. I hope this time I've been a
>>>>>>>> bit more considered in what I've put forward, and hence less of a waste of
>>>>>>>> time!
>>>>>>>>
>>>>>>>> Code snippet for calculation of array index and item offset:
>>>>>>>>
>>>>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed
>>>>>>>> + 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>>
>>>>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>>
>>>>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to *
>>>>>>>> s* + 1); seed only works for values for 2 or more at this moment,
>>>>>>>> fyi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Oh no worries Benedict, thanks for your interest in the topic.  Let
>>>>>>>>> me know if you have any other questions or if you have any related ideas or
>>>>>>>>> concerns.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance
>>>>>>>>>> than help. that part of the discussion was clearly truncated by the time I
>>>>>>>>>> had joined the list - I haven't been able to find the history in the
>>>>>>>>>> archives either...
>>>>>>>>>>
>>>>>>>>>> I was just wondering about the worst case cost of add() as
>>>>>>>>>> described by your javadoc; admittedly it is optimal with respect to unused
>>>>>>>>>> memory, but the worst case cost of an add is still sqrt(n), with a
>>>>>>>>>> relatively high constant factor. I had been thinking that once n passed a
>>>>>>>>>> threshold the cost of additions in this other structure would become a
>>>>>>>>>> constant factor, offering nice algorithmic complexity guarantees for large
>>>>>>>>>> n; however since sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new
>>>>>>>>>> array allocations would have to be unrealistically small (assuming linear
>>>>>>>>>> cost for allocation) for this to be the case. It would still be nice to have
>>>>>>>>>> a data structure that avoids needing to copy data with each grow, whilst
>>>>>>>>>> still maintaining good memory performance.
>>>>>>>>>>
>>>>>>>>>> That *all* being said, I had been going by your javadoc and
>>>>>>>>>> emails to ascertain the behaviour of this class, as I couldn't locate a free
>>>>>>>>>> copy of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>>>>> large values of n.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>
>>>>>>>>>>> Unless I am misreading your post, this now has a very similar
>>>>>>>>>>> feel to the first data structure that I posted to the list.  Martin Buchholz
>>>>>>>>>>> then pointed out that we can incorporate the ideas from
>>>>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Kevin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, as I was going to bed last night I realised I had not fully
>>>>>>>>>>>> addressed the problem that was originally being visited; only reduced the
>>>>>>>>>>>> constant factor for addition to the end of the list. A trivial modification
>>>>>>>>>>>> fixes that, however; same scheme but up to some maximum arraylet size (of a
>>>>>>>>>>>> power of 2), after which the array is increased in size linearly.
>>>>>>>>>>>> Performance doesn't seem to have been affected appreciably, although not
>>>>>>>>>>>> been exhaustive in the benchmarking:
>>>>>>>>>>>>
>>>>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22,
>>>>>>>>>>>> ExpArray=1.03
>>>>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82,
>>>>>>>>>>>> ExpArray=0.75
>>>>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>>>>
>>>>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>>>>
>>>>>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>>>>>
>>>>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>>>>  }
>>>>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift)
>>>>>>>>>>>> + maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>>>>>  }
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly
>>>>>>>>>>>> didn't think that one through fully!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <kevin.l.stern at gmail.com
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding:  "The idea is to simply double the new array size
>>>>>>>>>>>>> each time a new array needs to be allocated"
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it
>>>>>>>>>>>>> seems that the simplest solution would be to simply have two lists, with one
>>>>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>>>>
>>>>>>>>>>>>> What happens with your structure when you add n elements and
>>>>>>>>>>>>> then remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>>>>> resize.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion,
>>>>>>>>>>>>>> but I have tinkered with a "chunked" array list which seems to offer better
>>>>>>>>>>>>>> time performance in general at the cost of greater (worst case) memory
>>>>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have prototyped the data structure this evening and
>>>>>>>>>>>>>> benchmarked additions at the end of the list, for which the performance is
>>>>>>>>>>>>>> pretty impressive.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Some random statistics for addition only on the client JVM (I
>>>>>>>>>>>>>> have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per
>>>>>>>>>>>>>> round per statistic per list, and the second round results were used.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and server JVM:
>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Interestingly insertion at the beginning of the list appears
>>>>>>>>>>>>>> to be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>>>>> larger lists:
>>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05, ExpArray=0.86
>>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that the
>>>>>>>>>>>>>> simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining
>>>>>>>>>>>>>> in...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The idea is to get a replacement for arraylist that performs
>>>>>>>>>>>>>>> like arraydeque on remove(0). As a side effect, we should be able to get
>>>>>>>>>>>>>>> better performance on other operations by requiring fewer elements to be
>>>>>>>>>>>>>>> moved.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that
>>>>>>>>>>>>>>>> add does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from Java's
>>>>>>>>>>>>>>>> ArrayDeque?  I took only a cursory look at it, so please understand if I
>>>>>>>>>>>>>>>> have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd be interested if you have any comments in the context
>>>>>>>>>>>>>>>>> of this discussion. The code is not entirely ready yet, a couple of tests
>>>>>>>>>>>>>>>>> fail (6/789) because of a corner case I haven't nailed yet, but the idea is
>>>>>>>>>>>>>>>>> there at least. I'd like to add array shrinking later, when the size dips
>>>>>>>>>>>>>>>>> below capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the
>>>>>>>>>>>>>>>>> O(1) operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>>>>       102          90         240         191         138
>>>>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>>>>>     138          94         230         194         118
>>>>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>>>>>     538          91         324         382         119
>>>>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>>>>      4425          94        1296        2330         124
>>>>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>>>> ArrayList ----------------------------------------------------
>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>>>>        69       32913         162         130         105
>>>>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>>>>>     104       21944         169         134         135
>>>>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>>>>>     506        9602         364         333         526
>>>>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>>>>      4414        9947        2312        2280        4437
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's interesting to note that the old circular list trick
>>>>>>>>>>>>>>>>>>> will not suffice to turn this data structure into a deque since we might be
>>>>>>>>>>>>>>>>>>> copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and Erik
>>>>>>>>>>>>>>>>>>> D. Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and Space},
>>>>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out
>>>>>>>>>>>>>>>>>>>> bugs in the
>>>>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've
>>>>>>>>>>>>>>>>>>>> included below (note that the
>>>>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the
>>>>>>>>>>>>>>>>>>>> specified operation with
>>>>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList,
>>>>>>>>>>>>>>>>>>>> so 1.00 indicates
>>>>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent
>>>>>>>>>>>>>>>>>>>> rough performance
>>>>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and
>>>>>>>>>>>>>>>>>>>> then timed the process
>>>>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and
>>>>>>>>>>>>>>>>>>>> finally I used the iterator
>>>>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element
>>>>>>>>>>>>>>>>>>>> (of course, I performed
>>>>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away
>>>>>>>>>>>>>>>>>>>> the timing for the
>>>>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs
>>>>>>>>>>>>>>>>>>>> in java.util, I would
>>>>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of
>>>>>>>>>>>>>>>>>>>> view to eliminate the
>>>>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your
>>>>>>>>>>>>>>>>>>>> suggestion of achieving this by
>>>>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only
>>>>>>>>>>>>>>>>>>>> guarantees that no backing
>>>>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it also
>>>>>>>>>>>>>>>>>>>> provides dynamic
>>>>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead
>>>>>>>>>>>>>>>>>>>> than ArrayList, and
>>>>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a resize
>>>>>>>>>>>>>>>>>>>> than ArrayList.  Of
>>>>>>>>>>>>>>>>>>>> > course, this data structure does not do everything
>>>>>>>>>>>>>>>>>>>> better than ArrayList; in
>>>>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and
>>>>>>>>>>>>>>>>>>>> the additional memory
>>>>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more costly,
>>>>>>>>>>>>>>>>>>>> due to the multiple
>>>>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That
>>>>>>>>>>>>>>>>>>>> being said, I think that
>>>>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional
>>>>>>>>>>>>>>>>>>>> cost of
>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the
>>>>>>>>>>>>>>>>>>>> fact that
>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable
>>>>>>>>>>>>>>>>>>>> operation on ArrayList due
>>>>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the
>>>>>>>>>>>>>>>>>>>> surprising
>>>>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in
>>>>>>>>>>>>>>>>>>>> java.util?
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>>>>> >> > The data structure is available at the second link
>>>>>>>>>>>>>>>>>>>> that I originally
>>>>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the front
>>>>>>>>>>>>>>>>>>>> as yet as it was
>>>>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by
>>>>>>>>>>>>>>>>>>>> multiple small arrays
>>>>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support
>>>>>>>>>>>>>>>>>>>> efficient add-at-front or
>>>>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still
>>>>>>>>>>>>>>>>>>>> a much more important
>>>>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so
>>>>>>>>>>>>>>>>>>>> I'll be updating the
>>>>>>>>>>>>>>>>>>>> >> > document at the provided link as I find
>>>>>>>>>>>>>>>>>>>> improvements.
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the
>>>>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array
>>>>>>>>>>>>>>>>>>>> and the offset into the
>>>>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I
>>>>>>>>>>>>>>>>>>>> mentioned that this will
>>>>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a simple
>>>>>>>>>>>>>>>>>>>> bit shift and a bit
>>>>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be
>>>>>>>>>>>>>>>>>>>> doing this each time an
>>>>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) &
>>>>>>>>>>>>>>>>>>>> (powFloorKO2 - 1);
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " +
>>>>>>>>>>>>>>>>>>>> (p + b) + " " + e);
>>>>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works for
>>>>>>>>>>>>>>>>>>>> even numbered
>>>>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an additional
>>>>>>>>>>>>>>>>>>>> term added (the
>>>>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that my
>>>>>>>>>>>>>>>>>>>> example structure
>>>>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of
>>>>>>>>>>>>>>>>>>>> (1) below.  I must be
>>>>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays
>>>>>>>>>>>>>>>>>>>> in optimal time and
>>>>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with
>>>>>>>>>>>>>>>>>>>> the following
>>>>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully allocated,
>>>>>>>>>>>>>>>>>>>> it consists of
>>>>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed
>>>>>>>>>>>>>>>>>>>> this implies the
>>>>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i),
>>>>>>>>>>>>>>>>>>>> with i = 3:
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There
>>>>>>>>>>>>>>>>>>>> are only two data
>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given
>>>>>>>>>>>>>>>>>>>> (1) above, unless I'm
>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks
>>>>>>>>>>>>>>>>>>>> in superblocks prior
>>>>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much better
>>>>>>>>>>>>>>>>>>>> in my example above,
>>>>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my interpretation
>>>>>>>>>>>>>>>>>>>> of their data
>>>>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is
>>>>>>>>>>>>>>>>>>>> their mistake rather than
>>>>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin
>>>>>>>>>>>>>>>>>>>> Buchholz
>>>>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L. Stern
>>>>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer -
>>>>>>>>>>>>>>>>>>>> to insert at the
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer
>>>>>>>>>>>>>>>>>>>> (modulo the size).  We
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each
>>>>>>>>>>>>>>>>>>>> other.  Could you explain
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet
>>>>>>>>>>>>>>>>>>>> that is shared by the
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single
>>>>>>>>>>>>>>>>>>>> array.
>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow, the
>>>>>>>>>>>>>>>>>>>> paper that you
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and
>>>>>>>>>>>>>>>>>>>> space", gives a deque so
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the
>>>>>>>>>>>>>>>>>>>> Deque operations -
>>>>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/1c1e8945/attachment.html>

From kevin.l.stern at gmail.com  Sat Apr 24 18:41:06 2010
From: kevin.l.stern at gmail.com (Kevin L. Stern)
Date: Sat, 24 Apr 2010 13:41:06 -0500
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <z2x51a29d951004241127l120c8d6ek986f2ce93a7550d7@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<z2v51a29d951004170948l276a6d04x4975c0701efe52d0@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
	<i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
	<w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>
	<r2h1704b7a21004241114wfca78ae7lb3e9571316ca5b3f@mail.gmail.com>
	<z2x51a29d951004241127l120c8d6ek986f2ce93a7550d7@mail.gmail.com>
Message-ID: <x2v1704b7a21004241141u9836a344i2b9d59c7fb574159@mail.gmail.com>

Hi Benedict,

You are absolutely right - the constant on the cost of growth is going to be
higher with the new data structure.  That being said, the performance
numbers that I'm getting with it are actually quite impressive when compared
to ArrayList:

Note that all numbers are with a client VM and are generated using the same
routine that I used for the ChunkedArrayList performance numbers.

1000000 elements:

Add to ChunkedArrayDeque over ArrayList: 0.89
Indexed access ChunkedArrayDeque over ArrayList: 0.81
Iterator ChunkedArrayDeque over ArrayList: 0.51

100000 elements:

Add to ChunkedArrayDeque over ArrayList: 1.01
Indexed access ChunkedArrayDeque over ArrayList: 0.79
Iterator ChunkedArrayDeque over ArrayList: 0.50

10000 elements:

Add to ChunkedArrayDeque over ArrayList: 1.15
Indexed access ChunkedArrayDeque over ArrayList: 1.15
Iterator ChunkedArrayDeque over ArrayList: 0.51

1000 elements:

Add to ChunkedArrayDeque over ArrayList: 1.35
Indexed access ChunkedArrayDeque over ArrayList: 1.12
Iterator ChunkedArrayDeque over ArrayList: 0.54

Kevin

On Sat, Apr 24, 2010 at 1:27 PM, Benedict Elliott Smith <lists at laerad.com>wrote:

> Yes, I had spotted that benefit - but (please correct me if I am misreading
> this as it is quite possible) in order to maintain your array allocation
> invariant, array copies are needed (rather than straight allocations) - and
> so the cost of growth is likely to be noticeably larger. That said, if
> random access is considerably quicker this might be a sensible trade off,
> but perhaps there is a place for both data structures.
>
>
>
> On 24 April 2010 19:14, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>
>> Hi Benedict,
>>
>> Thanks, I'll definitely give this a try.  I certainly don't see any issue
>> with skipping the size 1 array in order to speed up general operation.
>>
>> By the way, I'm currently focused a bit more on the deque - I'm not sure
>> that anything is going to come of this ChunkedArrayList on its own.  With
>> the deque, index decomposition is much faster than any of these
>> implementations as it lends itself to bit operations without the requirement
>> of computing the logarithm.
>>
>> Kevin
>>
>>
>> On Sat, Apr 24, 2010 at 12:30 PM, Benedict Elliott Smith <
>> lists at laerad.com> wrote:
>>
>>> If you want a drop in replacement with minimal fuss to test it out, try
>>> below (although it's a quick hack of including the
>>> Integer.numberOfLeadingZeros() call to avoid its now unnecessary non-zero
>>> test, and instead utilise this op to return the zero index). Just delete the
>>> existing get() method and replace it with this code. Actually this version
>>> seems to improve throughput further - on my laptop regular ChunkedArrayList
>>> is currently averaging around 790ms per run of my hacky speed test, whereas
>>> with the modifications below it averages around 610ms, making it almost 25%
>>> faster. I've included my performance benchmark as well, for reference. I'm
>>> sure with some thought a better one could be put together.
>>>
>>>     private static final int arrayIndex2(int index) {
>>>         if (index == 1)
>>>             return 0 ;
>>>         int i = index >>> 1 ;
>>>         int n = 1;
>>>         if (i >>> 16 == 0) { n += 16; i <<= 16; }
>>>         if (i >>> 24 == 0) { n +=  8; i <<=  8; }
>>>         if (i >>> 28 == 0) { n +=  4; i <<=  4; }
>>>         if (i >>> 30 == 0) { n +=  2; i <<=  2; }
>>>         n -= i >>> 31;
>>>      final int arraySizeShiftMinusSeed = (31 - n) >>> 1 ;
>>>  final int b1 = (2 << arraySizeShiftMinusSeed <<
>>> arraySizeShiftMinusSeed) ;
>>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>>> final int b2 = index - b1 ;
>>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>>  final int av = a1 - a2 ;
>>> final int bv = b4 - 2 ;
>>>  return av + bv ;
>>>     }
>>>
>>>     @Override
>>>     public T get(int index) {
>>>         if ((0 > index) | (index >= _size))
>>>             throw new IndexOutOfBoundsException();
>>>         index += 1 ;
>>>         final Object[] array = _backingArray[arrayIndex2(index)] ;
>>>         return (T) array[index & (array.length - 1)] ;
>>>     }
>>>
>>>
>>> //// benchmarked by
>>>
>>> static final int count = 1 << 24 ;
>>> static final int mask = count - 1 ;
>>>  public static void main(String[] args) {
>>> double sum = 0 ;
>>> final List<Integer> list = new ChunkedArrayList<Integer>() ;
>>>  for (int i = 0 ; i != count ; i++)
>>> list.add(i) ;
>>> for (int r = 0 ; r != 100 ; r++) {
>>>  final long start = System.currentTimeMillis() ;
>>> int o = 0 ;
>>> for (int j = 0 ; j != count ; j++) {
>>>  list.get((j + o) & mask) ;
>>> o += 1 ;
>>> }
>>>  final long end = System.currentTimeMillis() ;
>>> sum += (end - start) ;
>>>                         System.out.println(String.format("run %d: %dms;
>>> avg=%.0fms", r, (end - start), sum / (r + 1))) ;
>>>  }
>>> }
>>>
>>>
>>> On 24 April 2010 09:34, Benedict Elliott Smith <lists at laerad.com> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> If you are willing to change the pattern of allocation just slightly, I
>>>> have come up with an alternate algorithm (optimised for the seed = 1 case)
>>>> that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
>>>> for random(ish) calls to get() on a list of size 1 << 24. The change is to
>>>> simply drop your first array allocation (so that there are no arrays of size
>>>> 1, but that all remaining allocations follow the existing pattern), as this
>>>> allows simplifying the algorithm noticeably (several ops in the previous
>>>> algorithm were unnecessary for any but the first two arrays).
>>>>
>>>> My get(index) method is defined as:
>>>>
>>>> if ((index < 0) | (index >= _size))
>>>> throw new IllegalArgumentException() ;
>>>>  index += 2 ;
>>>> final Object[] array = _backingArray[arrayFor(index)] ;
>>>>  return (T) array[index & (array.length - 1)] ;
>>>>
>>>> and arrayFor(index) is defined as:
>>>>
>>>> private static final int arrayFor(int index) {
>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>> Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
>>>> final int b1 = (2 << arraySizeShiftMinusSeed << arraySizeShiftMinusSeed)
>>>> ;
>>>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>>>> final int b2 = index - b1 ;
>>>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>>>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>>>  final int av = a1 - a2 ;
>>>> final int bv = b4 - 3 ;
>>>> return av + bv ;
>>>>  }
>>>>
>>>> I have deliberately interleaved the calculations here to make sure the
>>>> pipeline is being used (just in case javac+hotspot are not re-ordering the
>>>> intermediate calculations for us)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com>wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> It looks like this is because ChunkedArrayList creates only one initial
>>>>> array of size s, whereas my algorithm expects two . Apologies for not
>>>>> spotting this - the pattern of allocations is identical after this point.
>>>>> I'll see if it can be modified to support your pattern of allocation.
>>>>>
>>>>>
>>>>> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>
>>>>>> Hi Benedict,
>>>>>>
>>>>>> I took a look at your index decomposition routine; it was not working
>>>>>> for seed = 1 until I made index the query index plus one (similar to my r
>>>>>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>>>>>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>>>>>> changes the routine was actually slower than my version (although not by
>>>>>> much).  Let me know if you have a better way to bring your routine in line
>>>>>> with the arraylet structure.
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <
>>>>>> lists at laerad.com> wrote:
>>>>>>
>>>>>>> Hi Kevin,
>>>>>>>
>>>>>>> Unfortunately this week has been pretty hectic, and I haven't had
>>>>>>> much time to much more than theorise on this topic - and this weekend the
>>>>>>> weather looks set to be much too nice to stay in doors! It looks like you've
>>>>>>> made really good progress on the ChunkedArrayDeque - I haven't fully
>>>>>>> digested it yet, but it looks pretty impressive. I'm not sure (since I
>>>>>>> misunderstood your list implementation prior to reading it in detail) but I
>>>>>>> think I may have been toying with a similar growth strategy for a hash map
>>>>>>> variant since the copies are necessary there anyway.
>>>>>>>
>>>>>>> I have taken another look at the index algorithm and have tidied it
>>>>>>> up as below, and it now supports a seed size of 1; however I am not sure
>>>>>>> that using a value this small is advisable, given that the overhead for the
>>>>>>> first array is at least 60 bytes; with a seed size of 1 the first 8 indexes
>>>>>>> would utilise less than 20% of the allocated memory for data, the first 24
>>>>>>> less than 25%. I would have thought a seed of at least 2 and perhaps 3 would
>>>>>>> be advisable.
>>>>>>>
>>>>>>> As an aside, it is worth noting that the indexOffset calculation
>>>>>>> below can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>>>>>> although I am not certain what effect this will have on performance, given
>>>>>>> that this would be another memory operation to retrieve the length from the
>>>>>>> array; but it would make the separation of the calculation into functions
>>>>>>> more straight forward.
>>>>>>>
>>>>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>>>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>
>>>>>>>      final int firstArrayOfThisSize =
>>>>>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>>>>>      - (1 << arraySizeShiftMinusSeed)
>>>>>>>      - 1 - (arraySizeShift >>> 31) ;
>>>>>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>>>>>> arraySizeShiftMinusSeed) ;
>>>>>>>
>>>>>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>>>>>> Integer.MAX_VALUE ;
>>>>>>>
>>>>>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>
>>>>>>>> Hi Benedict,
>>>>>>>>
>>>>>>>> Have you had a chance to get your index decomposition procedure to
>>>>>>>> work with seed values less than two?
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>>
>>>>>>>>> As it happens I might have something useful still to contribute. As
>>>>>>>>> an exercise in saving face I revisited the problem to see if I could achieve
>>>>>>>>> the same complexity bounds as ChunkedArrayList but with a lower overhead. I
>>>>>>>>> must admit I still didn't fully appreciate how the algorithm in
>>>>>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>>>>>> similar properties. What I have ended up with is almost identical except
>>>>>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>>>>>
>>>>>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>>>>>> is approximately the same.
>>>>>>>>>
>>>>>>>>> Secondly, I have modified the algorithm so that a "seed" size can
>>>>>>>>> be specified (although I expect hard coding a suitable one will ultimately
>>>>>>>>> be best). Whereas ChunkedArrayList currently requires that the pattern of
>>>>>>>>> array allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we
>>>>>>>>> can now support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6),
>>>>>>>>> 8*s*(..*12), 16*s*(..*24)] etc. although when put in simple text
>>>>>>>>> like that it does appear to trivialise the change. The benefit of this,
>>>>>>>>> though, is two fold: 1) for small n the constant factor is reduced (both CPU
>>>>>>>>> and memory wise); and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>>>>>
>>>>>>>>> As an illustration, consider setting *s* to 4, and assume the
>>>>>>>>> backing array is size two and doubles in size with each growth; with
>>>>>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s*as 4 we would instead resize at i=8,i=24,i=80,i=288; the cost at each would
>>>>>>>>> be some multiple of 2,4,8,16 respectively. As you can see the latter is much
>>>>>>>>> closer to the sqrt(n) cost - both approach it eventually, but my suggestion
>>>>>>>>> is to reach it more quickly. This is at the expense of more slowly reaching
>>>>>>>>> the sqrt(n) wasted memory condition, but given the high constant factor cost
>>>>>>>>> wrt to memory at this early stage, this seems a very sensible trade off. It
>>>>>>>>> seems likely this should also have a positive impact on cache performance
>>>>>>>>> for smaller lists as well.
>>>>>>>>>
>>>>>>>>> Finally, after playing with this idea in my head I am confident I
>>>>>>>>> can extend the core ideas of this data structure to hashing relatively
>>>>>>>>> easily, getting the the same worst case O(sqrt(n)) insertion cost, and
>>>>>>>>> O(sqrt(n)) wasted memory guarantees. I notice that this case hasn't been
>>>>>>>>> addressed yet, although I see from Martin's recent mail that this was raised
>>>>>>>>> before. Unless there are better suggestions for solving the hash table
>>>>>>>>> problem I will have a go at it as it seems an interesting problem - that is,
>>>>>>>>> assuming there are no objections?
>>>>>>>>>
>>>>>>>>> I'm interested to hear your thoughts. I hope this time I've been a
>>>>>>>>> bit more considered in what I've put forward, and hence less of a waste of
>>>>>>>>> time!
>>>>>>>>>
>>>>>>>>> Code snippet for calculation of array index and item offset:
>>>>>>>>>
>>>>>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>>>  final int firstArrayOfThisSize = ((((1 << arraySizeShiftMinusSeed
>>>>>>>>> + 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) - 1 ;
>>>>>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>>>
>>>>>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>>>
>>>>>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to
>>>>>>>>> *s* + 1); seed only works for values for 2 or more at this moment,
>>>>>>>>> fyi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Oh no worries Benedict, thanks for your interest in the topic.
>>>>>>>>>> Let me know if you have any other questions or if you have any related ideas
>>>>>>>>>> or concerns.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance
>>>>>>>>>>> than help. that part of the discussion was clearly truncated by the time I
>>>>>>>>>>> had joined the list - I haven't been able to find the history in the
>>>>>>>>>>> archives either...
>>>>>>>>>>>
>>>>>>>>>>> I was just wondering about the worst case cost of add() as
>>>>>>>>>>> described by your javadoc; admittedly it is optimal with respect to unused
>>>>>>>>>>> memory, but the worst case cost of an add is still sqrt(n), with a
>>>>>>>>>>> relatively high constant factor. I had been thinking that once n passed a
>>>>>>>>>>> threshold the cost of additions in this other structure would become a
>>>>>>>>>>> constant factor, offering nice algorithmic complexity guarantees for large
>>>>>>>>>>> n; however since sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new
>>>>>>>>>>> array allocations would have to be unrealistically small (assuming linear
>>>>>>>>>>> cost for allocation) for this to be the case. It would still be nice to have
>>>>>>>>>>> a data structure that avoids needing to copy data with each grow, whilst
>>>>>>>>>>> still maintaining good memory performance.
>>>>>>>>>>>
>>>>>>>>>>> That *all* being said, I had been going by your javadoc and
>>>>>>>>>>> emails to ascertain the behaviour of this class, as I couldn't locate a free
>>>>>>>>>>> copy of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>>>>>> large values of n.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>
>>>>>>>>>>>> Unless I am misreading your post, this now has a very similar
>>>>>>>>>>>> feel to the first data structure that I posted to the list.  Martin Buchholz
>>>>>>>>>>>> then pointed out that we can incorporate the ideas from
>>>>>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Kevin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, as I was going to bed last night I realised I had not
>>>>>>>>>>>>> fully addressed the problem that was originally being visited; only reduced
>>>>>>>>>>>>> the constant factor for addition to the end of the list. A trivial
>>>>>>>>>>>>> modification fixes that, however; same scheme but up to some maximum
>>>>>>>>>>>>> arraylet size (of a power of 2), after which the array is increased in size
>>>>>>>>>>>>> linearly. Performance doesn't seem to have been affected appreciably,
>>>>>>>>>>>>> although not been exhaustive in the benchmarking:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22,
>>>>>>>>>>>>> ExpArray=1.03
>>>>>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22,
>>>>>>>>>>>>> ExpArray=1.03
>>>>>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82,
>>>>>>>>>>>>> ExpArray=0.75
>>>>>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>>>>>
>>>>>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Code for calculating index and array offset are pretty straight
>>>>>>>>>>>>> forward; haven't given much thought to optimisations just yet:
>>>>>>>>>>>>>
>>>>>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>>>>>  }
>>>>>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>> maxArraySizeShift)
>>>>>>>>>>>>> + maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i + 1) ;
>>>>>>>>>>>>>  }
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly
>>>>>>>>>>>>> didn't think that one through fully!
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <
>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding:  "The idea is to simply double the new array size
>>>>>>>>>>>>>> each time a new array needs to be allocated"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it
>>>>>>>>>>>>>> seems that the simplest solution would be to simply have two lists, with one
>>>>>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What happens with your structure when you add n elements and
>>>>>>>>>>>>>> then remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>>>>>> resize.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion,
>>>>>>>>>>>>>>> but I have tinkered with a "chunked" array list which seems to offer better
>>>>>>>>>>>>>>> time performance in general at the cost of greater (worst case) memory
>>>>>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have prototyped the data structure this evening and
>>>>>>>>>>>>>>> benchmarked additions at the end of the list, for which the performance is
>>>>>>>>>>>>>>> pretty impressive.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Some random statistics for addition only on the client JVM (I
>>>>>>>>>>>>>>> have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per
>>>>>>>>>>>>>>> round per statistic per list, and the second round results were used.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and server JVM:
>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Interestingly insertion at the beginning of the list appears
>>>>>>>>>>>>>>> to be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>>>>>> larger lists:
>>>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26, ExpArray=1.02
>>>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05,
>>>>>>>>>>>>>>> ExpArray=0.86
>>>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that
>>>>>>>>>>>>>>> the simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining
>>>>>>>>>>>>>>> in...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The idea is to get a replacement for arraylist that performs
>>>>>>>>>>>>>>>> like arraydeque on remove(0). As a side effect, we should be able to get
>>>>>>>>>>>>>>>> better performance on other operations by requiring fewer elements to be
>>>>>>>>>>>>>>>> moved.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that
>>>>>>>>>>>>>>>>> add does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from
>>>>>>>>>>>>>>>>> Java's ArrayDeque?  I took only a cursory look at it, so please understand
>>>>>>>>>>>>>>>>> if I have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'd be interested if you have any comments in the context
>>>>>>>>>>>>>>>>>> of this discussion. The code is not entirely ready yet, a couple of tests
>>>>>>>>>>>>>>>>>> fail (6/789) because of a corner case I haven't nailed yet, but the idea is
>>>>>>>>>>>>>>>>>> there at least. I'd like to add array shrinking later, when the size dips
>>>>>>>>>>>>>>>>>> below capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the
>>>>>>>>>>>>>>>>>> O(1) operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>>>>>       102          90         240         191         138
>>>>>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>>>>>>     138          94         230         194         118
>>>>>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>>>>>>     538          91         324         382         119
>>>>>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>>>>>      4425          94        1296        2330         124
>>>>>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>>>>> ArrayList ----------------------------------------------------
>>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>>>>>        69       32913         162         130         105
>>>>>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>>>>>>     104       21944         169         134         135
>>>>>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>>>>>>     506        9602         364         333         526
>>>>>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>>>>>      4414        9947        2312        2280        4437
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It's interesting to note that the old circular list
>>>>>>>>>>>>>>>>>>>> trick will not suffice to turn this data structure into a deque since we
>>>>>>>>>>>>>>>>>>>> might be copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and
>>>>>>>>>>>>>>>>>>>> Erik D. Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and
>>>>>>>>>>>>>>>>>>>> Space},
>>>>>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out
>>>>>>>>>>>>>>>>>>>>> bugs in the
>>>>>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be in
>>>>>>>>>>>>>>>>>>>>> decent shape.  I have
>>>>>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've
>>>>>>>>>>>>>>>>>>>>> included below (note that the
>>>>>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the
>>>>>>>>>>>>>>>>>>>>> specified operation with
>>>>>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList,
>>>>>>>>>>>>>>>>>>>>> so 1.00 indicates
>>>>>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent
>>>>>>>>>>>>>>>>>>>>> rough performance
>>>>>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements and
>>>>>>>>>>>>>>>>>>>>> then timed the process
>>>>>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and
>>>>>>>>>>>>>>>>>>>>> finally I used the iterator
>>>>>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element
>>>>>>>>>>>>>>>>>>>>> (of course, I performed
>>>>>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing away
>>>>>>>>>>>>>>>>>>>>> the timing for the
>>>>>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this belongs
>>>>>>>>>>>>>>>>>>>>> in java.util, I would
>>>>>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of
>>>>>>>>>>>>>>>>>>>>> view to eliminate the
>>>>>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your
>>>>>>>>>>>>>>>>>>>>> suggestion of achieving this by
>>>>>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only
>>>>>>>>>>>>>>>>>>>>> guarantees that no backing
>>>>>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it
>>>>>>>>>>>>>>>>>>>>> also provides dynamic
>>>>>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead
>>>>>>>>>>>>>>>>>>>>> than ArrayList, and
>>>>>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a
>>>>>>>>>>>>>>>>>>>>> resize than ArrayList.  Of
>>>>>>>>>>>>>>>>>>>>> > course, this data structure does not do everything
>>>>>>>>>>>>>>>>>>>>> better than ArrayList; in
>>>>>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to the
>>>>>>>>>>>>>>>>>>>>> required decomposition
>>>>>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and
>>>>>>>>>>>>>>>>>>>>> the additional memory
>>>>>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more
>>>>>>>>>>>>>>>>>>>>> costly, due to the multiple
>>>>>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That
>>>>>>>>>>>>>>>>>>>>> being said, I think that
>>>>>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the additional
>>>>>>>>>>>>>>>>>>>>> cost of
>>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the
>>>>>>>>>>>>>>>>>>>>> fact that
>>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable
>>>>>>>>>>>>>>>>>>>>> operation on ArrayList due
>>>>>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the
>>>>>>>>>>>>>>>>>>>>> surprising
>>>>>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in
>>>>>>>>>>>>>>>>>>>>> java.util?
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>>>>>> >> > The data structure is available at the second link
>>>>>>>>>>>>>>>>>>>>> that I originally
>>>>>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the
>>>>>>>>>>>>>>>>>>>>> front as yet as it was
>>>>>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by
>>>>>>>>>>>>>>>>>>>>> multiple small arrays
>>>>>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace only
>>>>>>>>>>>>>>>>>>>>> java.util.ArrayList
>>>>>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support
>>>>>>>>>>>>>>>>>>>>> efficient add-at-front or
>>>>>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is still
>>>>>>>>>>>>>>>>>>>>> a much more important
>>>>>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so
>>>>>>>>>>>>>>>>>>>>> I'll be updating the
>>>>>>>>>>>>>>>>>>>>> >> > document at the provided link as I find
>>>>>>>>>>>>>>>>>>>>> improvements.
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>>> martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who has
>>>>>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to undergo
>>>>>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the
>>>>>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array
>>>>>>>>>>>>>>>>>>>>> and the offset into the
>>>>>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I
>>>>>>>>>>>>>>>>>>>>> mentioned that this will
>>>>>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a
>>>>>>>>>>>>>>>>>>>>> simple bit shift and a bit
>>>>>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be
>>>>>>>>>>>>>>>>>>>>> doing this each time an
>>>>>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) << 1;
>>>>>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) &
>>>>>>>>>>>>>>>>>>>>> (powFloorKO2 - 1);
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " +
>>>>>>>>>>>>>>>>>>>>> (p + b) + " " + e);
>>>>>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L. Stern
>>>>>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works
>>>>>>>>>>>>>>>>>>>>> for even numbered
>>>>>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an
>>>>>>>>>>>>>>>>>>>>> additional term added (the
>>>>>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that
>>>>>>>>>>>>>>>>>>>>> my example structure
>>>>>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of
>>>>>>>>>>>>>>>>>>>>> (1) below.  I must be
>>>>>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L.
>>>>>>>>>>>>>>>>>>>>> Stern
>>>>>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable arrays
>>>>>>>>>>>>>>>>>>>>> in optimal time and
>>>>>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with
>>>>>>>>>>>>>>>>>>>>> the following
>>>>>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully
>>>>>>>>>>>>>>>>>>>>> allocated, it consists of
>>>>>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed
>>>>>>>>>>>>>>>>>>>>> this implies the
>>>>>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i),
>>>>>>>>>>>>>>>>>>>>> with i = 3:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There
>>>>>>>>>>>>>>>>>>>>> are only two data
>>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given
>>>>>>>>>>>>>>>>>>>>> (1) above, unless I'm
>>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data blocks
>>>>>>>>>>>>>>>>>>>>> in superblocks prior
>>>>>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much
>>>>>>>>>>>>>>>>>>>>> better in my example above,
>>>>>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my
>>>>>>>>>>>>>>>>>>>>> interpretation of their data
>>>>>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is
>>>>>>>>>>>>>>>>>>>>> their mistake rather than
>>>>>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin
>>>>>>>>>>>>>>>>>>>>> Buchholz
>>>>>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L.
>>>>>>>>>>>>>>>>>>>>> Stern
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer -
>>>>>>>>>>>>>>>>>>>>> to insert at the
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo the
>>>>>>>>>>>>>>>>>>>>> size) and to insert
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer
>>>>>>>>>>>>>>>>>>>>> (modulo the size).  We
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each
>>>>>>>>>>>>>>>>>>>>> other.  Could you explain
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an arraylet
>>>>>>>>>>>>>>>>>>>>> that is shared by the
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single
>>>>>>>>>>>>>>>>>>>>> array.
>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow,
>>>>>>>>>>>>>>>>>>>>> the paper that you
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and
>>>>>>>>>>>>>>>>>>>>> space", gives a deque so
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is specified.
>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the
>>>>>>>>>>>>>>>>>>>>> Deque operations -
>>>>>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100424/f5944a56/attachment.html>

From martinrb at google.com  Sun Apr 25 00:50:42 2010
From: martinrb at google.com (Martin Buchholz)
Date: Sat, 24 Apr 2010 17:50:42 -0700
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
Message-ID: <p2q1ccfd1c11004241750u26965ca2y14a953fcf3cc3e41@mail.gmail.com>

On Tue, Apr 13, 2010 at 04:18, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
> The FAQ for the Sun Contributor Agreement Q3
> (http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
> indicates that one should check with the project to determine where the SCA
> should be sent.? Do you know where I would find this information?

I don't know any more than what it says at

http://openjdk.java.net/contribute/

Like many other open-source communities, the OpenJDK Community
requires contributors to jointly assign their copyright on contributed
code. If you haven't yet signed the Sun Contributor Agreement (SCA)
then please do so and fax it to +1-408-715-2540, or scan it and e-mail
the result to sun_ca(at)sun.com.

Thanks,

Martin


From kelly.ohair at oracle.com  Sun Apr 25 01:36:59 2010
From: kelly.ohair at oracle.com (Kelly O'Hair)
Date: Sat, 24 Apr 2010 18:36:59 -0700
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <p2q1ccfd1c11004241750u26965ca2y14a953fcf3cc3e41@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<u2t1704b7a21003311636q35093107l60a1d931d6aa6f66@mail.gmail.com>
	<t2m1704b7a21003311708u345bc533g4ea87d2db2956373@mail.gmail.com>
	<u2r1704b7a21003311934y3f4917f1r18677488ac77e33c@mail.gmail.com>
	<l2t1ccfd1c11004012028w105b220g7bda7e3f14068b9e@mail.gmail.com>
	<r2i1704b7a21004040412nd2859de1xafeb1f215a7d335c@mail.gmail.com>
	<q2l1ccfd1c11004091742k446289a6u94732cad0a471b4a@mail.gmail.com>
	<y2r1704b7a21004110932t4bae3df7q22cb2c98a7fb5b06@mail.gmail.com>
	<h2p1ccfd1c11004111417i58c570eft8c320063707e178@mail.gmail.com>
	<q2y1704b7a21004130418q857fe354n1e9873c07297d56d@mail.gmail.com>
	<p2q1ccfd1c11004241750u26965ca2y14a953fcf3cc3e41@mail.gmail.com>
Message-ID: <BAF934A7-B671-4B57-A876-D7C1440067CB@oracle.com>

Try here:

https://sca.dev.java.net

https://sca.dev.java.net/CA_signatories.htm

-kto

On Apr 24, 2010, at 5:50 PM, Martin Buchholz wrote:

> On Tue, Apr 13, 2010 at 04:18, Kevin L. Stern  
> <kevin.l.stern at gmail.com> wrote:
>> The FAQ for the Sun Contributor Agreement Q3
>> (http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3 
>> )
>> indicates that one should check with the project to determine where  
>> the SCA
>> should be sent.  Do you know where I would find this information?
>
> I don't know any more than what it says at
>
> http://openjdk.java.net/contribute/
>
> Like many other open-source communities, the OpenJDK Community
> requires contributors to jointly assign their copyright on contributed
> code. If you haven't yet signed the Sun Contributor Agreement (SCA)
> then please do so and fax it to +1-408-715-2540, or scan it and e-mail
> the result to sun_ca(at)sun.com.
>
> Thanks,
>
> Martin



From xueming.shen at oracle.com  Sun Apr 25 06:56:44 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Sat, 24 Apr 2010 23:56:44 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>
References: <4BD00250.3020206@oracle.com>
	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>
Message-ID: <4BD3E7AC.2020801@oracle.com>

Martin Buchholz wrote:
> Providing script support is obvious and non-controversial,
> because other regex programming environments provide it.
> Check that the behavior and syntax of the extension is
> consistent with e.g. ICU, python, and especially perl
> (5.12 just released!)
>
> http://perldoc.perl.org/perlunicode.html
>   

\p{propName=propValue} is the unicode "compound form", which is supported in
perl 5.12. It also has a variant type \p{propName:propValue}. It was in 
my proposal,
but I removed it the last minutes. Two forms (\p{In/IsProp} and 
\p{propName=propValue}
should be good enough for now. Three is a little too much. We can always 
add it
in later, if desirable.

\p{IsScript}, \p{Isgc}, \p{InBlock} are perl compatible as well.

> I would add some documentation to the three special script values;
> their meaning is not obvious.
>
>   
I think it might be better to justt leave the detailed explain doc to 
the TR#24. The "script"
here in j.l.Character serves only the purpose of id, the API here should 
not be the place
to explain "what they really are".

> For implementation, the character matching problem is in general
> equivalent to the problem of compiling a switch statement, which is
> known to be non-trivial.  Guava contains a CharMatcher class that
> tries to solve related problems.
>
> http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/CharMatcher.html
>
> I'm thinking scripts and blocks should know about which ranges they contain.
> In particular, \p{BlockName} should not need binary search at
> regex compile time or runtime.
>   
It definitely is desirable if we can avoid the binary-search lookup 
during at least the runtime. The
cost will be to keep a separate/redundant block/script->ranges table in 
regex itself.

> ---
> There is one place you need to change
> key word => keyword
> ---
> InMongolian => {@code InMongolian}
> ---
>   

Good catch, thanks!

> I notice current Unicode block support in JDK is not updated to the
> latest standard.
> E.g. Samaritan is missing.
>
>   
The Character class has not been updated to the latest 5.20 yet. Yuka 
has a CCC pending for
that. My script data is from the 5.20.


> Martin
>
> On Thu, Apr 22, 2010 at 01:01, Xueming Shen <xueming.shen at oracle.com> wrote:
>   
>> Hi,
>>
>> Here is the webrev of the proposal to add Unicode script support in regex
>> and j.l.Character.
>>
>> http://cr.openjdk.java.net/~sherman/script/webrev
>>
>> and the corresponding blenderrev
>>
>> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
>>
>> Please comment on the APIs before I submit the CCC, especially
>>
>> (1) to use enum for the j.l.Character.UnicodeScript (compared to the
>> traditional j.l.c.Subset)
>> (2) the piggyback method j.l.c.getName() :-)
>> (3) the syntax for script constructs. In addition to the "normal"
>>    \p{InScriptName} and \P{InScriptName} for the script support
>>    I'm also adding
>>   \p{script=ScriptName} \P{script=ScriptName}  for the new script support
>>   \p{block=BlockName} \P{block=BlockName}  for the "existing" block support
>>   \p{general_category=CategoryName} \P{general_category=CategoryName} for
>> the "existing" gc
>>   Perl recently also started to accept this  \p{propName=propValue} Unicode
>> style.
>>   It opens the door for future "expanding", for example \p{name=XYZ} :-)
>> (4)and of course, the wording.
>>
>> Thanks,
>> Sherman
>>
>>
>>
>>     



From lists at laerad.com  Sun Apr 25 09:25:38 2010
From: lists at laerad.com (Benedict Elliott Smith)
Date: Sun, 25 Apr 2010 10:25:38 +0100
Subject: A List implementation backed by multiple small arrays rather than
	the traditional single large array.
In-Reply-To: <x2v1704b7a21004241141u9836a344i2b9d59c7fb574159@mail.gmail.com>
References: <1704b7a21003280455u784d4d2ape39a47e2367b79a8@mail.gmail.com>
	<q2k1704b7a21004230300o15d88bden5a5e6bfc8bb0953b@mail.gmail.com>
	<v2x51a29d951004231226qdc3453e0t4650bd6a3e57418d@mail.gmail.com>
	<x2z1704b7a21004231731m3b8a40e7ze1f3a8688e0762aa@mail.gmail.com>
	<g2s51a29d951004240024pdcf5f139if118902f4b29fedd@mail.gmail.com>
	<i2v51a29d951004240134ve8eac831n8dcd024eeaf43946@mail.gmail.com>
	<w2t51a29d951004241030j5f4b4e7axc7ffb5274b7fa3ca@mail.gmail.com>
	<r2h1704b7a21004241114wfca78ae7lb3e9571316ca5b3f@mail.gmail.com>
	<z2x51a29d951004241127l120c8d6ek986f2ce93a7550d7@mail.gmail.com>
	<x2v1704b7a21004241141u9836a344i2b9d59c7fb574159@mail.gmail.com>
Message-ID: <m2r51a29d951004250225z212e4e9ej1b15003bf84d5e60@mail.gmail.com>

Wow, those numbers really are very impressive. I had to double check against
your previous statistics to confirm the direction of the ratio... Those
numbers make this data structure look like the weapon of choice for any but
the smallest of lists.

On 24 April 2010 19:41, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:

> Hi Benedict,
>
> You are absolutely right - the constant on the cost of growth is going to
> be higher with the new data structure.  That being said, the performance
> numbers that I'm getting with it are actually quite impressive when compared
> to ArrayList:
>
> Note that all numbers are with a client VM and are generated using the same
> routine that I used for the ChunkedArrayList performance numbers.
>
> 1000000 elements:
>
> Add to ChunkedArrayDeque over ArrayList: 0.89
> Indexed access ChunkedArrayDeque over ArrayList: 0.81
> Iterator ChunkedArrayDeque over ArrayList: 0.51
>
> 100000 elements:
>
> Add to ChunkedArrayDeque over ArrayList: 1.01
> Indexed access ChunkedArrayDeque over ArrayList: 0.79
> Iterator ChunkedArrayDeque over ArrayList: 0.50
>
> 10000 elements:
>
> Add to ChunkedArrayDeque over ArrayList: 1.15
> Indexed access ChunkedArrayDeque over ArrayList: 1.15
> Iterator ChunkedArrayDeque over ArrayList: 0.51
>
> 1000 elements:
>
> Add to ChunkedArrayDeque over ArrayList: 1.35
> Indexed access ChunkedArrayDeque over ArrayList: 1.12
> Iterator ChunkedArrayDeque over ArrayList: 0.54
>
> Kevin
>
>
> On Sat, Apr 24, 2010 at 1:27 PM, Benedict Elliott Smith <lists at laerad.com>wrote:
>
>> Yes, I had spotted that benefit - but (please correct me if I am
>> misreading this as it is quite possible) in order to maintain your array
>> allocation invariant, array copies are needed (rather than straight
>> allocations) - and so the cost of growth is likely to be noticeably larger.
>> That said, if random access is considerably quicker this might be a sensible
>> trade off, but perhaps there is a place for both data structures.
>>
>>
>>
>> On 24 April 2010 19:14, Kevin L. Stern <kevin.l.stern at gmail.com> wrote:
>>
>>> Hi Benedict,
>>>
>>> Thanks, I'll definitely give this a try.  I certainly don't see any issue
>>> with skipping the size 1 array in order to speed up general operation.
>>>
>>> By the way, I'm currently focused a bit more on the deque - I'm not sure
>>> that anything is going to come of this ChunkedArrayList on its own.  With
>>> the deque, index decomposition is much faster than any of these
>>> implementations as it lends itself to bit operations without the requirement
>>> of computing the logarithm.
>>>
>>> Kevin
>>>
>>>
>>> On Sat, Apr 24, 2010 at 12:30 PM, Benedict Elliott Smith <
>>> lists at laerad.com> wrote:
>>>
>>>> If you want a drop in replacement with minimal fuss to test it out, try
>>>> below (although it's a quick hack of including the
>>>> Integer.numberOfLeadingZeros() call to avoid its now unnecessary non-zero
>>>> test, and instead utilise this op to return the zero index). Just delete the
>>>> existing get() method and replace it with this code. Actually this version
>>>> seems to improve throughput further - on my laptop regular ChunkedArrayList
>>>> is currently averaging around 790ms per run of my hacky speed test, whereas
>>>> with the modifications below it averages around 610ms, making it almost 25%
>>>> faster. I've included my performance benchmark as well, for reference. I'm
>>>> sure with some thought a better one could be put together.
>>>>
>>>>     private static final int arrayIndex2(int index) {
>>>>         if (index == 1)
>>>>             return 0 ;
>>>>         int i = index >>> 1 ;
>>>>         int n = 1;
>>>>         if (i >>> 16 == 0) { n += 16; i <<= 16; }
>>>>         if (i >>> 24 == 0) { n +=  8; i <<=  8; }
>>>>         if (i >>> 28 == 0) { n +=  4; i <<=  4; }
>>>>         if (i >>> 30 == 0) { n +=  2; i <<=  2; }
>>>>         n -= i >>> 31;
>>>>      final int arraySizeShiftMinusSeed = (31 - n) >>> 1 ;
>>>>  final int b1 = (2 << arraySizeShiftMinusSeed <<
>>>> arraySizeShiftMinusSeed) ;
>>>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>>>> final int b2 = index - b1 ;
>>>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>>>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>>>  final int av = a1 - a2 ;
>>>> final int bv = b4 - 2 ;
>>>>  return av + bv ;
>>>>     }
>>>>
>>>>     @Override
>>>>     public T get(int index) {
>>>>         if ((0 > index) | (index >= _size))
>>>>             throw new IndexOutOfBoundsException();
>>>>         index += 1 ;
>>>>         final Object[] array = _backingArray[arrayIndex2(index)] ;
>>>>         return (T) array[index & (array.length - 1)] ;
>>>>     }
>>>>
>>>>
>>>> //// benchmarked by
>>>>
>>>> static final int count = 1 << 24 ;
>>>> static final int mask = count - 1 ;
>>>>  public static void main(String[] args) {
>>>> double sum = 0 ;
>>>> final List<Integer> list = new ChunkedArrayList<Integer>() ;
>>>>  for (int i = 0 ; i != count ; i++)
>>>> list.add(i) ;
>>>> for (int r = 0 ; r != 100 ; r++) {
>>>>  final long start = System.currentTimeMillis() ;
>>>> int o = 0 ;
>>>> for (int j = 0 ; j != count ; j++) {
>>>>  list.get((j + o) & mask) ;
>>>> o += 1 ;
>>>> }
>>>>  final long end = System.currentTimeMillis() ;
>>>> sum += (end - start) ;
>>>>                         System.out.println(String.format("run %d: %dms;
>>>> avg=%.0fms", r, (end - start), sum / (r + 1))) ;
>>>>  }
>>>> }
>>>>
>>>>
>>>> On 24 April 2010 09:34, Benedict Elliott Smith <lists at laerad.com>wrote:
>>>>
>>>>> Hi Kevin,
>>>>>
>>>>> If you are willing to change the pattern of allocation just slightly, I
>>>>> have come up with an alternate algorithm (optimised for the seed = 1 case)
>>>>> that on my laptop I notice around a 10-20% speed up over ChunkedArrayList
>>>>> for random(ish) calls to get() on a list of size 1 << 24. The change is to
>>>>> simply drop your first array allocation (so that there are no arrays of size
>>>>> 1, but that all remaining allocations follow the existing pattern), as this
>>>>> allows simplifying the algorithm noticeably (several ops in the previous
>>>>> algorithm were unnecessary for any but the first two arrays).
>>>>>
>>>>> My get(index) method is defined as:
>>>>>
>>>>> if ((index < 0) | (index >= _size))
>>>>> throw new IllegalArgumentException() ;
>>>>>  index += 2 ;
>>>>> final Object[] array = _backingArray[arrayFor(index)] ;
>>>>>  return (T) array[index & (array.length - 1)] ;
>>>>>
>>>>> and arrayFor(index) is defined as:
>>>>>
>>>>> private static final int arrayFor(int index) {
>>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>>> Integer.numberOfLeadingZeros(index >>> 1)) >>> 1 ;
>>>>> final int b1 = (2 << arraySizeShiftMinusSeed <<
>>>>> arraySizeShiftMinusSeed) ;
>>>>>      final int a1 = (1 << arraySizeShiftMinusSeed + 2) ;
>>>>> final int b2 = index - b1 ;
>>>>>      final int a2 = (1 << arraySizeShiftMinusSeed) ;
>>>>> final int b4 = b2 >>> arraySizeShiftMinusSeed + 1 ;
>>>>>  final int av = a1 - a2 ;
>>>>> final int bv = b4 - 3 ;
>>>>> return av + bv ;
>>>>>  }
>>>>>
>>>>> I have deliberately interleaved the calculations here to make sure the
>>>>> pipeline is being used (just in case javac+hotspot are not re-ordering the
>>>>> intermediate calculations for us)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 24 April 2010 08:24, Benedict Elliott Smith <lists at laerad.com>wrote:
>>>>>
>>>>>> Hi Kevin,
>>>>>>
>>>>>> It looks like this is because ChunkedArrayList creates only one
>>>>>> initial array of size s, whereas my algorithm expects two . Apologies for
>>>>>> not spotting this - the pattern of allocations is identical after this
>>>>>> point. I'll see if it can be modified to support your pattern of allocation.
>>>>>>
>>>>>>
>>>>>> On 24 April 2010 01:31, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>
>>>>>>> Hi Benedict,
>>>>>>>
>>>>>>> I took a look at your index decomposition routine; it was not working
>>>>>>> for seed = 1 until I made index the query index plus one (similar to my r
>>>>>>> variable) and arrayIndex ((firstArrayOfThisSize + arrayOffset) &
>>>>>>> Integer.MAX_VALUE) - 1 (notice I'm subtracting one).  Once I made these
>>>>>>> changes the routine was actually slower than my version (although not by
>>>>>>> much).  Let me know if you have a better way to bring your routine in line
>>>>>>> with the arraylet structure.
>>>>>>>
>>>>>>> Kevin
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 23, 2010 at 2:26 PM, Benedict Elliott Smith <
>>>>>>> lists at laerad.com> wrote:
>>>>>>>
>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> Unfortunately this week has been pretty hectic, and I haven't had
>>>>>>>> much time to much more than theorise on this topic - and this weekend the
>>>>>>>> weather looks set to be much too nice to stay in doors! It looks like you've
>>>>>>>> made really good progress on the ChunkedArrayDeque - I haven't fully
>>>>>>>> digested it yet, but it looks pretty impressive. I'm not sure (since I
>>>>>>>> misunderstood your list implementation prior to reading it in detail) but I
>>>>>>>> think I may have been toying with a similar growth strategy for a hash map
>>>>>>>> variant since the copies are necessary there anyway.
>>>>>>>>
>>>>>>>> I have taken another look at the index algorithm and have tidied it
>>>>>>>> up as below, and it now supports a seed size of 1; however I am not sure
>>>>>>>> that using a value this small is advisable, given that the overhead for the
>>>>>>>> first array is at least 60 bytes; with a seed size of 1 the first 8 indexes
>>>>>>>> would utilise less than 20% of the allocated memory for data, the first 24
>>>>>>>> less than 25%. I would have thought a seed of at least 2 and perhaps 3 would
>>>>>>>> be advisable.
>>>>>>>>
>>>>>>>> As an aside, it is worth noting that the indexOffset calculation
>>>>>>>> below can be replaced with index & (_backingArray[arrayIndex].length - 1) ,
>>>>>>>> although I am not certain what effect this will have on performance, given
>>>>>>>> that this would be another memory operation to retrieve the length from the
>>>>>>>> array; but it would make the separation of the calculation into functions
>>>>>>>> more straight forward.
>>>>>>>>
>>>>>>>>      final int arraySizeShiftMinusSeed = (31 -
>>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1 ;
>>>>>>>>      final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>>
>>>>>>>>      final int firstArrayOfThisSize =
>>>>>>>>       (1 << arraySizeShiftMinusSeed + 2)
>>>>>>>>      - (1 << arraySizeShiftMinusSeed)
>>>>>>>>      - 1 - (arraySizeShift >>> 31) ;
>>>>>>>>      final int indexRemainder = index - (1 << arraySizeShift <<
>>>>>>>> arraySizeShiftMinusSeed) ;
>>>>>>>>
>>>>>>>>      final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>>      final int arrayIndex = (firstArrayOfThisSize + arrayOffset) &
>>>>>>>> Integer.MAX_VALUE ;
>>>>>>>>
>>>>>>>>      final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23 April 2010 11:00, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> Hi Benedict,
>>>>>>>>>
>>>>>>>>> Have you had a chance to get your index decomposition procedure to
>>>>>>>>> work with seed values less than two?
>>>>>>>>>
>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Apr 17, 2010 at 11:48 AM, Benedict Elliott Smith <
>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Kevin,
>>>>>>>>>>
>>>>>>>>>> As it happens I might have something useful still to contribute.
>>>>>>>>>> As an exercise in saving face I revisited the problem to see if I could
>>>>>>>>>> achieve the same complexity bounds as ChunkedArrayList but with a lower
>>>>>>>>>> overhead. I must admit I still didn't fully appreciate how the algorithm in
>>>>>>>>>> ChunkedArrayList worked until I tried to come up with an algorithm with
>>>>>>>>>> similar properties. What I have ended up with is almost identical except
>>>>>>>>>> adds I think a couple of incremental improvements, simply by redefining the
>>>>>>>>>> arrayIndex() method. I should note that I have not yet implemented more than
>>>>>>>>>> a prototype as it seems to me your implementation is excellent already, and
>>>>>>>>>> if it is decided to include my modifications the changes should be modest.
>>>>>>>>>>
>>>>>>>>>> Firstly, (I hope that) what I have produced is a little more CPU
>>>>>>>>>> pipe-line friendly; there is less dependency on immediately preceding
>>>>>>>>>> calculations at each stage (i.e. so more operations should be able to
>>>>>>>>>> proceed simultaneously in the pipeline), and consists exclusively of shifts,
>>>>>>>>>> addition/subtraction and bit-wise (&)ands (except for the conditionals in
>>>>>>>>>> Integer.numberOfLeadingZeros(i)), although the total number of instructions
>>>>>>>>>> is approximately the same.
>>>>>>>>>>
>>>>>>>>>> Secondly, I have modified the algorithm so that a "seed" size can
>>>>>>>>>> be specified (although I expect hard coding a suitable one will ultimately
>>>>>>>>>> be best). Whereas ChunkedArrayList currently requires that the pattern of
>>>>>>>>>> array allocation sizes be [1, 1, 2, 2, 2, 4(..*6), 8(..*12), 16(..*24)] we
>>>>>>>>>> can now support, for some "*s*", [*s*(..*2), 2*s*(..*3), 4*s*(..*6),
>>>>>>>>>> 8*s*(..*12), 16*s*(..*24)] etc. although when put in simple text
>>>>>>>>>> like that it does appear to trivialise the change. The benefit of this,
>>>>>>>>>> though, is two fold: 1) for small n the constant factor is reduced (both CPU
>>>>>>>>>> and memory wise); and 2) the sqrt(n) bounds are reached more quickly also.
>>>>>>>>>>
>>>>>>>>>> As an illustration, consider setting *s* to 4, and assume the
>>>>>>>>>> backing array is size two and doubles in size with each growth; with
>>>>>>>>>> ChunkedArrayList we would resize at i=2, i=6, i=20, i=72; with *s
>>>>>>>>>> * as 4 we would instead resize at i=8,i=24,i=80,i=288; the cost
>>>>>>>>>> at each would be some multiple of 2,4,8,16 respectively. As you can see the
>>>>>>>>>> latter is much closer to the sqrt(n) cost - both approach it eventually, but
>>>>>>>>>> my suggestion is to reach it more quickly. This is at the expense of more
>>>>>>>>>> slowly reaching the sqrt(n) wasted memory condition, but given the high
>>>>>>>>>> constant factor cost wrt to memory at this early stage, this seems a very
>>>>>>>>>> sensible trade off. It seems likely this should also have a positive impact
>>>>>>>>>> on cache performance for smaller lists as well.
>>>>>>>>>>
>>>>>>>>>> Finally, after playing with this idea in my head I am confident I
>>>>>>>>>> can extend the core ideas of this data structure to hashing relatively
>>>>>>>>>> easily, getting the the same worst case O(sqrt(n)) insertion cost, and
>>>>>>>>>> O(sqrt(n)) wasted memory guarantees. I notice that this case hasn't been
>>>>>>>>>> addressed yet, although I see from Martin's recent mail that this was raised
>>>>>>>>>> before. Unless there are better suggestions for solving the hash table
>>>>>>>>>> problem I will have a go at it as it seems an interesting problem - that is,
>>>>>>>>>> assuming there are no objections?
>>>>>>>>>>
>>>>>>>>>> I'm interested to hear your thoughts. I hope this time I've been a
>>>>>>>>>> bit more considered in what I've put forward, and hence less of a waste of
>>>>>>>>>> time!
>>>>>>>>>>
>>>>>>>>>> Code snippet for calculation of array index and item offset:
>>>>>>>>>>
>>>>>>>>>> final int arraySizeShiftMinusSeed = ((31 -
>>>>>>>>>> Integer.numberOfLeadingZeros(index >>> seed)) >>> 1) ;
>>>>>>>>>> final int arraySizeShift = arraySizeShiftMinusSeed + seed ;
>>>>>>>>>>  final int firstArrayOfThisSize = ((((1 <<
>>>>>>>>>> arraySizeShiftMinusSeed + 3) - (1 << arraySizeShiftMinusSeed + 1))) >>> 1) -
>>>>>>>>>> 1 ;
>>>>>>>>>>  final int indexRemainder = index - ((1 << seed) <<
>>>>>>>>>> arraySizeShiftMinusSeed + arraySizeShiftMinusSeed) ;
>>>>>>>>>> final int arrayOffset = indexRemainder >>> arraySizeShift ;
>>>>>>>>>>
>>>>>>>>>> final int arrayIndex = firstArrayOfThisSize + arrayOffset ;
>>>>>>>>>> final int itemIndex = index & ((1 << arraySizeShift) - 1) ;
>>>>>>>>>>
>>>>>>>>>> the first array size will be 1 << seed - 1 (i.e. seed is equal to
>>>>>>>>>> *s* + 1); seed only works for values for 2 or more at this
>>>>>>>>>> moment, fyi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 16 April 2010 00:18, Kevin L. Stern <kevin.l.stern at gmail.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Oh no worries Benedict, thanks for your interest in the topic.
>>>>>>>>>>> Let me know if you have any other questions or if you have any related ideas
>>>>>>>>>>> or concerns.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Apr 15, 2010 at 8:00 AM, Benedict Elliott Smith <
>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry Kevin - it sounds like I might be being of more hindrance
>>>>>>>>>>>> than help. that part of the discussion was clearly truncated by the time I
>>>>>>>>>>>> had joined the list - I haven't been able to find the history in the
>>>>>>>>>>>> archives either...
>>>>>>>>>>>>
>>>>>>>>>>>> I was just wondering about the worst case cost of add() as
>>>>>>>>>>>> described by your javadoc; admittedly it is optimal with respect to unused
>>>>>>>>>>>> memory, but the worst case cost of an add is still sqrt(n), with a
>>>>>>>>>>>> relatively high constant factor. I had been thinking that once n passed a
>>>>>>>>>>>> threshold the cost of additions in this other structure would become a
>>>>>>>>>>>> constant factor, offering nice algorithmic complexity guarantees for large
>>>>>>>>>>>> n; however since sqrt(Integer.MAX_VALUE) is ~46,000, the maximum size of new
>>>>>>>>>>>> array allocations would have to be unrealistically small (assuming linear
>>>>>>>>>>>> cost for allocation) for this to be the case. It would still be nice to have
>>>>>>>>>>>> a data structure that avoids needing to copy data with each grow, whilst
>>>>>>>>>>>> still maintaining good memory performance.
>>>>>>>>>>>>
>>>>>>>>>>>> That *all* being said, I had been going by your javadoc and
>>>>>>>>>>>> emails to ascertain the behaviour of this class, as I couldn't locate a free
>>>>>>>>>>>> copy of [Brodnik99resizablearrays], and it seems this was a bad idea; as the
>>>>>>>>>>>> sqrt(n) cost appears to be associated with growing the backing array, rather
>>>>>>>>>>>> than with what I assumed to be copying data between arraylets, and it seems
>>>>>>>>>>>> this cost is pretty optimal. That will teach me to post to a list without
>>>>>>>>>>>> getting my facts straight first. The interesting thing is simply that the
>>>>>>>>>>>> constant factor for this implementation still seems to be quite high,
>>>>>>>>>>>> although perhaps that is simply because I was not benchmarking sufficiently
>>>>>>>>>>>> large values of n.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 15 April 2010 12:12, Kevin L. Stern <kevin.l.stern at gmail.com
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unless I am misreading your post, this now has a very similar
>>>>>>>>>>>>> feel to the first data structure that I posted to the list.  Martin Buchholz
>>>>>>>>>>>>> then pointed out that we can incorporate the ideas from
>>>>>>>>>>>>> [Brodnik99resizablearrays] and reap additional benefits.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Apr 15, 2010 at 4:07 AM, Benedict Elliott Smith <
>>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, as I was going to bed last night I realised I had not
>>>>>>>>>>>>>> fully addressed the problem that was originally being visited; only reduced
>>>>>>>>>>>>>> the constant factor for addition to the end of the list. A trivial
>>>>>>>>>>>>>> modification fixes that, however; same scheme but up to some maximum
>>>>>>>>>>>>>> arraylet size (of a power of 2), after which the array is increased in size
>>>>>>>>>>>>>> linearly. Performance doesn't seem to have been affected appreciably,
>>>>>>>>>>>>>> although not been exhaustive in the benchmarking:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 10 items inserts versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>>>> 10 items inserts Chunked / ExpArray = 0.99
>>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.15, ExpArray=1.16
>>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 0.99
>>>>>>>>>>>>>> 100 items inserts versus ArrayList: Chunked=1.24,
>>>>>>>>>>>>>> ExpArray=1.01
>>>>>>>>>>>>>> 100 items inserts Chunked / ExpArray = 1.23
>>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.24, ExpArray=1.01
>>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.23
>>>>>>>>>>>>>> 1000 items inserts versus ArrayList: Chunked=1.22,
>>>>>>>>>>>>>> ExpArray=1.03
>>>>>>>>>>>>>> 1000 items inserts Chunked / ExpArray = 1.19
>>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.19
>>>>>>>>>>>>>> 10000 items inserts versus ArrayList: Chunked=1.22,
>>>>>>>>>>>>>> ExpArray=1.03
>>>>>>>>>>>>>> 10000 items inserts Chunked / ExpArray = 1.18
>>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.22, ExpArray=1.03
>>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.18
>>>>>>>>>>>>>> 100000 items inserts versus ArrayList: Chunked=0.82,
>>>>>>>>>>>>>> ExpArray=0.75
>>>>>>>>>>>>>> 100000 items inserts Chunked / ExpArray = 1.09
>>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=0.82, ExpArray=0.75
>>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.09
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The nice thing about this is that the maximum amount of wasted
>>>>>>>>>>>>>> memory is user configurable. Even with a low setting as above (65K)
>>>>>>>>>>>>>> performance seems pretty consistent.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Code for calculating index and array offset are pretty
>>>>>>>>>>>>>> straight forward; haven't given much thought to optimisations just yet:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> private final int indexFor(int a, int i) {
>>>>>>>>>>>>>> return 1 + i - (a > maxArrayIndex ? (1 + a - maxArrayIndex) <<
>>>>>>>>>>>>>> maxArraySizeShift : 1 << a) ;
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>> private final int arrayFor(int i) {
>>>>>>>>>>>>>> return i >= (maxArraySize << 1) ? (i + 1 >>>
>>>>>>>>>>>>>> maxArraySizeShift) + maxArrayIndex - 1 : 31 - Integer.numberOfLeadingZeros(i
>>>>>>>>>>>>>> + 1) ;
>>>>>>>>>>>>>>  }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regarding the double list idea - yes, I agree, I certainly
>>>>>>>>>>>>>> didn't think that one through fully!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 15 April 2010 02:44, Kevin L. Stern <
>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Benedict,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Like you, I am relatively new to this mailing list; I am also
>>>>>>>>>>>>>>> trying to tread lightly so as not to step on any toes.  That being said, I
>>>>>>>>>>>>>>> think that I can offer a response to your inquiry.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding:  "The idea is to simply double the new array size
>>>>>>>>>>>>>>> each time a new array needs to be allocated"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems this would not address the desire of offering an
>>>>>>>>>>>>>>> alternative to the allocation of a large backing array for ArrayList (your
>>>>>>>>>>>>>>> largest backing array could still reach a size of 1/2 * Integer.MAX_VALUE)
>>>>>>>>>>>>>>> and would not address the desire of wasting the (asymptotically) minimum
>>>>>>>>>>>>>>> amount of memory in the worst case while maintaining O(1) amortized time
>>>>>>>>>>>>>>> bounds.  The data structure described in [Brodnik99resizablearrays] has a
>>>>>>>>>>>>>>> maximum backing array size of sqrt(n) and caps wasted memory at sqrt(n).
>>>>>>>>>>>>>>> What advantage over ArrayList do you see in your data structure?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regarding:  "Also, with regard to a Deque implementation, it
>>>>>>>>>>>>>>> seems that the simplest solution would be to simply have two lists, with one
>>>>>>>>>>>>>>> accepting inserts for near the beginning and being ordered in reverse whilst
>>>>>>>>>>>>>>> the other accepted inserts for near to the end."
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What happens with your structure when you add n elements and
>>>>>>>>>>>>>>> then remove element 0 n times?  I think that once you work out all the kinks
>>>>>>>>>>>>>>> you'll end up with the two stacks approach, which is mentioned in
>>>>>>>>>>>>>>> [Brodnik99resizablearrays] and which I mentioned in an earlier email, or
>>>>>>>>>>>>>>> you'll end up with the circular list approach, which is not friendly to O(1)
>>>>>>>>>>>>>>> amortized time bounds in a data structure that resizes more often than O(n)
>>>>>>>>>>>>>>> due to the 'unshift' to the front = 0 position.  I think the best approach
>>>>>>>>>>>>>>> is the one mentioned in [Brodnik99resizablearrays], which is the approach
>>>>>>>>>>>>>>> that I am currently working on.  Incidentally, this approach also provides
>>>>>>>>>>>>>>> for a much improved index unpacking procedure using only bit shifts and bit
>>>>>>>>>>>>>>> masks, although it is at the expense of (O(1)) additional work during
>>>>>>>>>>>>>>> resize.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Apr 14, 2010 at 4:42 PM, Benedict Elliott Smith <
>>>>>>>>>>>>>>> lists at laerad.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I hope you don't consider it rude to involve myself in this
>>>>>>>>>>>>>>>> conversation towards the end - I joined the mailing list only recently.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure if this offers a huge amount to the discussion,
>>>>>>>>>>>>>>>> but I have tinkered with a "chunked" array list which seems to offer better
>>>>>>>>>>>>>>>> time performance in general at the cost of greater (worst case) memory
>>>>>>>>>>>>>>>> utilisation. It is easier to understand IMHO as well, although this is not
>>>>>>>>>>>>>>>> necessarily a great benefit here. It turns out the idea is very similar to
>>>>>>>>>>>>>>>> the one implemented already by Kevin, though; but perhaps simpler. The idea
>>>>>>>>>>>>>>>> is to simply double the new array size each time a new array needs to be
>>>>>>>>>>>>>>>> allocated, or in effect allocate an array that is the size of all existing
>>>>>>>>>>>>>>>> arrays put together. With this scheme the calculation for array and offset
>>>>>>>>>>>>>>>> are really very straight forward ( floor(log(i)) and 1 + i -
>>>>>>>>>>>>>>>> 2^floor(log(i))) ). Memory utilisation is the same as for ArrayList, but
>>>>>>>>>>>>>>>> obviously inserts at the end are much quicker.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have prototyped the data structure this evening and
>>>>>>>>>>>>>>>> benchmarked additions at the end of the list, for which the performance is
>>>>>>>>>>>>>>>> pretty impressive.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Some random statistics for addition only on the client JVM
>>>>>>>>>>>>>>>> (I have quickly dubbed my implementation ExpArrayList)
>>>>>>>>>>>>>>>> All statistics were run in two rounds with ~1000 runs per
>>>>>>>>>>>>>>>> round per statistic per list, and the second round results were used.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.14, ExpArray=1.02
>>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.20, ExpArray=0.82
>>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.45
>>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.03, ExpArray=0.51
>>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.02
>>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.88, ExpArray=0.49
>>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.79
>>>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.32, ExpArray=0.20
>>>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.64
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and server JVM:
>>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=1.00, ExpArray=1.16
>>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 0.86
>>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=1.29, ExpArray=0.96
>>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 1.34
>>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=1.16, ExpArray=0.92
>>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 1.27
>>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=0.93, ExpArray=0.84
>>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 1.12
>>>>>>>>>>>>>>>> 100000 items versus ArrayList: Chunked=0.71, ExpArray=0.65
>>>>>>>>>>>>>>>> 100000 items Chunked / ExpArray = 1.10
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Interestingly insertion at the beginning of the list appears
>>>>>>>>>>>>>>>> to be quicker with ExpArrayList, at least on the server JVM, whereas I would
>>>>>>>>>>>>>>>> have expected them to be fairly close.
>>>>>>>>>>>>>>>> Amazingly ExpArrayList is faster even than ArrayList for
>>>>>>>>>>>>>>>> insertion at the beginning of large lists, which I haven't yet tried to
>>>>>>>>>>>>>>>> understand. Insertion in the middle is similar.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 10 items versus ArrayList: Chunked=9.82, ExpArray=3.80
>>>>>>>>>>>>>>>> 10 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>>>> 100 items versus ArrayList: Chunked=7.30, ExpArray=3.41
>>>>>>>>>>>>>>>> 100 items Chunked / ExpArray = 2.14
>>>>>>>>>>>>>>>> 1000 items versus ArrayList: Chunked=2.83, ExpArray=1.09
>>>>>>>>>>>>>>>> 1000 items Chunked / ExpArray = 2.59
>>>>>>>>>>>>>>>> 10000 items versus ArrayList: Chunked=1.56, ExpArray=0.72
>>>>>>>>>>>>>>>> 10000 items Chunked / ExpArray = 2.16
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Finally, there are promising results for get() from the
>>>>>>>>>>>>>>>> ExpArrayList as well (server JVM), again somehow beating ArrayList for
>>>>>>>>>>>>>>>> larger lists:
>>>>>>>>>>>>>>>> 10 items get versus ArrayList: Chunked=1.27, ExpArray=1.16
>>>>>>>>>>>>>>>> 10 items get Chunked / ExpArray = 1.10
>>>>>>>>>>>>>>>> 100 items get versus ArrayList: Chunked=1.45, ExpArray=1.17
>>>>>>>>>>>>>>>> 100 items get Chunked / ExpArray = 1.25
>>>>>>>>>>>>>>>> 1000 items get versus ArrayList: Chunked=1.42, ExpArray=1.07
>>>>>>>>>>>>>>>> 1000 items get Chunked / ExpArray = 1.33
>>>>>>>>>>>>>>>> 10000 items get versus ArrayList: Chunked=1.26,
>>>>>>>>>>>>>>>> ExpArray=1.02
>>>>>>>>>>>>>>>> 10000 items get Chunked / ExpArray = 1.24
>>>>>>>>>>>>>>>> 100000 items get versus ArrayList: Chunked=1.05,
>>>>>>>>>>>>>>>> ExpArray=0.86
>>>>>>>>>>>>>>>> 100000 items get Chunked / ExpArray = 1.22
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm willing to explore this further but I'm not sure how
>>>>>>>>>>>>>>>> desirable that is, given that Kevin's data structure appears to perform
>>>>>>>>>>>>>>>> pretty well already wrt to CPU time, and better wrt to memory utilisation,
>>>>>>>>>>>>>>>> and in effect this mostly changes only the function to determine which array
>>>>>>>>>>>>>>>> to use, not the body of the implementation. Let me know if you would like a
>>>>>>>>>>>>>>>> copy of the source code and I will find somewhere to upload it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, with regard to a Deque implementation, it seems that
>>>>>>>>>>>>>>>> the simplest solution would be to simply have two lists, with one accepting
>>>>>>>>>>>>>>>> inserts for near the beginning and being ordered in reverse whilst the other
>>>>>>>>>>>>>>>> accepted inserts for near to the end. The only trick would be having the
>>>>>>>>>>>>>>>> list at the beginning support iteration in reverse order cheaply, but this
>>>>>>>>>>>>>>>> could easily be achieved by creating an extension of List with a
>>>>>>>>>>>>>>>> reverseIterator() method.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyway, not sure if this helped at all but fancied joining
>>>>>>>>>>>>>>>> in...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 14 April 2010 12:25, Joe Kearney <
>>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It implements List, as well as Deque. It is indeed based on
>>>>>>>>>>>>>>>>> ArrayDeque, with the added operations to implement list. It does so
>>>>>>>>>>>>>>>>> reasonably efficiently, moving the fewest elements possible on each
>>>>>>>>>>>>>>>>> operation, that is zero for the queue operations, at most n/2 for the rest
>>>>>>>>>>>>>>>>> and all of them for a backing array resize.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The idea is to get a replacement for arraylist that
>>>>>>>>>>>>>>>>> performs like arraydeque on remove(0). As a side effect, we should be able
>>>>>>>>>>>>>>>>> to get better performance on other operations by requiring fewer elements to
>>>>>>>>>>>>>>>>> be moved.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2010/4/14 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Joe,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I was referring to the ChunkedArrayList when I stated that
>>>>>>>>>>>>>>>>>> add does not amortize to constant time when the data structure employs the
>>>>>>>>>>>>>>>>>> circular list trick to achieve deque behavior; ChunkedArrayList potentially
>>>>>>>>>>>>>>>>>> resizes every n^(1/2) operations.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regarding your CircularArrayList, does it differ from
>>>>>>>>>>>>>>>>>> Java's ArrayDeque?  I took only a cursory look at it, so please understand
>>>>>>>>>>>>>>>>>> if I have missed your reason for creating CircularArrayList altogether.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:52 AM, Joe Kearney <
>>>>>>>>>>>>>>>>>> joe.j.kearney at googlemail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Kevin, Martin,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To add another discussion point, I've been writing a
>>>>>>>>>>>>>>>>>>> draft/proof-of-concept of retrofitting the List interface onto ArrayDeque.
>>>>>>>>>>>>>>>>>>> This works over the raw array, it doesn't use the fancier structures being
>>>>>>>>>>>>>>>>>>> discussed elsewhere on this list that deal with splitting huge arrays into
>>>>>>>>>>>>>>>>>>> arraylets, or that provide for O(1) insert in the middle.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://code.google.com/p/libjoe/source/browse/trunk/src/joe/collect/CircularArrayList.java
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'd be interested if you have any comments in the context
>>>>>>>>>>>>>>>>>>> of this discussion. The code is not entirely ready yet, a couple of tests
>>>>>>>>>>>>>>>>>>> fail (6/789) because of a corner case I haven't nailed yet, but the idea is
>>>>>>>>>>>>>>>>>>> there at least. I'd like to add array shrinking later, when the size dips
>>>>>>>>>>>>>>>>>>> below capacity*0.4 perhaps, to avoid flickering up and down around...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Tests show performance to be close to ArrayList for the
>>>>>>>>>>>>>>>>>>> O(1) operations. Timings for indexed reads and writes showed
>>>>>>>>>>>>>>>>>>> no discernible difference between implementations last time I ran the
>>>>>>>>>>>>>>>>>>> tests. I don't understand at the moment why the iterator add at index
>>>>>>>>>>>>>>>>>>> size/3, size/2 perform 30% slower than ArrayList on smaller lists, nor the
>>>>>>>>>>>>>>>>>>> dodgy numbers for ArrayList.insert(5), I'll look at this soon.  Those
>>>>>>>>>>>>>>>>>>> operations that become O(1) in a circular implementation (that are
>>>>>>>>>>>>>>>>>>> implemented and tested here) are faster than in ArrayList. Insert/remove in
>>>>>>>>>>>>>>>>>>> the middle are somewhat faster than ArrayList because we only have to copy
>>>>>>>>>>>>>>>>>>> at most half of the elements, except when resizing the array.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Kevin, I don't fully understand your point about not
>>>>>>>>>>>>>>>>>>> amortizing to O(1). Certainly that's true for insert not at head or tail.
>>>>>>>>>>>>>>>>>>> Otherwise this implementation only moves array elements to the front on an
>>>>>>>>>>>>>>>>>>> array resize operation which happens every O(ln n) operations at most, if we
>>>>>>>>>>>>>>>>>>> do lots of adds, maybe a little more if we add array shrinking too.  This is
>>>>>>>>>>>>>>>>>>> the same as ArrayList. Are you just referring to the add-in-the-middle case?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Some performance results below, code for these is in the
>>>>>>>>>>>>>>>>>>> repository above too. This was the second run, after a warmup.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Joe
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>>>>>>>> CircularArrayList ------------------------------------------------
>>>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>>>      10          20          67          70         125
>>>>>>>>>>>>>>>>>>>         102          90         240         191         138
>>>>>>>>>>>>>>>>>>>    100          19          67          70         166
>>>>>>>>>>>>>>>>>>>       138          94         230         194         118
>>>>>>>>>>>>>>>>>>>   1000          28          64          67         681
>>>>>>>>>>>>>>>>>>>       538          91         324         382         119
>>>>>>>>>>>>>>>>>>>   10000          30          65          67        5884
>>>>>>>>>>>>>>>>>>>        4425          94        1296        2330         124
>>>>>>>>>>>>>>>>>>> ----------------------------------------------------
>>>>>>>>>>>>>>>>>>> ArrayList ----------------------------------------------------
>>>>>>>>>>>>>>>>>>>   size         add         get         set   iterAdd/3
>>>>>>>>>>>>>>>>>>> iterAdd/2   insert(5)   removeRnd   removeMid   remove(0)
>>>>>>>>>>>>>>>>>>>      10          23          68          70         100
>>>>>>>>>>>>>>>>>>>          69       32913         162         130         105
>>>>>>>>>>>>>>>>>>>    100          20          67          70         129
>>>>>>>>>>>>>>>>>>>       104       21944         169         134         135
>>>>>>>>>>>>>>>>>>>   1000          29          63          67         651
>>>>>>>>>>>>>>>>>>>       506        9602         364         333         526
>>>>>>>>>>>>>>>>>>>   10000          30          63          66        5878
>>>>>>>>>>>>>>>>>>>        4414        9947        2312        2280        4437
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2010/4/13 Kevin L. Stern <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I had intended to address your request for absolute O(1)
>>>>>>>>>>>>>>>>>>>> operations in the previous email.  The approach to achieving this suggested
>>>>>>>>>>>>>>>>>>>> in [Brodnik99resizablearrays] is tantamount to making ArrayList operations
>>>>>>>>>>>>>>>>>>>> absolute O(1) by keeping around an array of size (3/2)*n and filling it with
>>>>>>>>>>>>>>>>>>>> a constant number of entries from the main array each time add is called.
>>>>>>>>>>>>>>>>>>>> Although this distributes the work done during a resize across the n
>>>>>>>>>>>>>>>>>>>> operations required to enter a resize-required state, it is at the expense
>>>>>>>>>>>>>>>>>>>> of additional memory usage and slower add operations.  My thought is that
>>>>>>>>>>>>>>>>>>>> this would be a fine approach for a real-time application that requires hard
>>>>>>>>>>>>>>>>>>>> guarantees on performance but would be a liability in so many Java
>>>>>>>>>>>>>>>>>>>> applications that do not require these hard guarantees.  I look forward to
>>>>>>>>>>>>>>>>>>>> hearing your thoughts on the matter, though.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Apr 13, 2010 at 6:18 AM, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It's interesting to note that the old circular list
>>>>>>>>>>>>>>>>>>>>> trick will not suffice to turn this data structure into a deque since we
>>>>>>>>>>>>>>>>>>>>> might be copying all n elements back to the front = 0 position every n^(1/2)
>>>>>>>>>>>>>>>>>>>>> operations (add wouldn't amortize to O(1)).  We could use the old two stacks
>>>>>>>>>>>>>>>>>>>>> trick (push elements onto one stack, flip (the bottom) half (of) the
>>>>>>>>>>>>>>>>>>>>> elements to the 'other' stack when the 'other' stack becomes empty),
>>>>>>>>>>>>>>>>>>>>> mentioned in [Brodnik99resizablearrays], but I find this to be a bit CS
>>>>>>>>>>>>>>>>>>>>> 101.  In [Brodnik99resizablearrays] the authors suggest a method for making
>>>>>>>>>>>>>>>>>>>>> all blocks roughly the same size, allowing us to expand/shrink capacity at
>>>>>>>>>>>>>>>>>>>>> the beginning or the end; this is the approach that I will take to create a
>>>>>>>>>>>>>>>>>>>>> deque.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The FAQ for the Sun Contributor Agreement Q3 (
>>>>>>>>>>>>>>>>>>>>> http://www.sun.com/software/opensource/contributor_agreement.jsp#sa_3)
>>>>>>>>>>>>>>>>>>>>> indicates that one should check with the project to determine where the SCA
>>>>>>>>>>>>>>>>>>>>> should be sent.  Do you know where I would find this information?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Kevin
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> @MISC{Brodnik99resizablearrays,
>>>>>>>>>>>>>>>>>>>>>     author = {Andrej Brodnik and Svante Carlsson and
>>>>>>>>>>>>>>>>>>>>> Erik D. Demaine and J. Ian Munro and Robert Sedgewick},
>>>>>>>>>>>>>>>>>>>>>     title = {Resizable Arrays in Optimal Time and
>>>>>>>>>>>>>>>>>>>>> Space},
>>>>>>>>>>>>>>>>>>>>>     year = {1999}
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 4:17 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for your continuing work on this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I like the test results, and agree with your analysis.
>>>>>>>>>>>>>>>>>>>>>> I'm especially happy that you're beating
>>>>>>>>>>>>>>>>>>>>>> ArrayList at some operations.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'd like to see O(1) addition at the beginning,
>>>>>>>>>>>>>>>>>>>>>> implement both List and Deque (I regret
>>>>>>>>>>>>>>>>>>>>>> our not having done this with ArrayDeque).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> An additional property that would be nice to
>>>>>>>>>>>>>>>>>>>>>> have (but don't try too hard)
>>>>>>>>>>>>>>>>>>>>>> is to provide some kind of real-time
>>>>>>>>>>>>>>>>>>>>>> guarantees on the cost of an individual operation,
>>>>>>>>>>>>>>>>>>>>>> not just amortized time.  E.g. ArrayList.add
>>>>>>>>>>>>>>>>>>>>>> is worst-case O(n), making it unsuitable for use
>>>>>>>>>>>>>>>>>>>>>> in some real-time applications.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I will help get your changes into the obvious
>>>>>>>>>>>>>>>>>>>>>> software distributions.  I assume you're happy
>>>>>>>>>>>>>>>>>>>>>> with having this class included in any of
>>>>>>>>>>>>>>>>>>>>>> Doug Lea's jsr166, guava-libraries, or the JDK itself.
>>>>>>>>>>>>>>>>>>>>>> You should sign a Sun contributor agreement,
>>>>>>>>>>>>>>>>>>>>>> or whatever the Oracle equivalent is,
>>>>>>>>>>>>>>>>>>>>>> if you have not done so yet.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Doug Lea likes public domain,
>>>>>>>>>>>>>>>>>>>>>> guava-libraries likes the Apache license.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We should get various people a chance to give
>>>>>>>>>>>>>>>>>>>>>> a thumbs up on the design of this class -
>>>>>>>>>>>>>>>>>>>>>> Doug Lea, Josh Bloch.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Martin
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 11, 2010 at 09:32, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> > Hello Martin,
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > I spent some time this weekend trying to bring out
>>>>>>>>>>>>>>>>>>>>>> bugs in the
>>>>>>>>>>>>>>>>>>>>>> > implementation; I believe the latest version to be
>>>>>>>>>>>>>>>>>>>>>> in decent shape.  I have
>>>>>>>>>>>>>>>>>>>>>> > also gathered some data on the performance of
>>>>>>>>>>>>>>>>>>>>>> ChunkedArrayList over
>>>>>>>>>>>>>>>>>>>>>> > ArrayList using the latest 1.6 JDK, which I've
>>>>>>>>>>>>>>>>>>>>>> included below (note that the
>>>>>>>>>>>>>>>>>>>>>> > numbers represent the time spent performing the
>>>>>>>>>>>>>>>>>>>>>> specified operation with
>>>>>>>>>>>>>>>>>>>>>> > ChunkedArrayList over the time spent with ArrayList,
>>>>>>>>>>>>>>>>>>>>>> so 1.00 indicates
>>>>>>>>>>>>>>>>>>>>>> > equivalent performance, < 1.00 indicates that
>>>>>>>>>>>>>>>>>>>>>> ChunkedArrayList is less
>>>>>>>>>>>>>>>>>>>>>> > costly and > 1.00 indicates that ArrayList is less
>>>>>>>>>>>>>>>>>>>>>> costly).  I've noticed
>>>>>>>>>>>>>>>>>>>>>> > relatively significant variability in a few of the
>>>>>>>>>>>>>>>>>>>>>> numbers when I switch
>>>>>>>>>>>>>>>>>>>>>> > hardware; though, these data do seem to represent
>>>>>>>>>>>>>>>>>>>>>> rough performance
>>>>>>>>>>>>>>>>>>>>>> > expectations.  For my test I generated x elements
>>>>>>>>>>>>>>>>>>>>>> and then timed the process
>>>>>>>>>>>>>>>>>>>>>> > of adding them to ArrayList/ChunkedArrayList, then I
>>>>>>>>>>>>>>>>>>>>>> performed a get
>>>>>>>>>>>>>>>>>>>>>> > operation on each for indices 0 through x-1 and
>>>>>>>>>>>>>>>>>>>>>> finally I used the iterator
>>>>>>>>>>>>>>>>>>>>>> > mechanism to retrieve the first through xth element
>>>>>>>>>>>>>>>>>>>>>> (of course, I performed
>>>>>>>>>>>>>>>>>>>>>> > each of these operations multiple times throwing
>>>>>>>>>>>>>>>>>>>>>> away the timing for the
>>>>>>>>>>>>>>>>>>>>>> > first few iterations to warm up the JVM).
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Regarding the question of whether or not this
>>>>>>>>>>>>>>>>>>>>>> belongs in java.util, I would
>>>>>>>>>>>>>>>>>>>>>> > suggest that if it is desirable from a GC point of
>>>>>>>>>>>>>>>>>>>>>> view to eliminate the
>>>>>>>>>>>>>>>>>>>>>> > large backing array from ArrayList then your
>>>>>>>>>>>>>>>>>>>>>> suggestion of achieving this by
>>>>>>>>>>>>>>>>>>>>>> > way of a data structure that is both time and space
>>>>>>>>>>>>>>>>>>>>>> optimal is a
>>>>>>>>>>>>>>>>>>>>>> > particularly elegant solution as it not only
>>>>>>>>>>>>>>>>>>>>>> guarantees that no backing
>>>>>>>>>>>>>>>>>>>>>> > array will be larger than sqrt(n) elements but it
>>>>>>>>>>>>>>>>>>>>>> also provides dynamic
>>>>>>>>>>>>>>>>>>>>>> > shrinking behavior, has less maximum memory overhead
>>>>>>>>>>>>>>>>>>>>>> than ArrayList, and
>>>>>>>>>>>>>>>>>>>>>> > copies (asymptotically) fewer elements during a
>>>>>>>>>>>>>>>>>>>>>> resize than ArrayList.  Of
>>>>>>>>>>>>>>>>>>>>>> > course, this data structure does not do everything
>>>>>>>>>>>>>>>>>>>>>> better than ArrayList; in
>>>>>>>>>>>>>>>>>>>>>> > particular, indexed access is more costly, due to
>>>>>>>>>>>>>>>>>>>>>> the required decomposition
>>>>>>>>>>>>>>>>>>>>>> > of the index into backing array index and offset and
>>>>>>>>>>>>>>>>>>>>>> the additional memory
>>>>>>>>>>>>>>>>>>>>>> > indirection, and insertion-at-an-index is more
>>>>>>>>>>>>>>>>>>>>>> costly, due to the multiple
>>>>>>>>>>>>>>>>>>>>>> > array copies necessary to complete the shift.  That
>>>>>>>>>>>>>>>>>>>>>> being said, I think that
>>>>>>>>>>>>>>>>>>>>>> > the additional cost of indexed access is partially
>>>>>>>>>>>>>>>>>>>>>> mitigated by the
>>>>>>>>>>>>>>>>>>>>>> > availability of iterator and listIterator, whose
>>>>>>>>>>>>>>>>>>>>>> implementations do not use
>>>>>>>>>>>>>>>>>>>>>> > the index decomposition procedure, and the
>>>>>>>>>>>>>>>>>>>>>> additional cost of
>>>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is partially mitigated by the
>>>>>>>>>>>>>>>>>>>>>> fact that
>>>>>>>>>>>>>>>>>>>>>> > insertion-at-an-index is already an undesirable
>>>>>>>>>>>>>>>>>>>>>> operation on ArrayList due
>>>>>>>>>>>>>>>>>>>>>> > to its linear time complexity.
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Kevin
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > 1000000 elements:
>>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.30
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.80
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.52
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.81
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.87
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.31
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > 100000 elements:
>>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.86
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.48
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.96
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.89
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.68
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > 10000 elements:
>>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 1.04
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.33
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.53
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.97
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.45
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 2.52
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > 1000 elements:
>>>>>>>>>>>>>>>>>>>>>> > Client JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.99
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 2.27
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 0.54
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > Server JVM:
>>>>>>>>>>>>>>>>>>>>>> > Add to ChunkedArrayList over ArrayList: 0.84
>>>>>>>>>>>>>>>>>>>>>> > Indexed access ChunkedArrayList over ArrayList: 1.23
>>>>>>>>>>>>>>>>>>>>>> > Iterator ChunkedArrayList over ArrayList: 1.11
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> > On Fri, Apr 9, 2010 at 7:42 PM, Martin Buchholz <
>>>>>>>>>>>>>>>>>>>>>> martinrb at google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> My feeling on whether to support O(1) at both ends
>>>>>>>>>>>>>>>>>>>>>> >> is that any flavor of this that ends up in the JDK
>>>>>>>>>>>>>>>>>>>>>> eventually
>>>>>>>>>>>>>>>>>>>>>> >> should really do this.  My idea is that we can
>>>>>>>>>>>>>>>>>>>>>> >> wholeheartedly recommend this collection class
>>>>>>>>>>>>>>>>>>>>>> >> for overall good behavior without any of the
>>>>>>>>>>>>>>>>>>>>>> surprising
>>>>>>>>>>>>>>>>>>>>>> >> performance traps of existing collection classes.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> But for the preliminary version, it makes sense to
>>>>>>>>>>>>>>>>>>>>>> >> support only O(1) at one end, if it simplifies the
>>>>>>>>>>>>>>>>>>>>>> >> implementation.  Random access will of course
>>>>>>>>>>>>>>>>>>>>>> >> be worse than ArrayList, but by how much?
>>>>>>>>>>>>>>>>>>>>>> >> We can do some benchmarking and look for
>>>>>>>>>>>>>>>>>>>>>> >> micro-optimizations now.
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Kevin, what is you own personal feeling?
>>>>>>>>>>>>>>>>>>>>>> >> Is the algorithm correct, and efficient enough?
>>>>>>>>>>>>>>>>>>>>>> >> Do you think your new collection belongs in
>>>>>>>>>>>>>>>>>>>>>> java.util?
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> Martin
>>>>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>>>>> >> On Sun, Apr 4, 2010 at 04:12, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> wrote:
>>>>>>>>>>>>>>>>>>>>>> >> > The data structure is available at the second
>>>>>>>>>>>>>>>>>>>>>> link that I originally
>>>>>>>>>>>>>>>>>>>>>> >> > provided (once again, it is
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> https://docs.google.com/Doc?docid=0Aabrz3MPBDdhZGdrbnEzejdfM2M3am5wM2Mz&hl=en
>>>>>>>>>>>>>>>>>>>>>> ).
>>>>>>>>>>>>>>>>>>>>>> >> > This does not have O(1) time insertion at the
>>>>>>>>>>>>>>>>>>>>>> front as yet as it was
>>>>>>>>>>>>>>>>>>>>>> >> > unclear
>>>>>>>>>>>>>>>>>>>>>> >> > to me whether or not it was agreed upon:
>>>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>>>> >> > From: Osvaldo Doederlein <opinali at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> > Date: Mon, Mar 29, 2010 at 10:08 AM
>>>>>>>>>>>>>>>>>>>>>> >> > Subject: Re: A List implementation backed by
>>>>>>>>>>>>>>>>>>>>>> multiple small arrays
>>>>>>>>>>>>>>>>>>>>>> >> > rather
>>>>>>>>>>>>>>>>>>>>>> >> > than the traditional single large array.
>>>>>>>>>>>>>>>>>>>>>> >> > To: Martin Buchholz <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>>> >> > Cc: "Kevin L. Stern" <kevin.l.stern at gmail.com>,
>>>>>>>>>>>>>>>>>>>>>> >> > core-libs-dev at openjdk.java.net
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > Initially, it would be good enough to replace
>>>>>>>>>>>>>>>>>>>>>> only java.util.ArrayList
>>>>>>>>>>>>>>>>>>>>>> >> > with
>>>>>>>>>>>>>>>>>>>>>> >> > minimal overhead. ArrayList does not support
>>>>>>>>>>>>>>>>>>>>>> efficient add-at-front or
>>>>>>>>>>>>>>>>>>>>>> >> > other
>>>>>>>>>>>>>>>>>>>>>> >> > enhancements of ArrayDeque; but ArrayList is
>>>>>>>>>>>>>>>>>>>>>> still a much more important
>>>>>>>>>>>>>>>>>>>>>> >> > and
>>>>>>>>>>>>>>>>>>>>>> >> > popular collection, it's the primary "straight
>>>>>>>>>>>>>>>>>>>>>> replacement for primitive
>>>>>>>>>>>>>>>>>>>>>> >> > arrrays" and I guess it should continue with that
>>>>>>>>>>>>>>>>>>>>>> role.
>>>>>>>>>>>>>>>>>>>>>> >> > _________________
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > As a disclaimer, I'm still tinkering with this so
>>>>>>>>>>>>>>>>>>>>>> I'll be updating the
>>>>>>>>>>>>>>>>>>>>>> >> > document at the provided link as I find
>>>>>>>>>>>>>>>>>>>>>> improvements.
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > Thoughts?
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > Thanks,
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > Kevin
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> > On Thu, Apr 1, 2010 at 10:28 PM, Martin Buchholz
>>>>>>>>>>>>>>>>>>>>>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>>> >> > wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> Hi Kevin,
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> You're probably the only one on this list who
>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>> >> >> seriously read the paper.  It is not surprising
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> >> >> taking a research paper into production would
>>>>>>>>>>>>>>>>>>>>>> >> >> discover bugs - the research never had to
>>>>>>>>>>>>>>>>>>>>>> undergo
>>>>>>>>>>>>>>>>>>>>>> >> >> rigorous testing.  (I like the Java culture of
>>>>>>>>>>>>>>>>>>>>>> >> >> combining spec + implementation + test suite)
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> I suggest you ask the authors directly about the
>>>>>>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>>>>>>> >> >> They would probably also be interested to hear
>>>>>>>>>>>>>>>>>>>>>> >> >> about your implementation.
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> Are you aware of Integer.numberOfLeadingZeros?
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros(int)<http://download.java.net/jdk7/docs/api/java/lang/Integer.html#numberOfLeadingZeros%28int%29>
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> Martin
>>>>>>>>>>>>>>>>>>>>>> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> On Wed, Mar 31, 2010 at 19:34, Kevin L. Stern <
>>>>>>>>>>>>>>>>>>>>>> kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> >> wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >> > I'm almost convinced now that the paper is
>>>>>>>>>>>>>>>>>>>>>> incorrect.  The code below
>>>>>>>>>>>>>>>>>>>>>> >> >> > gives
>>>>>>>>>>>>>>>>>>>>>> >> >> > me the appropriate index into the index array
>>>>>>>>>>>>>>>>>>>>>> and the offset into the
>>>>>>>>>>>>>>>>>>>>>> >> >> > data
>>>>>>>>>>>>>>>>>>>>>> >> >> > block.  That being said, remember when I
>>>>>>>>>>>>>>>>>>>>>> mentioned that this will
>>>>>>>>>>>>>>>>>>>>>> >> >> > include a
>>>>>>>>>>>>>>>>>>>>>> >> >> > bit more work to access an element than a
>>>>>>>>>>>>>>>>>>>>>> simple bit shift and a bit
>>>>>>>>>>>>>>>>>>>>>> >> >> > mask?
>>>>>>>>>>>>>>>>>>>>>> >> >> > Well this is more than a bit more - we'll be
>>>>>>>>>>>>>>>>>>>>>> doing this each time an
>>>>>>>>>>>>>>>>>>>>>> >> >> > index
>>>>>>>>>>>>>>>>>>>>>> >> >> > is requested.  I'll spend some time trying to
>>>>>>>>>>>>>>>>>>>>>> twiddle the bits to see
>>>>>>>>>>>>>>>>>>>>>> >> >> > if
>>>>>>>>>>>>>>>>>>>>>> >> >> > I
>>>>>>>>>>>>>>>>>>>>>> >> >> > can eliminate/combine some of the operations.
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >> >         for (int r = 1; r < 33; r++) {
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int k = lg(r);
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int floorKO2 = k >> 1;
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int powFloorKO2 = (1 << floorKO2);
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int p = ((1 << floorKO2) - 1) <<
>>>>>>>>>>>>>>>>>>>>>> 1;
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int ceilKO2;
>>>>>>>>>>>>>>>>>>>>>> >> >> >             if ((k & 1) == 1) {
>>>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2 + 1;
>>>>>>>>>>>>>>>>>>>>>> >> >> >                 p += powFloorKO2;
>>>>>>>>>>>>>>>>>>>>>> >> >> >             } else {
>>>>>>>>>>>>>>>>>>>>>> >> >> >                 ceilKO2 = floorKO2;
>>>>>>>>>>>>>>>>>>>>>> >> >> >             }
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int e = r & ((1 << ceilKO2) - 1);
>>>>>>>>>>>>>>>>>>>>>> >> >> >             int b = (r >> ceilKO2) &
>>>>>>>>>>>>>>>>>>>>>> (powFloorKO2 - 1);
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >> >             System.out.println((r - 1) + " " +
>>>>>>>>>>>>>>>>>>>>>> (p + b) + " " + e);
>>>>>>>>>>>>>>>>>>>>>> >> >> >         }
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >> > Kevin
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >> > On Wed, Mar 31, 2010 at 7:08 PM, Kevin L.
>>>>>>>>>>>>>>>>>>>>>> Stern
>>>>>>>>>>>>>>>>>>>>>> >> >> > <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> >> > wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> >> I realize that 2 * (2^(k/2) - 1) only works
>>>>>>>>>>>>>>>>>>>>>> for even numbered
>>>>>>>>>>>>>>>>>>>>>> >> >> >> superblocks,
>>>>>>>>>>>>>>>>>>>>>> >> >> >> the odd numbered superblocks need an
>>>>>>>>>>>>>>>>>>>>>> additional term added (the
>>>>>>>>>>>>>>>>>>>>>> >> >> >> number
>>>>>>>>>>>>>>>>>>>>>> >> >> >> of
>>>>>>>>>>>>>>>>>>>>>> >> >> >> data blocks in SB_[k-1]) to jive with my
>>>>>>>>>>>>>>>>>>>>>> interpretation; anyhow, I
>>>>>>>>>>>>>>>>>>>>>> >> >> >> also
>>>>>>>>>>>>>>>>>>>>>> >> >> >> came
>>>>>>>>>>>>>>>>>>>>>> >> >> >> across an alternative characterization of
>>>>>>>>>>>>>>>>>>>>>> superblock in the paper
>>>>>>>>>>>>>>>>>>>>>> >> >> >> which
>>>>>>>>>>>>>>>>>>>>>> >> >> >> states that data blocks are grouped within a
>>>>>>>>>>>>>>>>>>>>>> superblock when they
>>>>>>>>>>>>>>>>>>>>>> >> >> >> are
>>>>>>>>>>>>>>>>>>>>>> >> >> >> the
>>>>>>>>>>>>>>>>>>>>>> >> >> >> same size - to me, though, that implies that
>>>>>>>>>>>>>>>>>>>>>> my example structure
>>>>>>>>>>>>>>>>>>>>>> >> >> >> below
>>>>>>>>>>>>>>>>>>>>>> >> >> >> would be
>>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_0: [1]
>>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_1: [2][2][2]
>>>>>>>>>>>>>>>>>>>>>> >> >> >> SB_2: [4][4][4][4][4][4]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> >> which seems to contradict my understanding of
>>>>>>>>>>>>>>>>>>>>>> (1) below.  I must be
>>>>>>>>>>>>>>>>>>>>>> >> >> >> reading this upside down.
>>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> >> On Wed, Mar 31, 2010 at 6:36 PM, Kevin L.
>>>>>>>>>>>>>>>>>>>>>> Stern
>>>>>>>>>>>>>>>>>>>>>> >> >> >> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> >> >> wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> What am I missing here?  In "Resizable
>>>>>>>>>>>>>>>>>>>>>> arrays in optimal time and
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> space"
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> the authors define their data structure with
>>>>>>>>>>>>>>>>>>>>>> the following
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> property:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> (1)  "When superblock SB_k is fully
>>>>>>>>>>>>>>>>>>>>>> allocated, it consists of
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> 2^(floor(k/2)) data blocks, each of size
>>>>>>>>>>>>>>>>>>>>>> 2^(ceil(k/2))."
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> Since the superblock is zero-based indexed
>>>>>>>>>>>>>>>>>>>>>> this implies the
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> following
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> structure:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_0: [1]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_1: [2]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_2: [2][2]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_3: [4][4]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_4: [4][4][4][4]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> [...]
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> Let's have a look at Algorithm 3, Locate(i),
>>>>>>>>>>>>>>>>>>>>>> with i = 3:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> r = 100 (the binary expansion of i + 1)
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> k = |r| - 1 = 2
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> p = 2^k - 1 = 3
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> What concerns me is their statement that p
>>>>>>>>>>>>>>>>>>>>>> represents "the number
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> of
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> data
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks in superblocks prior to SB_k."  There
>>>>>>>>>>>>>>>>>>>>>> are only two data
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> blocks
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> in
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> superblocks prior to SB_2, not three.  Given
>>>>>>>>>>>>>>>>>>>>>> (1) above, unless I'm
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpreting it, the number of data
>>>>>>>>>>>>>>>>>>>>>> blocks in superblocks prior
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> to
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> SB_k
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> should be:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> 2 * Sum[i=0->k/2-1] 2^i = 2 * (2^(k/2) - 1)
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> This, of course, seems to work out much
>>>>>>>>>>>>>>>>>>>>>> better in my example above,
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> giving the correct answer to my
>>>>>>>>>>>>>>>>>>>>>> interpretation of their data
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> structure, but
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> I have a hard time believing that this is
>>>>>>>>>>>>>>>>>>>>>> their mistake rather than
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> my
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> misinterpretation.
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> Thoughts?
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> Kevin
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> On Tue, Mar 30, 2010 at 5:20 PM, Martin
>>>>>>>>>>>>>>>>>>>>>> Buchholz
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> <martinrb at google.com>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>> wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> On Tue, Mar 30, 2010 at 04:25, Kevin L.
>>>>>>>>>>>>>>>>>>>>>> Stern
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> <kevin.l.stern at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Hi Martin,
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> >
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > Thanks much for your feedback.  The first
>>>>>>>>>>>>>>>>>>>>>> approach that comes to
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > mind
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > implement O(1) time front as well as rear
>>>>>>>>>>>>>>>>>>>>>> insertion is to create
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > a
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > cyclic
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > list structure with a front/rear pointer
>>>>>>>>>>>>>>>>>>>>>> - to insert at the
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > decrementing the front pointer (modulo
>>>>>>>>>>>>>>>>>>>>>> the size) and to insert
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > at
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > rear
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > requires incrementing the rear pointer
>>>>>>>>>>>>>>>>>>>>>> (modulo the size).  We
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > need
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > to
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > resize
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > when the two pointers bump into each
>>>>>>>>>>>>>>>>>>>>>> other.  Could you explain
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > more
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > about
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > your suggestion of introducing an
>>>>>>>>>>>>>>>>>>>>>> arraylet that is shared by the
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > front
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > and
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > the rear?
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> It was a half-baked idea - I don't know if
>>>>>>>>>>>>>>>>>>>>>> there's a way to turn
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> it
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> into
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> something useful.  I was thinking of the
>>>>>>>>>>>>>>>>>>>>>> ArrayDeque
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> implementation,
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> where all the elements live in a single
>>>>>>>>>>>>>>>>>>>>>> array.
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> >  It's not clear to me how that would help
>>>>>>>>>>>>>>>>>>>>>> and/or be a better
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > approach than the cyclic list.  Anyhow,
>>>>>>>>>>>>>>>>>>>>>> the paper that you
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > reference,
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > "Resizable arrays in optimal time and
>>>>>>>>>>>>>>>>>>>>>> space", gives a deque so
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > if
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > we
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > take
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> > that approach then the deque is
>>>>>>>>>>>>>>>>>>>>>> specified.
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> Technically, ArrayList also supports the
>>>>>>>>>>>>>>>>>>>>>> Deque operations -
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>> just not efficiently.
>>>>>>>>>>>>>>>>>>>>>> >> >> >>>
>>>>>>>>>>>>>>>>>>>>>> >> >> >>
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100425/8fa25e88/attachment.html>

From xueming.shen at oracle.com  Mon Apr 26 05:28:56 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Sun, 25 Apr 2010 22:28:56 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD3E7AC.2020801@oracle.com>
References: <4BD00250.3020206@oracle.com>
	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>
	<4BD3E7AC.2020801@oracle.com>
Message-ID: <4BD52498.2090006@oracle.com>


Can I assume we are all OK with at least the API part of the latest 
webrev/blenderrev of
the script support in j.l.Character and j.u.r.Pattern, including the 
j.l.Chareacter.getName().

http://cr.openjdk.java.net/~sherman/script/blenderrev.html
http://cr.openjdk.java.net/~sherman/script/webrev

Okutsu-san, Yuka, can one of you help review the corresponding CCC at
http://ccc.sfbay.sun.com/6945564?

This is for the j.l.Character part only. I'm still trying to figure out 
how to take over
the ownership of 4860714 in CCC system, we have a placeholder for this 
one in
CCC back to 2003.

Thanks,
-Sherman




Xueming Shen wrote:
> Martin Buchholz wrote:
>> Providing script support is obvious and non-controversial,
>> because other regex programming environments provide it.
>> Check that the behavior and syntax of the extension is
>> consistent with e.g. ICU, python, and especially perl
>> (5.12 just released!)
>>
>> http://perldoc.perl.org/perlunicode.html
>>   
>
> \p{propName=propValue} is the unicode "compound form", which is 
> supported in
> perl 5.12. It also has a variant type \p{propName:propValue}. It was 
> in my proposal,
> but I removed it the last minutes. Two forms (\p{In/IsProp} and 
> \p{propName=propValue}
> should be good enough for now. Three is a little too much. We can 
> always add it
> in later, if desirable.
>
> \p{IsScript}, \p{Isgc}, \p{InBlock} are perl compatible as well.
>
>> I would add some documentation to the three special script values;
>> their meaning is not obvious.
>>
>>   
> I think it might be better to justt leave the detailed explain doc to 
> the TR#24. The "script"
> here in j.l.Character serves only the purpose of id, the API here 
> should not be the place
> to explain "what they really are".
>
>> For implementation, the character matching problem is in general
>> equivalent to the problem of compiling a switch statement, which is
>> known to be non-trivial.  Guava contains a CharMatcher class that
>> tries to solve related problems.
>>
>> http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/CharMatcher.html 
>>
>>
>> I'm thinking scripts and blocks should know about which ranges they 
>> contain.
>> In particular, \p{BlockName} should not need binary search at
>> regex compile time or runtime.
>>   
> It definitely is desirable if we can avoid the binary-search lookup 
> during at least the runtime. The
> cost will be to keep a separate/redundant block/script->ranges table 
> in regex itself.
>
>> ---
>> There is one place you need to change
>> key word => keyword
>> ---
>> InMongolian => {@code InMongolian}
>> ---
>>   
>
> Good catch, thanks!
>
>> I notice current Unicode block support in JDK is not updated to the
>> latest standard.
>> E.g. Samaritan is missing.
>>
>>   
> The Character class has not been updated to the latest 5.20 yet. Yuka 
> has a CCC pending for
> that. My script data is from the 5.20.
>
>
>> Martin
>>
>> On Thu, Apr 22, 2010 at 01:01, Xueming Shen <xueming.shen at oracle.com> 
>> wrote:
>>  
>>> Hi,
>>>
>>> Here is the webrev of the proposal to add Unicode script support in 
>>> regex
>>> and j.l.Character.
>>>
>>> http://cr.openjdk.java.net/~sherman/script/webrev
>>>
>>> and the corresponding blenderrev
>>>
>>> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
>>>
>>> Please comment on the APIs before I submit the CCC, especially
>>>
>>> (1) to use enum for the j.l.Character.UnicodeScript (compared to the
>>> traditional j.l.c.Subset)
>>> (2) the piggyback method j.l.c.getName() :-)
>>> (3) the syntax for script constructs. In addition to the "normal"
>>>    \p{InScriptName} and \P{InScriptName} for the script support
>>>    I'm also adding
>>>   \p{script=ScriptName} \P{script=ScriptName}  for the new script 
>>> support
>>>   \p{block=BlockName} \P{block=BlockName}  for the "existing" block 
>>> support
>>>   \p{general_category=CategoryName} 
>>> \P{general_category=CategoryName} for
>>> the "existing" gc
>>>   Perl recently also started to accept this  \p{propName=propValue} 
>>> Unicode
>>> style.
>>>   It opens the door for future "expanding", for example \p{name=XYZ} 
>>> :-)
>>> (4)and of course, the wording.
>>>
>>> Thanks,
>>> Sherman
>>>
>>>
>>>
>>>     
>
>



From Ulf.Zibis at gmx.de  Mon Apr 26 09:23:20 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Mon, 26 Apr 2010 11:23:20 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD52498.2090006@oracle.com>
References: <4BD00250.3020206@oracle.com>	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>	<4BD3E7AC.2020801@oracle.com>
	<4BD52498.2090006@oracle.com>
Message-ID: <4BD55B88.3020501@gmx.de>

Am 26.04.2010 07:28, schrieb Xueming Shen:
>
> Can I assume we are all OK with at least the API part of the latest 
> webrev/blenderrev of
> the script support in j.l.Character and j.u.r.Pattern, including the 
> j.l.Chareacter.getName().

I guess you mean:
     public static enum UnicodeScript {
         COMMON,
         ...;
         public static UnicodeScript of(int codePoint);
         public static final UnicodeScript forName(String scriptName);
     }
     public static String getName(int codePoint);

I'm ok with this api on enum base.

I would like to see the full names redundantly in the aliases map. Needs 
only ~100 * (4 + 4) bytes in HashMap<String, Character.UnicodeScript>.
I think there should be some more words in the javadoc about 
correlation/usecase/advantage of UnicodeScript against against UnicodeBlock.
I would like to have the 3 special cases INHERITED, COMMON and UNKNOWN 
together at the beginning or end of the enum list.


>
> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
> http://cr.openjdk.java.net/~sherman/script/webrev
>
> Okutsu-san, Yuka, can one of you help review the corresponding CCC at
> http://ccc.sfbay.sun.com/6945564?

I get server not found: ccc.sfbay.sun.com

-Ulf




From Ulf.Zibis at gmx.de  Mon Apr 26 11:22:21 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Mon, 26 Apr 2010 13:22:21 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD228A3.7010103@oracle.com>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com>
Message-ID: <4BD5776D.9040808@gmx.de>

Am 24.04.2010 01:09, schrieb Xueming Shen:
> Ulf Zibis wrote:
>>
>> - I like the idea, saving the data in a compressed binary file, 
>> instead classfile static data.
>> - wouldn't PreHashMaps be faster initialized as a normal HashMaps in 
>> j.l.Character.UnicodeScript and j.l.CharacterName?
> I don't think so. The key for these 2 cases is the whole unicode 
> range. But you can always try. Current
> binary-search for script definitely is not a perfect solution.

I think, the aliases map is perfectly predestined for PreHashMap.

Anyway I more would like (just syntax sugar + initial capacity):
-        private static HashMap<String, Character.UnicodeScript> aliases;
+        private final static HashMap<String, Character.UnicodeScript> 
aliases = new HashMap<>(values().length);
         static {
-            aliases = new HashMap<String, UnicodeScript>();
             aliases.put("ARAB", ARABIC);
             ...

or (see project coin):
     private final static HashMap<String, Character.UnicodeScript> 
aliases = {
         "ARAB" : ARABIC,
         ...,
     }


-Ulf




From xueming.shen at oracle.com  Mon Apr 26 15:08:08 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Mon, 26 Apr 2010 08:08:08 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD5776D.9040808@gmx.de>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com> <4BD5776D.9040808@gmx.de>
Message-ID: <4BD5AC58.40807@oracle.com>

Ulf Zibis wrote:
> Am 24.04.2010 01:09, schrieb Xueming Shen:
>> Ulf Zibis wrote:
>>>
>>> - I like the idea, saving the data in a compressed binary file, 
>>> instead classfile static data.
>>> - wouldn't PreHashMaps be faster initialized as a normal HashMaps in 
>>> j.l.Character.UnicodeScript and j.l.CharacterName?
>> I don't think so. The key for these 2 cases is the whole unicode 
>> range. But you can always try. Current
>> binary-search for script definitely is not a perfect solution.
>
> I think, the aliases map is perfectly predestined for PreHashMap.
>
PreHashMap normally is used in place that the inti cost is critical for 
startup time. I
don't think it's a concern here.

-Sherman


From vladimir.iaroslavski at googlemail.com  Mon Apr 26 21:50:08 2010
From: vladimir.iaroslavski at googlemail.com (Vladimir Iaroslavski)
Date: Tue, 27 Apr 2010 01:50:08 +0400
Subject: New portion of improvements for Dual-Pivot Quicksort
Message-ID: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>

Hello, everyone!

I've investigated the implementation of the Dual-Pivot Quicksort
which is used for sorting primitives and here is my result:

http://cr.openjdk.java.net/~alanb/6947216/webrev.00

New implementation of Dual-Pivot Quicksort is faster
than previous one of 12% for client VM and few percents for
server VM. Tests on Bentley's test suite (Win XP, JDK 7,
build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
for client VM and 0.98-1.00 for server VM.

In compare with sorting from JDK 6 by Jon L. Bentley and
M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
(client / server).

See the execution time for sorting array of 2`000`000 int elements
50 times, client / server VM, in milliseconds:

random
 new: 16723 18776
jdk7: 17571 18975
jdk6: 22241 26524

ascendant
 new:  3541  4702
jdk7:  4486  4669
jdk6:  8667  7978

descendant
 new:  3854  4907
jdk7:  4942  5034
jdk6:  8787  8243

equal
 new:   234   281
jdk7:   291   230
jdk6:   602  1018

organ pipes
 new:  7673  8613
jdk7:  8167  8993
jdk6: 11902 14044

stagger 1
 new:  7648  8591
jdk7:  8161  8979
jdk6: 11908 13810

stagger 2
 new:  8349  9299
jdk7: 10968 11916
jdk6: 12194 14734

stagger 4
 new:  8475  9622
jdk7:  9221  9682
jdk6: 10523 12006

stagger 8
 new:  9321 10689
jdk7: 11125 12387
jdk6: 13829 16214

period 1..2
 new:   758   751
jdk7:   870   754
jdk6:  1038  1227

period 1..4
 new:  1004   963
jdk7:  1365  1209
jdk6:  1511  1847

period 1..8
 new:  1588  1573
jdk7:  1599  1790
jdk6:  2602  3045

random 1..2
 new:  1327  1125
jdk7:  1362  1496
jdk6:  1531  2182

random 1..4
 new:  1830  2118
jdk7:  1851  2236
jdk6:  2292  3025

where stagger(m) is array like a[i] = i * (m + 1) % length.

The summary of changes is:

1. For sorting small arrays is used insertion sort with sentinel
instead of traditional, which has the structure:

for (int i = left + 1; i <= right; i++) {
    for (j = i; j > left && a[j-1] > a[j]; j--) {
        swap(a[i], a[j-1]);
    }
}

Note that range check j > left is performed on each iteration,
but really helps very rare. To avoid this expensive range check,
it was suggested to set minimum value (negative infinity) on the
first position. This type of suggestion is used in new version:

if left bound > 0, we can put sentinel on a[left - 1], do insertion
sort without expensive check range, and then restore a[left - 1]
value. If left == 0, traditional insertion sort is used. Please,
look at the webrev for details.

2. In previous implementation 5 evenly spaced elements

sixth = length / 6;
a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]

were used as candidates of pivots elements. This case is very
sensitive for period inputs, especially by 6. The new suggestion
is to take 5 center evenly spaced elements like this:

int seventh = length / 7;
int midpoint = (left + right) >>> 1;

a[midpoint - 2 * seventh], a[midpoint -     seventh], a[midpoint],
a[midpoint +     seventh], a[midpoint + 2 * seventh]

and moreover, the seventh is calculated inexpensively:
seventh = (length >>> 3) + (length >>> 6) + 1;

This schema works the same on random, ascendant, descendant, equal
inputs, but much better for period / stagger.

3. The whole structure

<choose pivots>

if (pivotsDiffer) {
    <do partitioning for two pivots>
}
else {
    <do partitioning for one pivot>
}

<sort left and right parts>

if (!pivotsDiffer) {
    return;
}
<swap internal pivot values to ends>

<sort center part>

was modified to:
----------------

<choose pivots>

if (pivot1 < pivot2) {
    <do partitioning for two pivots>
    <swap pivots into their final positions>
    <sort left and right parts>
    <swap internal pivot values to ends>
    <sort center part>
}
else {
    <do partitioning for one pivot>
    <sort left and right parts>
}

4. Partitioning for both cases have not been changed at all.

5. Minor javadoc and format changes.

Please, review new implementation,
any comments / suggestions are welcome!

Thank you,
Vladimir


From xueming.shen at oracle.com  Mon Apr 26 22:01:54 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Mon, 26 Apr 2010 15:01:54 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD55B88.3020501@gmx.de>
References: <4BD00250.3020206@oracle.com>	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>	<4BD3E7AC.2020801@oracle.com>
	<4BD52498.2090006@oracle.com> <4BD55B88.3020501@gmx.de>
Message-ID: <4BD60D52.3080907@oracle.com>

Ulf Zibis wrote:
> Am 26.04.2010 07:28, schrieb Xueming Shen:
>>
>> Can I assume we are all OK with at least the API part of the latest 
>> webrev/blenderrev of
>> the script support in j.l.Character and j.u.r.Pattern, including the 
>> j.l.Chareacter.getName().
>
> I guess you mean:
>     public static enum UnicodeScript {
>         COMMON,
>         ...;
>         public static UnicodeScript of(int codePoint);
>         public static final UnicodeScript forName(String scriptName);
>     }
>     public static String getName(int codePoint);
>
> I'm ok with this api on enum base.
>
> I would like to see the full names redundantly in the aliases map. 
> Needs only ~100 * (4 + 4) bytes in HashMap<String, Character.
This is the implementation details, we can defer the difference for now.

> UnicodeScript>.
> I think there should be some more words in the javadoc about 
> correlation/usecase/advantage of UnicodeScript against against 
> UnicodeBlock.

Martin raised the same comment. But I still believe j.l.C.UnicodeScript 
simply defines the syntax of the Unicode script name
in the Java libraries, it does not try to interpret/implement anything 
further at semantics level. It just serves as a ID to the
Unicode script name, so it'd be better to leave the semantics 
definition/explanation to the TR#24.


> I would like to have the 3 special cases INHERITED, COMMON and UNKNOWN 
> together at the beginning or end of the enum list.

Why?  Since the current list is generated by the script from the 
Scripts.txt, it's in the order of what
they are in the Scripts.txt, any particular reason they should be listed 
differently? We do have the
links at the beginning already. I don't see any advantage of putting 
them physically together.

-Sherman


From Ulf.Zibis at gmx.de  Tue Apr 27 01:03:04 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Tue, 27 Apr 2010 03:03:04 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD60D52.3080907@oracle.com>
References: <4BD00250.3020206@oracle.com>	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>	<4BD3E7AC.2020801@oracle.com>
	<4BD52498.2090006@oracle.com> <4BD55B88.3020501@gmx.de>
	<4BD60D52.3080907@oracle.com>
Message-ID: <4BD637C8.3050206@gmx.de>

Am 27.04.2010 00:01, schrieb Xueming Shen:
> Ulf Zibis wrote:
>> I would like to see the full names redundantly in the aliases map. 
>> Needs only ~100 * (4 + 4) bytes in HashMap<String, Character>.
> This is the implementation details, we can defer the difference for now.

I said that with the alternative of UnicodeScript as _normal class_ in 
my head, if saving the redundant internal hash map should matter.

>
>> UnicodeScript>.
>> I think there should be some more words in the javadoc about 
>> correlation/usecase/advantage of UnicodeScript against against 
>> UnicodeBlock.
>
> Martin raised the same comment. But I still believe 
> j.l.C.UnicodeScript simply defines the syntax of the Unicode script name
> in the Java libraries, it does not try to interpret/implement anything 
> further at semantics level. It just serves as a ID to the
> Unicode script name, so it'd be better to leave the semantics 
> definition/explanation to the TR#24.

Yes, for the semantics definition/explanation of Unicode script name, 
user should refer to the TR#24.
But he might like to be briefly informed about the different 
semantic/usecase/disadvantage of UnicodeBlock

>
>
>> I would like to have the 3 special cases INHERITED, COMMON and 
>> UNKNOWN together at the beginning or end of the enum list.
>
> Why?  Since the current list is generated by the script from the 
> Scripts.txt, it's in the order of what
> they are in the Scripts.txt, any particular reason they should be 
> listed differently? We do have the
> links at the beginning already. I don't see any advantage of putting 
> them physically together.

Someone might find it useful to code for example
     if (script < UnicodeScript.LATIN)
to easily filter the special cases.
Same might be considered for SURROGATE, PRIVATE_USE, UNASSIGNED.

-Ulf




From Ulf.Zibis at gmx.de  Tue Apr 27 01:36:28 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Tue, 27 Apr 2010 03:36:28 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD228A3.7010103@oracle.com>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com>
Message-ID: <4BD63F9C.9010906@gmx.de>

Am 24.04.2010 01:09, schrieb Xueming Shen:
>
> I changed the data file "format" a bit, so now the overal uniName.dat 
> is less than 88k (last version is 122+k), but
> the I can no long use cpLen as the capacity for the hashmap. I'm now 
> using a hardcoded 20000 for 5.2.

Again, is 88k the compressed or the uncompressed size ?

>> -- Is it faster, first copying the whole date in a byte[], and then 
>> using ByteBuffer.getInt etc. against directly using DataInputStream 
>> methods?
>> -- You could create a very long String with the whole data and then 
>> use subString for the individual strings which could share the same 
>> backing char[].

See attachment.

>> -- I don't think, it's a good idea, holding the whole data in memory, 
>> especiallly as String objects; Additionally the backing char[]'s 
>> occupy twice the space than a byte[]
>> -- the big new byte[total] and later the huge amount of String 
>> objects could result in OOM error on small VM heap.
>> -- as compromise, you could put the cp->nameOff pointers in a 
>> separate not-compressed data file, only hold this in memory, or 
>> access it via DirectByteBuffer, and read the string data from 
>> separate file only on request from Character.getName(int codePoint). 
>> As option, a PreHashMap could cache individual loaded strings.
>> -- Anyway, having DirectByteBuffer access on deflated data would be a 
>> performace/footprint gain.
>>
> Sorry, I don't think I fully understand your points here.

See above, the others I try tomorrow.

-Ulf

-------------- next part --------------
A non-text attachment was scrubbed...
Name: CharacterName1.java
Type: java/*
Size: 3571 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100427/6e396367/CharacterName1.java>

From xueming.shen at oracle.com  Tue Apr 27 04:25:28 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Mon, 26 Apr 2010 21:25:28 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD63F9C.9010906@gmx.de>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com> <4BD63F9C.9010906@gmx.de>
Message-ID: <4BD66738.90806@oracle.com>

Ulf Zibis wrote:
> Am 24.04.2010 01:09, schrieb Xueming Shen:
>>
>> I changed the data file "format" a bit, so now the overal uniName.dat 
>> is less than 88k (last version is 122+k), but
>> the I can no long use cpLen as the capacity for the hashmap. I'm now 
>> using a hardcoded 20000 for 5.2.
>
> Again, is 88k the compressed or the uncompressed size ?

Yes, it's the size of compressed data. Your smart "save one more byte" 
suggestion will save
400+byte, a tiny 0.5%, unfortunately:-)

>
>>> -- Is it faster, first copying the whole date in a byte[], and then 
>>> using ByteBuffer.getInt etc. against directly using DataInputStream 
>>> methods?
The current impl use neither ByteBuffer nor DataInputStream now, so no 
compare here.
Yes, to use DataInputStream will definitely makes code look better (no 
more those "ugly"
shifts), but it also will slow down thing a little since it adds one 
more layer. But speed
may not really a concern here.

>>> -- You could create a very long String with the whole data and then 
>>> use subString for the individual strings which could share the same 
>>> backing char[].
>
The disadvantage of using a big buffer String to hold everything then 
have the individual names to substring
from it is that it might simply break the softreference logic here. The 
big char[] will never been gc-ed as
long as there is still one single name object (substring-ed from it) is 
still walking around in system somewhere.
I don't think the vm/gc is that smart, isn't it?

But this will definitely be faster, given the burden of creating a 
String from bytes (we put in the optimization
earlier, so this operation should be faster now compared to 6u).

-Sherman


From xueming.shen at oracle.com  Tue Apr 27 04:32:26 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Mon, 26 Apr 2010 21:32:26 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD637C8.3050206@gmx.de>
References: <4BD00250.3020206@oracle.com>	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>	<4BD3E7AC.2020801@oracle.com>
	<4BD52498.2090006@oracle.com> <4BD55B88.3020501@gmx.de>
	<4BD60D52.3080907@oracle.com> <4BD637C8.3050206@gmx.de>
Message-ID: <4BD668DA.9060209@oracle.com>

Ulf Zibis wrote:
>> I would like to have the 3 special cases INHERITED, COMMON and 
>> UNKNOWN together at the beginning or end of the enum list.
>>
>> Why?  Since the current list is generated by the script from the 
>> Scripts.txt, it's in the order of what
>> they are in the Scripts.txt, any particular reason they should be 
>> listed differently? We do have the
>> links at the beginning already. I don't see any advantage of putting 
>> them physically together.
>
> Someone might find it useful to code for example
>     if (script < UnicodeScript.LATIN)
> to easily filter the special cases.
> Same might be considered for SURROGATE, PRIVATE_USE, UNASSIGNED.
>
I don't think Java should DEFINE and FORCE a logical order of Unicode 
script names for Unicode consortium. It
would be better to Leave that to the appropriate party.

-Sherman


From chris.hegarty at oracle.com  Tue Apr 27 08:44:27 2010
From: chris.hegarty at oracle.com (chris.hegarty at oracle.com)
Date: Tue, 27 Apr 2010 08:44:27 +0000
Subject: hg: jdk7/tl/jdk: 6718504: IN6_IS_ADDR_ANY tests only 12 bytes of
	16-byte address
Message-ID: <20100427084440.6CAE0441AD@hg.openjdk.java.net>

Changeset: 0c27202d66c1
Author:    chegar
Date:      2010-04-27 09:42 +0100
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/0c27202d66c1

6718504: IN6_IS_ADDR_ANY tests only 12 bytes of 16-byte address
Reviewed-by: alanb

! src/windows/native/java/net/net_util_md.h
+ test/java/net/DatagramSocket/LocalSocketAddress.java



From Ulf.Zibis at gmx.de  Tue Apr 27 14:35:58 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Tue, 27 Apr 2010 16:35:58 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD66738.90806@oracle.com>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com> <4BD63F9C.9010906@gmx.de>
	<4BD66738.90806@oracle.com>
Message-ID: <4BD6F64E.3050909@gmx.de>

Am 27.04.2010 06:25, schrieb Xueming Shen:
> Ulf Zibis wrote:
>> Am 24.04.2010 01:09, schrieb Xueming Shen:
>>>
>>> I changed the data file "format" a bit, so now the overal 
>>> uniName.dat is less than 88k (last version is 122+k), but
>>> the I can no long use cpLen as the capacity for the hashmap. I'm now 
>>> using a hardcoded 20000 for 5.2.
>>
>> Again, is 88k the compressed or the uncompressed size ?
>
> Yes, it's the size of compressed data.

I'm wondering, as script.txt only has ~120k.

> Your smart "save one more byte" suggestion will save
> 400+byte, a tiny 0.5%, unfortunately:-)

I didn't mean the save by total file footprint, I meant it by byte-wise 
read() count.
My code only needs to read 1 int per character block against 1 byte + 1 
int, which looks kinda ugly too.
Anyway, the theoretical max win would be < 20 %.

>
>>
>>>> -- Is it faster, first copying the whole date in a byte[], and then 
>>>> using ByteBuffer.getInt etc. against directly using DataInputStream 
>>>> methods?
> The current impl use neither ByteBuffer nor DataInputStream now, so no 
> compare here.
If JIT-compiled, bb.get() should be as fast as ba[cpOff++] & 0xff.
My compare is about the manually byte2int assembling + triple buffering 
the data (getResourceAsStream() is a buffered stream, and I believe 
InflaterInputStream too)

> Yes, to use DataInputStream will definitely makes code look better (no 
> more those "ugly"
> shifts), but it also will slow down thing a little since it adds one 
> more layer. But speed
> may not really a concern here.

On the other hand:
- layer shouldn't matter if DIS is yet JIT-compiled.
- readInt() might be faster than 4 times read() + manually assembling 
the int value. (if not, DataInputStream needs reengineering)
- readFully() might be better optimized than your hand-coded read loop 
(if not, let's do it ;-) )
-- hand-coded loop might only make sense, if using thread.sleep() after 
each chunk,
     so concurrent threads could continue their work, while waiting for 
the harddisk to read.
- your code will surely run in interpreter mode, as GIT wouldn't have 
time to compile it fast enough.
- there is some chance, that DIS will be yet JIT-compiled from usage of 
other program parts before.
- and last but not least, use the given API's for byte code footprint 
reduction as most as you can. Give good programming example as newbies 
tend to use API sources as first template for their own code. Seeing API 
use cases helps to become familiar with the complexity of the Java-API. 
(Same for Arrays.binarySearch())


>
>>>> -- You could create a very long String with the whole data and then 
>>>> use subString for the individual strings which could share the same 
>>>> backing char[].
>>
> The disadvantage of using a big buffer String to hold everything then 
> have the individual names to substring
> from it is that it might simply break the softreference logic here. 
> The big char[] will never been gc-ed as
> long as there is still one single name object (substring-ed from it) 
> is still walking around in system somewhere.
> I don't think the vm/gc is that smart, isn't it?

Good point, I missed that.
But I'm still no friend of the SR usage here. It doesn't solve my main 
complain:
- In-economically initializing the whole amount of data for likely 1 or 
few invocations of getName(int cp), and repetitively, if SR was cleaned.
- Don't pollute the GC more than necessary (it would have to handle each 
of the strings + char[]s separate), especially if memory comes towards 
it's limit.
Additionally, if not interned, equal character name strings would be 
hold in memory for as many copies, as SR fails, if interned, they would 
never be GC'd.
You may argue, that code is rarely used, but if all corners of the Java 
API would be coded such memory/performance-wasting, we ... I don't think 
about it better.
We could add (Attention: CCC change) a cacheCharacterNames(boolean 
yesNo) method to serve users, which excessively need this functionality.


>
> But this will definitely be faster, given the burden of creating a 
> String from bytes (we put in the optimization
> earlier, so this operation should be faster now compared to 6u).

+ saving the memory overhead + GC work for the cpNum char[]s.


Additionally:
- No need to compare iis != null in finally block, possible NPE would be 
thrown earlier.
- Move SR logic to get() method to omit the possible remaining SR->NPE 
problem:
     public static String get(int cp) {
         HashMap<Integer, String> names;
         if (refNames == null || (names = refNames.get()) == null)
             refNames = new SoftReference<>(names = getNames());
         return names.get(cp);
     }
- then synchronize entire getNames() method.
- save 2nd null-check after sync, as fail would still be much more 
unlikely as getName(int cp) usage at all, and only risks 2nd superfluous 
init.
- Is it good idea to return null in case of io fail to calling code, 
instead propagating the given exception or better throwing an error?
- use Integer.toHexString(cp) instead Integer.toString(cp, 16);
- IMPORTANT (check if CCC is affected):
   Do I understand right, that j.l.Ch.getName('5') would return:
       "Basic Latin 35"
   ... but j.l.Ch.getName('0') would return:
       "DIGIT ZERO..DIGIT NINE"
   I think both should return:
       "DIGIT ZERO..DIGIT NINE" (otherwise we don't have to cache that 
value ;-) )
   or at least:
       "Basic Latin U+0035"

See new version in attachment.


-Ulf




From Ulf.Zibis at gmx.de  Tue Apr 27 15:35:34 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Tue, 27 Apr 2010 17:35:34 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD6F64E.3050909@gmx.de>
References: <4BD00250.3020206@oracle.com>
	<4BD05426.7030309@gmx.de>	<4BD228A3.7010103@oracle.com>
	<4BD63F9C.9010906@gmx.de>	<4BD66738.90806@oracle.com>
	<4BD6F64E.3050909@gmx.de>
Message-ID: <4BD70446.3010508@gmx.de>

Oops, added attachment.

-Ulf

Am 27.04.2010 16:35, schrieb Ulf Zibis:
> Am 27.04.2010 06:25, schrieb Xueming Shen:
>> Ulf Zibis wrote:
>>> Am 24.04.2010 01:09, schrieb Xueming Shen:
>>>>
>>>> I changed the data file "format" a bit, so now the overal 
>>>> uniName.dat is less than 88k (last version is 122+k), but
>>>> the I can no long use cpLen as the capacity for the hashmap. I'm 
>>>> now using a hardcoded 20000 for 5.2.
>>>
>>> Again, is 88k the compressed or the uncompressed size ?
>>
>> Yes, it's the size of compressed data.
>
> I'm wondering, as script.txt only has ~120k.
>
>> Your smart "save one more byte" suggestion will save
>> 400+byte, a tiny 0.5%, unfortunately:-)
>
> I didn't mean the save by total file footprint, I meant it by 
> byte-wise read() count.
> My code only needs to read 1 int per character block against 1 byte + 
> 1 int, which looks kinda ugly too.
> Anyway, the theoretical max win would be < 20 %.
>
>>
>>>
>>>>> -- Is it faster, first copying the whole date in a byte[], and 
>>>>> then using ByteBuffer.getInt etc. against directly using 
>>>>> DataInputStream methods?
>> The current impl use neither ByteBuffer nor DataInputStream now, so 
>> no compare here.
> If JIT-compiled, bb.get() should be as fast as ba[cpOff++] & 0xff.
> My compare is about the manually byte2int assembling + triple 
> buffering the data (getResourceAsStream() is a buffered stream, and I 
> believe InflaterInputStream too)
>
>> Yes, to use DataInputStream will definitely makes code look better 
>> (no more those "ugly"
>> shifts), but it also will slow down thing a little since it adds one 
>> more layer. But speed
>> may not really a concern here.
>
> On the other hand:
> - layer shouldn't matter if DIS is yet JIT-compiled.
> - readInt() might be faster than 4 times read() + manually assembling 
> the int value. (if not, DataInputStream needs reengineering)
> - readFully() might be better optimized than your hand-coded read loop 
> (if not, let's do it ;-) )
> -- hand-coded loop might only make sense, if using thread.sleep() 
> after each chunk,
>     so concurrent threads could continue their work, while waiting for 
> the harddisk to read.
> - your code will surely run in interpreter mode, as GIT wouldn't have 
> time to compile it fast enough.
> - there is some chance, that DIS will be yet JIT-compiled from usage 
> of other program parts before.
> - and last but not least, use the given API's for byte code footprint 
> reduction as most as you can. Give good programming example as newbies 
> tend to use API sources as first template for their own code. Seeing 
> API use cases helps to become familiar with the complexity of the 
> Java-API. (Same for Arrays.binarySearch())
>
>
>>
>>>>> -- You could create a very long String with the whole data and 
>>>>> then use subString for the individual strings which could share 
>>>>> the same backing char[].
>>>
>> The disadvantage of using a big buffer String to hold everything then 
>> have the individual names to substring
>> from it is that it might simply break the softreference logic here. 
>> The big char[] will never been gc-ed as
>> long as there is still one single name object (substring-ed from it) 
>> is still walking around in system somewhere.
>> I don't think the vm/gc is that smart, isn't it?
>
> Good point, I missed that.
> But I'm still no friend of the SR usage here. It doesn't solve my main 
> complain:
> - In-economically initializing the whole amount of data for likely 1 
> or few invocations of getName(int cp), and repetitively, if SR was 
> cleaned.
> - Don't pollute the GC more than necessary (it would have to handle 
> each of the strings + char[]s separate), especially if memory comes 
> towards it's limit.
> Additionally, if not interned, equal character name strings would be 
> hold in memory for as many copies, as SR fails, if interned, they 
> would never be GC'd.
> You may argue, that code is rarely used, but if all corners of the 
> Java API would be coded such memory/performance-wasting, we ... I 
> don't think about it better.
> We could add (Attention: CCC change) a cacheCharacterNames(boolean 
> yesNo) method to serve users, which excessively need this functionality.
>
>
>>
>> But this will definitely be faster, given the burden of creating a 
>> String from bytes (we put in the optimization
>> earlier, so this operation should be faster now compared to 6u).
>
> + saving the memory overhead + GC work for the cpNum char[]s.
>
>
> Additionally:
> - No need to compare iis != null in finally block, possible NPE would 
> be thrown earlier.
> - Move SR logic to get() method to omit the possible remaining SR->NPE 
> problem:
>     public static String get(int cp) {
>         HashMap<Integer, String> names;
>         if (refNames == null || (names = refNames.get()) == null)
>             refNames = new SoftReference<>(names = getNames());
>         return names.get(cp);
>     }
> - then synchronize entire getNames() method.
> - save 2nd null-check after sync, as fail would still be much more 
> unlikely as getName(int cp) usage at all, and only risks 2nd 
> superfluous init.
> - Is it good idea to return null in case of io fail to calling code, 
> instead propagating the given exception or better throwing an error?
> - use Integer.toHexString(cp) instead Integer.toString(cp, 16);
> - IMPORTANT (check if CCC is affected):
>   Do I understand right, that j.l.Ch.getName('5') would return:
>       "Basic Latin 35"
>   ... but j.l.Ch.getName('0') would return:
>       "DIGIT ZERO..DIGIT NINE"
>   I think both should return:
>       "DIGIT ZERO..DIGIT NINE" (otherwise we don't have to cache that 
> value ;-) )
>   or at least:
>       "Basic Latin U+0035"
>
> See new version in attachment.
>
>
> -Ulf
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CharacterName1.java
Type: java/*
Size: 3408 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100427/e32aa5e7/CharacterName1.java>

From xueming.shen at oracle.com  Tue Apr 27 17:03:52 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Tue, 27 Apr 2010 10:03:52 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD6F64E.3050909@gmx.de>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com> <4BD63F9C.9010906@gmx.de>
	<4BD66738.90806@oracle.com> <4BD6F64E.3050909@gmx.de>
Message-ID: <4BD718F8.8080903@oracle.com>

Ulf Zibis wrote:
> Am 27.04.2010 06:25, schrieb Xueming Shen:
>> Ulf Zibis wrote:
>>> Am 24.04.2010 01:09, schrieb Xueming Shen:
>>>>
>>>> I changed the data file "format" a bit, so now the overal 
>>>> uniName.dat is less than 88k (last version is 122+k), but
>>>> the I can no long use cpLen as the capacity for the hashmap. I'm 
>>>> now using a hardcoded 20000 for 5.2.
>>>
>>> Again, is 88k the compressed or the uncompressed size ?
>>
>> Yes, it's the size of compressed data.
>
> I'm wondering, as script.txt only has ~120k.

Ulf, you know we are not talking about Unicode scirpt but Unicode 
character name here, right?
Unicode character name data is stored in UnicodeData.txt, you can find 
it at make/tools/UnicodeData.

>
> - and last but not least, use the given API's for byte code footprint 
> reduction as most as you can. Give good programming example as newbies 
> tend to use API sources as first template for their own code. Seeing 
> API use cases helps to become familiar with the complexity of the 
> Java-API. (Same for Arrays.binarySearch())
>
Good advice. I will keep it for the rest of my career:-)

>
>
> Additionally:
> - No need to compare iis != null in finally block, possible NPE would 
> be thrown earlier.
Maybe I'm paranoid but the check in finally block is for the scenario 
that the getResourceAsStream()
fails unexpectedly, for example, the uniName.dat is missing, in that 
case the iis might be null. And the
corresponding exception has already been caught in my catch block 
already. The current impl simply
print out the exception stack trace. The alternative might be throw a 
fatal error.

> -
>   Do I understand right, that j.l.Ch.getName('5') would return:
>       "Basic Latin 35"
>   ... but j.l.Ch.getName('0') would return:
>       "DIGIT ZERO..DIGIT NINE"
>   I think both should return:
>       "DIGIT ZERO..DIGIT NINE" (otherwise we don't have to cache that 
> value ;-) )
>   or at least:
>       "Basic Latin U+0035"
j.l.Ch.getName('0')  returns
DIGIT ZERO

j.l.Ch.getName('5')  returns
DIGIT FIVE

The name comes from 
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt. You need to
convince the Unicode consortium if you prefer anything else:-)

I would review other coding style issue later.

-Sherman







From Ulf.Zibis at gmx.de  Tue Apr 27 21:50:02 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Tue, 27 Apr 2010 23:50:02 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD718F8.8080903@oracle.com>
References: <4BD00250.3020206@oracle.com> <4BD05426.7030309@gmx.de>
	<4BD228A3.7010103@oracle.com> <4BD63F9C.9010906@gmx.de>
	<4BD66738.90806@oracle.com> <4BD6F64E.3050909@gmx.de>
	<4BD718F8.8080903@oracle.com>
Message-ID: <4BD75C0A.5050107@gmx.de>

Am 27.04.2010 19:03, schrieb Xueming Shen:
> Ulf Zibis wrote:
>>
>> I'm wondering, as script.txt only has ~120k.
>
> Ulf, you know we are not talking about Unicode scirpt but Unicode 
> character name here, right?
> Unicode character name data is stored in UnicodeData.txt, you can find 
> it at make/tools/UnicodeData.

Oop, thanks for solving my confusion. As UnicodeData.txt isn't part of 
your webrev, I mixed the two.

>
>>
>> - and last but not least, use the given API's for byte code footprint 
>> reduction as most as you can. Give good programming example as 
>> newbies tend to use API sources as first template for their own code. 
>> Seeing API use cases helps to become familiar with the complexity of 
>> the Java-API. (Same for Arrays.binarySearch())
>>
> Good advice. I will keep it for the rest of my career:-)

Thanks for your humour. :-D


>
>>
>>
>> Additionally:
>> - No need to compare iis != null in finally block, possible NPE would 
>> be thrown earlier.
> Maybe I'm paranoid but the check in finally block is for the scenario 
> that the getResourceAsStream()
> fails unexpectedly, for example, the uniName.dat is missing, in that 
> case the iis might be null. And the
> corresponding exception has already been caught in my catch block 
> already. The current impl simply
> print out the exception stack trace. The alternative might be throw a 
> fatal error.

Yes, my assumption was, that if the getResourceAsStream() fails, an 
exception would be raised, or iis would be null, so referencing it would 
raise an NPE anyway, so both should be caught before the final clause 
would come to account. Maybe I'm wrong with that.

>
>> -
>>   Do I understand right, that j.l.Ch.getName('5') would return:
>>       "Basic Latin 35"
>>   ... but j.l.Ch.getName('0') would return:
>>       "DIGIT ZERO..DIGIT NINE"
>>   I think both should return:
>>       "DIGIT ZERO..DIGIT NINE" (otherwise we don't have to cache that 
>> value ;-) )
>>   or at least:
>>       "Basic Latin U+0035"
> j.l.Ch.getName('0')  returns
> DIGIT ZERO
>
> j.l.Ch.getName('5')  returns
> DIGIT FIVE
>
> The name comes from 
> http://www.unicode.org/Public/UNIDATA/UnicodeData.txt. You need to
> convince the Unicode consortium if you prefer anything else:-)

Confusion caused from my mix-up above, thanks.

-Ulf




From jjb at google.com  Wed Apr 28 07:38:53 2010
From: jjb at google.com (Joshua Bloch)
Date: Wed, 28 Apr 2010 00:38:53 -0700
Subject: New portion of improvements for Dual-Pivot Quicksort
In-Reply-To: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
References: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
Message-ID: <u2g17b2302a1004280038g54a2c487lf68fc96d815ba3b8@mail.gmail.com>

Vladimir,

Cool!  I will look at the changes tomorrow (Wednesday).  Have you told Jon
Bentely?

    Josh

On Mon, Apr 26, 2010 at 2:50 PM, Vladimir Iaroslavski <
vladimir.iaroslavski at googlemail.com> wrote:

> Hello, everyone!
>
> I've investigated the implementation of the Dual-Pivot Quicksort
> which is used for sorting primitives and here is my result:
>
> http://cr.openjdk.java.net/~alanb/6947216/webrev.00
>
> New implementation of Dual-Pivot Quicksort is faster
> than previous one of 12% for client VM and few percents for
> server VM. Tests on Bentley's test suite (Win XP, JDK 7,
> build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
> for client VM and 0.98-1.00 for server VM.
>
> In compare with sorting from JDK 6 by Jon L. Bentley and
> M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
> (client / server).
>
> See the execution time for sorting array of 2`000`000 int elements
> 50 times, client / server VM, in milliseconds:
>
> random
>  new: 16723 18776
> jdk7: 17571 18975
> jdk6: 22241 26524
>
> ascendant
>  new:  3541  4702
> jdk7:  4486  4669
> jdk6:  8667  7978
>
> descendant
>  new:  3854  4907
> jdk7:  4942  5034
> jdk6:  8787  8243
>
> equal
>  new:   234   281
> jdk7:   291   230
> jdk6:   602  1018
>
> organ pipes
>  new:  7673  8613
> jdk7:  8167  8993
> jdk6: 11902 14044
>
> stagger 1
>  new:  7648  8591
> jdk7:  8161  8979
> jdk6: 11908 13810
>
> stagger 2
>  new:  8349  9299
> jdk7: 10968 11916
> jdk6: 12194 14734
>
> stagger 4
>  new:  8475  9622
> jdk7:  9221  9682
> jdk6: 10523 12006
>
> stagger 8
>  new:  9321 10689
> jdk7: 11125 12387
> jdk6: 13829 16214
>
> period 1..2
>  new:   758   751
> jdk7:   870   754
> jdk6:  1038  1227
>
> period 1..4
>  new:  1004   963
> jdk7:  1365  1209
> jdk6:  1511  1847
>
> period 1..8
>  new:  1588  1573
> jdk7:  1599  1790
> jdk6:  2602  3045
>
> random 1..2
>  new:  1327  1125
> jdk7:  1362  1496
> jdk6:  1531  2182
>
> random 1..4
>  new:  1830  2118
> jdk7:  1851  2236
> jdk6:  2292  3025
>
> where stagger(m) is array like a[i] = i * (m + 1) % length.
>
> The summary of changes is:
>
> 1. For sorting small arrays is used insertion sort with sentinel
> instead of traditional, which has the structure:
>
> for (int i = left + 1; i <= right; i++) {
>    for (j = i; j > left && a[j-1] > a[j]; j--) {
>        swap(a[i], a[j-1]);
>    }
> }
>
> Note that range check j > left is performed on each iteration,
> but really helps very rare. To avoid this expensive range check,
> it was suggested to set minimum value (negative infinity) on the
> first position. This type of suggestion is used in new version:
>
> if left bound > 0, we can put sentinel on a[left - 1], do insertion
> sort without expensive check range, and then restore a[left - 1]
> value. If left == 0, traditional insertion sort is used. Please,
> look at the webrev for details.
>
> 2. In previous implementation 5 evenly spaced elements
>
> sixth = length / 6;
> a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]
>
> were used as candidates of pivots elements. This case is very
> sensitive for period inputs, especially by 6. The new suggestion
> is to take 5 center evenly spaced elements like this:
>
> int seventh = length / 7;
> int midpoint = (left + right) >>> 1;
>
> a[midpoint - 2 * seventh], a[midpoint -     seventh], a[midpoint],
> a[midpoint +     seventh], a[midpoint + 2 * seventh]
>
> and moreover, the seventh is calculated inexpensively:
> seventh = (length >>> 3) + (length >>> 6) + 1;
>
> This schema works the same on random, ascendant, descendant, equal
> inputs, but much better for period / stagger.
>
> 3. The whole structure
>
> <choose pivots>
>
> if (pivotsDiffer) {
>    <do partitioning for two pivots>
> }
> else {
>    <do partitioning for one pivot>
> }
>
> <sort left and right parts>
>
> if (!pivotsDiffer) {
>    return;
> }
> <swap internal pivot values to ends>
>
> <sort center part>
>
> was modified to:
> ----------------
>
> <choose pivots>
>
> if (pivot1 < pivot2) {
>    <do partitioning for two pivots>
>    <swap pivots into their final positions>
>    <sort left and right parts>
>    <swap internal pivot values to ends>
>    <sort center part>
> }
> else {
>    <do partitioning for one pivot>
>    <sort left and right parts>
> }
>
> 4. Partitioning for both cases have not been changed at all.
>
> 5. Minor javadoc and format changes.
>
> Please, review new implementation,
> any comments / suggestions are welcome!
>
> Thank you,
> Vladimir
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100428/23c0b3cc/attachment.html>

From xueming.shen at oracle.com  Wed Apr 28 17:12:21 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Wed, 28 Apr 2010 10:12:21 -0700
Subject: [Fwd: CCC request approved: 6934265: Add public method ...]
Message-ID: <4BD86C75.6070309@oracle.com>

Martin,

I re-submitted the CCC request for 6934265 again with the more 
"generally accepted" naming convention
j.l.C.isBmpCodePoint, and I has been approved as showed in attached 
email. So you're OK to putback
this change with isBmpCodePoint now, if you are ready.

-Sherman

-------- Original Message --------
Subject: 	CCC request approved: 6934265: Add public method ...
Date: 	Wed, 28 Apr 2010 10:06:01 -0700 (PDT)
From: 	j2se-ccc-daemon at sun.com
Reply-To: 	j2se-ccc at sun.com
To: 	xueming.shen at sun.com



Your CCC request

    6934265: Add public method Character.isBMPCodePoint

has been approved.

For more information: http://ccc.sfbay/6934265




From martinrb at google.com  Wed Apr 28 22:38:28 2010
From: martinrb at google.com (Martin Buchholz)
Date: Wed, 28 Apr 2010 15:38:28 -0700
Subject: [Fwd: CCC request approved: 6934265: Add public method ...]
In-Reply-To: <4BD86C75.6070309@oracle.com>
References: <4BD86C75.6070309@oracle.com>
Message-ID: <r2o1ccfd1c11004281538webf24c75m1395d3c09254f0fe@mail.gmail.com>

Sherman,

Thanks.  I updated all the related webrevs.
I hope to work on my backlog of core-libs changes soon.

Martin

On Wed, Apr 28, 2010 at 10:12, Xueming Shen <xueming.shen at oracle.com> wrote:
> Martin,
>
> I re-submitted the CCC request for 6934265 again with the more "generally
> accepted" naming convention
> j.l.C.isBmpCodePoint, and I has been approved as showed in attached email.
> So you're OK to putback
> this change with isBmpCodePoint now, if you are ready.
>
> -Sherman
>
> -------- Original Message --------
> Subject: ? ? ? ?CCC request approved: 6934265: Add public method ...
> Date: ? Wed, 28 Apr 2010 10:06:01 -0700 (PDT)
> From: ? j2se-ccc-daemon at sun.com
> Reply-To: ? ? ? j2se-ccc at sun.com
> To: ? ? xueming.shen at sun.com
>
>
>
> Your CCC request
>
> ? 6934265: Add public method Character.isBMPCodePoint
>
> has been approved.
>
> For more information: http://ccc.sfbay/6934265
>
>
>


From Ulf.Zibis at gmx.de  Thu Apr 29 00:32:14 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Thu, 29 Apr 2010 02:32:14 +0200
Subject: [Fwd: CCC request approved: 6934265: Add public method ...]
In-Reply-To: <r2o1ccfd1c11004281538webf24c75m1395d3c09254f0fe@mail.gmail.com>
References: <4BD86C75.6070309@oracle.com>
	<r2o1ccfd1c11004281538webf24c75m1395d3c09254f0fe@mail.gmail.com>
Message-ID: <4BD8D38E.4040808@gmx.de>

Martin, can you list the related webrevs again here, I'm confused.

I regret you didn't like my last more flexible form: 
isBmpCodePoint(boolean includeSurrogates). :-(

-Ulf


Am 29.04.2010 00:38, schrieb Martin Buchholz:
> Sherman,
>
> Thanks.  I updated all the related webrevs.
> I hope to work on my backlog of core-libs changes soon.
>
> Martin
>
> On Wed, Apr 28, 2010 at 10:12, Xueming Shen<xueming.shen at oracle.com>  wrote:
>    
>> Martin,
>>
>> I re-submitted the CCC request for 6934265 again with the more "generally
>> accepted" naming convention
>> j.l.C.isBmpCodePoint, and I has been approved as showed in attached email.
>> So you're OK to putback
>> this change with isBmpCodePoint now, if you are ready.
>>
>> -Sherman
>>
>> -------- Original Message --------
>> Subject:        CCC request approved: 6934265: Add public method ...
>> Date:   Wed, 28 Apr 2010 10:06:01 -0700 (PDT)
>> From:   j2se-ccc-daemon at sun.com
>> Reply-To:       j2se-ccc at sun.com
>> To:     xueming.shen at sun.com
>>
>>
>>
>> Your CCC request
>>
>>    6934265: Add public method Character.isBMPCodePoint
>>
>> has been approved.
>>
>> For more information: http://ccc.sfbay/6934265
>>
>>
>>
>>      
>
>    



From weijun.wang at sun.com  Thu Apr 29 07:52:42 2010
From: weijun.wang at sun.com (weijun.wang at sun.com)
Date: Thu, 29 Apr 2010 07:52:42 +0000
Subject: hg: jdk7/tl/jdk: 2 new changesets
Message-ID: <20100429075321.E03B4442BF@hg.openjdk.java.net>

Changeset: b833a422c776
Author:    weijun
Date:      2010-04-29 15:50 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/b833a422c776

6947487: use HexDumpEncoder.encodeBuffer()
Reviewed-by: mullan

! src/share/classes/com/sun/security/auth/module/Krb5LoginModule.java
! src/share/classes/com/sun/security/jgss/AuthorizationDataEntry.java
! src/share/classes/javax/security/auth/kerberos/KeyImpl.java
! src/share/classes/sun/security/krb5/EncryptionKey.java
! src/share/classes/sun/security/provider/certpath/CertId.java
! src/share/classes/sun/security/tools/KeyTool.java

Changeset: 4c8075f8c0ee
Author:    weijun
Date:      2010-04-29 15:51 +0800
URL:       http://hg.openjdk.java.net/jdk7/tl/jdk/rev/4c8075f8c0ee

6844193: support max_retries in krb5.conf
Reviewed-by: valeriep

! src/share/classes/sun/security/krb5/Config.java
! src/share/classes/sun/security/krb5/KrbKdcReq.java
+ test/sun/security/krb5/auto/MaxRetries.java



From gokdogan at gmail.com  Thu Apr 29 08:18:13 2010
From: gokdogan at gmail.com (Goktug Gokdogan)
Date: Thu, 29 Apr 2010 01:18:13 -0700
Subject: New portion of improvements for Dual-Pivot Quicksort
In-Reply-To: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
References: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
Message-ID: <t2pf5b3aa371004290118z56814bbsac4f42ab26ffd2c6@mail.gmail.com>

One quick comment: you can consider moving trivial array check so it will be
executed only if insertion sort threshold check passes:

if (length < INSERTION_SORT_THRESHOLD) {
  if (length < 2) {
    return;
  }
  ....



On Mon, Apr 26, 2010 at 2:50 PM, Vladimir Iaroslavski <
vladimir.iaroslavski at googlemail.com> wrote:

> Hello, everyone!
>
> I've investigated the implementation of the Dual-Pivot Quicksort
> which is used for sorting primitives and here is my result:
>
> http://cr.openjdk.java.net/~alanb/6947216/webrev.00
>
> New implementation of Dual-Pivot Quicksort is faster
> than previous one of 12% for client VM and few percents for
> server VM. Tests on Bentley's test suite (Win XP, JDK 7,
> build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
> for client VM and 0.98-1.00 for server VM.
>
> In compare with sorting from JDK 6 by Jon L. Bentley and
> M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
> (client / server).
>
> See the execution time for sorting array of 2`000`000 int elements
> 50 times, client / server VM, in milliseconds:
>
> random
>  new: 16723 18776
> jdk7: 17571 18975
> jdk6: 22241 26524
>
> ascendant
>  new:  3541  4702
> jdk7:  4486  4669
> jdk6:  8667  7978
>
> descendant
>  new:  3854  4907
> jdk7:  4942  5034
> jdk6:  8787  8243
>
> equal
>  new:   234   281
> jdk7:   291   230
> jdk6:   602  1018
>
> organ pipes
>  new:  7673  8613
> jdk7:  8167  8993
> jdk6: 11902 14044
>
> stagger 1
>  new:  7648  8591
> jdk7:  8161  8979
> jdk6: 11908 13810
>
> stagger 2
>  new:  8349  9299
> jdk7: 10968 11916
> jdk6: 12194 14734
>
> stagger 4
>  new:  8475  9622
> jdk7:  9221  9682
> jdk6: 10523 12006
>
> stagger 8
>  new:  9321 10689
> jdk7: 11125 12387
> jdk6: 13829 16214
>
> period 1..2
>  new:   758   751
> jdk7:   870   754
> jdk6:  1038  1227
>
> period 1..4
>  new:  1004   963
> jdk7:  1365  1209
> jdk6:  1511  1847
>
> period 1..8
>  new:  1588  1573
> jdk7:  1599  1790
> jdk6:  2602  3045
>
> random 1..2
>  new:  1327  1125
> jdk7:  1362  1496
> jdk6:  1531  2182
>
> random 1..4
>  new:  1830  2118
> jdk7:  1851  2236
> jdk6:  2292  3025
>
> where stagger(m) is array like a[i] = i * (m + 1) % length.
>
> The summary of changes is:
>
> 1. For sorting small arrays is used insertion sort with sentinel
> instead of traditional, which has the structure:
>
> for (int i = left + 1; i <= right; i++) {
>    for (j = i; j > left && a[j-1] > a[j]; j--) {
>        swap(a[i], a[j-1]);
>    }
> }
>
> Note that range check j > left is performed on each iteration,
> but really helps very rare. To avoid this expensive range check,
> it was suggested to set minimum value (negative infinity) on the
> first position. This type of suggestion is used in new version:
>
> if left bound > 0, we can put sentinel on a[left - 1], do insertion
> sort without expensive check range, and then restore a[left - 1]
> value. If left == 0, traditional insertion sort is used. Please,
> look at the webrev for details.
>
> 2. In previous implementation 5 evenly spaced elements
>
> sixth = length / 6;
> a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]
>
> were used as candidates of pivots elements. This case is very
> sensitive for period inputs, especially by 6. The new suggestion
> is to take 5 center evenly spaced elements like this:
>
> int seventh = length / 7;
> int midpoint = (left + right) >>> 1;
>
> a[midpoint - 2 * seventh], a[midpoint -     seventh], a[midpoint],
> a[midpoint +     seventh], a[midpoint + 2 * seventh]
>
> and moreover, the seventh is calculated inexpensively:
> seventh = (length >>> 3) + (length >>> 6) + 1;
>
> This schema works the same on random, ascendant, descendant, equal
> inputs, but much better for period / stagger.
>
> 3. The whole structure
>
> <choose pivots>
>
> if (pivotsDiffer) {
>    <do partitioning for two pivots>
> }
> else {
>    <do partitioning for one pivot>
> }
>
> <sort left and right parts>
>
> if (!pivotsDiffer) {
>    return;
> }
> <swap internal pivot values to ends>
>
> <sort center part>
>
> was modified to:
> ----------------
>
> <choose pivots>
>
> if (pivot1 < pivot2) {
>    <do partitioning for two pivots>
>    <swap pivots into their final positions>
>    <sort left and right parts>
>    <swap internal pivot values to ends>
>    <sort center part>
> }
> else {
>    <do partitioning for one pivot>
>    <sort left and right parts>
> }
>
> 4. Partitioning for both cases have not been changed at all.
>
> 5. Minor javadoc and format changes.
>
> Please, review new implementation,
> any comments / suggestions are welcome!
>
> Thank you,
> Vladimir
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100429/52a094f3/attachment.html>

From vladimir.iaroslavski at googlemail.com  Thu Apr 29 11:05:47 2010
From: vladimir.iaroslavski at googlemail.com (Vladimir Iaroslavski)
Date: Thu, 29 Apr 2010 15:05:47 +0400
Subject: New portion of improvements for Dual-Pivot Quicksort
In-Reply-To: <t2pf5b3aa371004290118z56814bbsac4f42ab26ffd2c6@mail.gmail.com>
References: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
	<t2pf5b3aa371004290118z56814bbsac4f42ab26ffd2c6@mail.gmail.com>
Message-ID: <z2oe04523d1004290405pafa876b4y42644f1d12426d3c@mail.gmail.com>

Hello,

Thank you for pointing to this check, I have run test again and it
works little it faster
(this check was necessary for old version of insertion sort):

60.90% / 48.35% instead of 61.00% /  51.00% (client / server).

I attached new version of the class (+ minor javdoc changes).

Best regards,
Vladimir

On Thu, Apr 29, 2010 at 12:18 PM, Goktug Gokdogan <gokdogan at gmail.com> wrote:
> One quick comment: you can consider moving trivial array check so it will be
> executed only if insertion sort threshold check passes:
>
> if (length < INSERTION_SORT_THRESHOLD) {
>   if (length < 2) {
>     return;
>   }
>   ....
>
> On Mon, Apr 26, 2010 at 2:50 PM, Vladimir Iaroslavski
> <vladimir.iaroslavski at googlemail.com> wrote:
>>
>> Hello, everyone!
>>
>> I've investigated the implementation of the Dual-Pivot Quicksort
>> which is used for sorting primitives and here is my result:
>>
>> http://cr.openjdk.java.net/~alanb/6947216/webrev.00
>>
>> New implementation of Dual-Pivot Quicksort is faster
>> than previous one of 12% for client VM and few percents for
>> server VM. Tests on Bentley's test suite (Win XP, JDK 7,
>> build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
>> for client VM and 0.98-1.00 for server VM.
>>
>> In compare with sorting from JDK 6 by Jon L. Bentley and
>> M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
>> (client / server).
>>
>> See the execution time for sorting array of 2`000`000 int elements
>> 50 times, client / server VM, in milliseconds:
>>
>> random
>> ?new: 16723 18776
>> jdk7: 17571 18975
>> jdk6: 22241 26524
>>
>> ascendant
>> ?new: ?3541 ?4702
>> jdk7: ?4486 ?4669
>> jdk6: ?8667 ?7978
>>
>> descendant
>> ?new: ?3854 ?4907
>> jdk7: ?4942 ?5034
>> jdk6: ?8787 ?8243
>>
>> equal
>> ?new: ? 234 ? 281
>> jdk7: ? 291 ? 230
>> jdk6: ? 602 ?1018
>>
>> organ pipes
>> ?new: ?7673 ?8613
>> jdk7: ?8167 ?8993
>> jdk6: 11902 14044
>>
>> stagger 1
>> ?new: ?7648 ?8591
>> jdk7: ?8161 ?8979
>> jdk6: 11908 13810
>>
>> stagger 2
>> ?new: ?8349 ?9299
>> jdk7: 10968 11916
>> jdk6: 12194 14734
>>
>> stagger 4
>> ?new: ?8475 ?9622
>> jdk7: ?9221 ?9682
>> jdk6: 10523 12006
>>
>> stagger 8
>> ?new: ?9321 10689
>> jdk7: 11125 12387
>> jdk6: 13829 16214
>>
>> period 1..2
>> ?new: ? 758 ? 751
>> jdk7: ? 870 ? 754
>> jdk6: ?1038 ?1227
>>
>> period 1..4
>> ?new: ?1004 ? 963
>> jdk7: ?1365 ?1209
>> jdk6: ?1511 ?1847
>>
>> period 1..8
>> ?new: ?1588 ?1573
>> jdk7: ?1599 ?1790
>> jdk6: ?2602 ?3045
>>
>> random 1..2
>> ?new: ?1327 ?1125
>> jdk7: ?1362 ?1496
>> jdk6: ?1531 ?2182
>>
>> random 1..4
>> ?new: ?1830 ?2118
>> jdk7: ?1851 ?2236
>> jdk6: ?2292 ?3025
>>
>> where stagger(m) is array like a[i] = i * (m + 1) % length.
>>
>> The summary of changes is:
>>
>> 1. For sorting small arrays is used insertion sort with sentinel
>> instead of traditional, which has the structure:
>>
>> for (int i = left + 1; i <= right; i++) {
>> ? ?for (j = i; j > left && a[j-1] > a[j]; j--) {
>> ? ? ? ?swap(a[i], a[j-1]);
>> ? ?}
>> }
>>
>> Note that range check j > left is performed on each iteration,
>> but really helps very rare. To avoid this expensive range check,
>> it was suggested to set minimum value (negative infinity) on the
>> first position. This type of suggestion is used in new version:
>>
>> if left bound > 0, we can put sentinel on a[left - 1], do insertion
>> sort without expensive check range, and then restore a[left - 1]
>> value. If left == 0, traditional insertion sort is used. Please,
>> look at the webrev for details.
>>
>> 2. In previous implementation 5 evenly spaced elements
>>
>> sixth = length / 6;
>> a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]
>>
>> were used as candidates of pivots elements. This case is very
>> sensitive for period inputs, especially by 6. The new suggestion
>> is to take 5 center evenly spaced elements like this:
>>
>> int seventh = length / 7;
>> int midpoint = (left + right) >>> 1;
>>
>> a[midpoint - 2 * seventh], a[midpoint - ? ? seventh], a[midpoint],
>> a[midpoint + ? ? seventh], a[midpoint + 2 * seventh]
>>
>> and moreover, the seventh is calculated inexpensively:
>> seventh = (length >>> 3) + (length >>> 6) + 1;
>>
>> This schema works the same on random, ascendant, descendant, equal
>> inputs, but much better for period / stagger.
>>
>> 3. The whole structure
>>
>> <choose pivots>
>>
>> if (pivotsDiffer) {
>> ? ?<do partitioning for two pivots>
>> }
>> else {
>> ? ?<do partitioning for one pivot>
>> }
>>
>> <sort left and right parts>
>>
>> if (!pivotsDiffer) {
>> ? ?return;
>> }
>> <swap internal pivot values to ends>
>>
>> <sort center part>
>>
>> was modified to:
>> ----------------
>>
>> <choose pivots>
>>
>> if (pivot1 < pivot2) {
>> ? ?<do partitioning for two pivots>
>> ? ?<swap pivots into their final positions>
>> ? ?<sort left and right parts>
>> ? ?<swap internal pivot values to ends>
>> ? ?<sort center part>
>> }
>> else {
>> ? ?<do partitioning for one pivot>
>> ? ?<sort left and right parts>
>> }
>>
>> 4. Partitioning for both cases have not been changed at all.
>>
>> 5. Minor javadoc and format changes.
>>
>> Please, review new implementation,
>> any comments / suggestions are welcome!
>>
>> Thank you,
>> Vladimir
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DualPivotQuicksort.java
Type: application/octet-stream
Size: 97083 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100429/2fd27a4a/DualPivotQuicksort.java>

From Ulf.Zibis at gmx.de  Thu Apr 29 18:07:05 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Thu, 29 Apr 2010 20:07:05 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD6F64E.3050909@gmx.de>
References: <4BD00250.3020206@oracle.com>
	<4BD05426.7030309@gmx.de>	<4BD228A3.7010103@oracle.com>
	<4BD63F9C.9010906@gmx.de>	<4BD66738.90806@oracle.com>
	<4BD6F64E.3050909@gmx.de>
Message-ID: <4BD9CAC9.6090207@gmx.de>

Am 24.04.2010 01:09, schrieb Xueming Shen:
>
> Yes, the final table takes about 500k, we might consider to use a 
> weakref or something, if memory really
> a concern. But the table will get initialized only if you invoke 
> Character.getName(),

Sherman, how did you compute that value:
- A Map.Entry object counts 24 bytes (40 on 64-bit machine)
- An Integer object for the key counts 12 bytes (20 on 64-bit machine)
- A String object counts 36 + 2*length, so for average character name 
length of 24:
       84 bytes (98 on 64-bit machine)
--> one character name in HashMap would count including buckets overhead 
~135 bytes (~170 on 64-bit machine)
--> 20.000 character names would count ~2.7 MByte (~3.4 on 64-bit machine)


See my new version in attachment.

I estimate:
- for byte[] names: 480.000 bytes
- for int[][] indexes:
-- base array size with 4353 elements: 17.420 bytes
-- one int[] index for block with average length of 32: 140 bytes
-- sum: 626.700 bytes
over all sum: 1.106.700 bytes (pretty enough)
If the block offset would be smaller than 256, I guess it would be more 
less. (with the impact of little decreased performance)

- Initializing the indexes array should be *much* faster than filling 
the hash map.
- Retrieving an index should be little faster or equivalent, but 
instantiation of one new String object must be added.

We could go further:
- separate caches (and data files) for the 17 Unicode planes
- calculate short 1/2-byte keys for textual words and frequent phrases. 
I estimate, there are 1000..4000 different words and 100..300 redundant 
phrases in the data.
Are you interested in that ?

We could add (Attention: CCC change) a cacheCharacterNames(boolean 
yesNo) method to serve users, which excessively need this functionality.

-Ulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100429/946204eb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CharacterName2.java
Type: java/*
Size: 4401 bytes
Desc: not available
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100429/946204eb/CharacterName2.java>

From jonathan.gibbons at sun.com  Thu Apr 29 21:26:49 2010
From: jonathan.gibbons at sun.com (jonathan.gibbons at sun.com)
Date: Thu, 29 Apr 2010 21:26:49 +0000
Subject: hg: jdk7/tl/langtools: 6948251: need to quote args in langtools
	launcher script
Message-ID: <20100429212653.A14CC44370@hg.openjdk.java.net>

Changeset: 32675c68a3ca
Author:    jjg
Date:      2010-04-29 14:25 -0700
URL:       http://hg.openjdk.java.net/jdk7/tl/langtools/rev/32675c68a3ca

6948251: need to quote args in langtools launcher script
Reviewed-by: darcy

! src/share/bin/launcher.sh-template



From Ulf.Zibis at gmx.de  Fri Apr 30 01:31:46 2010
From: Ulf.Zibis at gmx.de (Ulf Zibis)
Date: Fri, 30 Apr 2010 03:31:46 +0200
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD9CAC9.6090207@gmx.de>
References: <4BD00250.3020206@oracle.com>	<4BD05426.7030309@gmx.de>	<4BD228A3.7010103@oracle.com>	<4BD63F9C.9010906@gmx.de>	<4BD66738.90806@oracle.com>	<4BD6F64E.3050909@gmx.de>
	<4BD9CAC9.6090207@gmx.de>
Message-ID: <4BDA3302.7020805@gmx.de>

I have corrected the statistics:

current code from Sherman:
- A Map.Entry object counts 24 bytes (40 on 64-bit machine)
- An Integer object for the key counts 12 bytes (20 on 64-bit machine)
- A String object counts 36 + 2*length, so for average character name 
length of 26:
       88 bytes (102 on 64-bit machine)
--> one character name in HashMap would count including buckets overhead 
~140 bytes (~170 on 64-bit machine)
--> 20.000 character names would count ~2.8 MByte (~3.4 on 64-bit machine)


Measures, resulting from my latest version:
- for byte[] names: 509.554 bytes
- for int[][] indexes:
-- base array size with 4353 elements: 17.420 bytes
-- one int[] index for block with average length of 220: 892 bytes
-- sum: 17.420 + 97* 892 bytes = 103.944 bytes
over all sum: 613.498 bytes (pretty enough)


I did some statistics:

total name bytes: 509554
total names count: 19336
average name length: 26,35
used blocks: 97
average block length: 220,89
total words count: 7352
total words chars: 37427
     word  LETTER               occurs 5777 times
     word  WITH                 occurs 2271 times
     word  SMALL                occurs 2088 times
     word  SYLLABLE             occurs 1991 times
     word  SIGN                 occurs 1853 times
     word  CAPITAL              occurs 1578 times
     word  LATIN                occurs 1261 times
     word  YI                   occurs 1236 times
     word  CJK                  occurs 1160 times
     word  IDEOGRAPH            occurs 1094 times
     word  ARABIC               occurs 1047 times
     word  COMPATIBILITY        occurs 1009 times
     word  MATHEMATICAL         occurs 1007 times
     word  CUNEIFORM            occurs 982 times
     word  SYMBOL               occurs 962 times
     word  FORM                 occurs 798 times
     word  SYLLABICS            occurs 630 times
     word  CANADIAN             occurs 630 times
     word  BOLD                 occurs 567 times
     word  GREEK                occurs 517 times
     word  AND                  occurs 508 times
     word  LIGATURE             occurs 508 times
     word  DIGIT                occurs 506 times
     word  MUSICAL              occurs 498 times
     word  TIMES                occurs 492 times
     word  ETHIOPIC             occurs 461 times
     word  HANGUL               occurs 446 times
     word  ITALIC               occurs 423 times
     word  CYRILLIC             occurs 403 times
     word  RADICAL              occurs 385 times
     word  ABOVE                occurs 379 times
     word  SANS                 occurs 368 times
     word  -SERIF               occurs 368 times
     word  VOWEL                occurs 357 times
     word  ARROW                occurs 338 times
     word  DOTS                 occurs 328 times
     word  RIGHT                occurs 326 times
     word  FOR                  occurs 321 times
     word  LEFT                 occurs 316 times
     word  CIRCLED              occurs 312 times
     word  DOUBLE               occurs 308 times
     word  SQUARE               occurs 308 times
     word  VAI                  occurs 300 times
     word  FINAL                occurs 295 times
     word  COMBINING            occurs 293 times
     word  A                    occurs 284 times
     word  B                    occurs 277 times
     word  U                    occurs 269 times
     word  VARIATION            occurs 260 times
     word  SELECTOR             occurs 259 times
     word  PATTERN              occurs 257 times
     word  BRAILLE              occurs 256 times
     word  BYZANTINE            occurs 246 times
     word  O                    occurs 236 times
     word  ISOLATED             occurs 236 times
     word  VERTICAL             occurs 228 times
     word  BELOW                occurs 227 times
     word  DOT                  occurs 227 times
     word  KATAKANA             occurs 222 times
     word  MARK                 occurs 218 times
     word  E                    occurs 216 times
     word  KANGXI               occurs 214 times
     word  LINEAR               occurs 211 times
     word  MODIFIER             occurs 207 times
     word  TIBETAN              occurs 201 times
     word  TWO                  occurs 200 times
     word  I                    occurs 199 times
     word  STROKE               occurs 196 times
     word  MEEM                 occurs 192 times
     word  INITIAL              occurs 177 times
     word  WHITE                occurs 177 times
     word  CARRIER              occurs 175 times
     word  YEH                  occurs 174 times
     word  TO                   occurs 173 times
     word  BLACK                occurs 173 times
     word  ONE                  occurs 165 times
     word  NUMBER               occurs 160 times
     word  MONGOLIAN            occurs 156 times
     word  MYANMAR              occurs 156 times
     word  THREE                occurs 154 times
     word  HOOK                 occurs 152 times
     word  COPTIC               occurs 150 times
     word  KHMER                occurs 146 times
     word  TILE                 occurs 145 times
     word  BOX                  occurs 143 times
     word  PLUS                 occurs 142 times
     word  HORIZONTAL           occurs 137 times
     word  BRACKET              occurs 135 times
     word  HEBREW               occurs 133 times
     word  RIGHTWARDS           occurs 131 times
     word  OF                   occurs 128 times
     word  UP                   occurs 128 times
     word  DRAWINGS             occurs 128 times
     word  KA                   occurs 127 times
     word  ALEF                 occurs 126 times
     word  DOWN                 occurs 125 times
     word  OLD                  occurs 124 times
     word  HALFWIDTH            occurs 122 times
     word  FOUR                 occurs 121 times
     word  GEORGIAN             occurs 121 times
     word  BAR                  occurs 121 times
     word  BALINESE             occurs 121 times
     word  -THAN                occurs 120 times
     word  -CREE                occurs 119 times
     word  L                    occurs 117 times
     word  R                    occurs 117 times
     word  IDEOGRAM             occurs 117 times
     word  HEAVY                occurs 117 times
     word  EQUAL                occurs 115 times
     word  TAI                  occurs 115 times
     word  IDEOGRAPHIC          occurs 115 times
     word  WEST                 occurs 113 times
     word  PARENTHESIZED        occurs 113 times
     word  N                    occurs 112 times
     word  DEVANAGARI           occurs 112 times
     word  FIVE                 occurs 110 times
     word  SCRIPT               occurs 109 times
     word  TAG                  occurs 105 times
     word  HAH                  occurs 104 times
     word  FULLWIDTH            occurs 103 times
     word  TILDE                occurs 101 times
     word  OVER                 occurs 101 times
     word  LIGHT                occurs 100 times
     word  CHARACTER            occurs 100 times
     word  DOMINO               occurs 100 times
     word  NUMERIC              occurs 99 times
     word  LEFTWARDS            occurs 99 times
     word  FRAKTUR              occurs 99 times
     word  HALF                 occurs 98 times
     word  S                    occurs 97 times
     word  MALAYALAM            occurs 95 times
     word  GLAGOLITIC           occurs 94 times
     word  C                    occurs 93 times
     word  JEEM                 occurs 93 times
     word  TELUGU               occurs 93 times
     word  MEDIAL               occurs 91 times
     word  CHOSEONG             occurs 91 times
     word  ACUTE                occurs 91 times
     word  ARMENIAN             occurs 91 times
     word  BENGALI              occurs 91 times
     word  TONE                 occurs 90 times
     word  OR                   occurs 89 times
     word  HIRAGANA             occurs 89 times
     word  HA                   occurs 87 times
     word  THAI                 occurs 87 times
     word  Z                    occurs 86 times
     word  CIRCLE               occurs 86 times
     word  KANNADA              occurs 86 times
     word  Y                    occurs 85 times
     word  CHEROKEE             occurs 85 times
     word  EIGHT                occurs 84 times
     word  ORIYA                occurs 84 times
     word  GUJARATI             occurs 83 times
     word  CHAM                 occurs 83 times
     word  SIX                  occurs 83 times
     word  DASIA                occurs 83 times
     word  JONGSEONG            occurs 82 times
     word  M                    occurs 81 times
     word  H                    occurs 81 times
     word  T                    occurs 81 times
     word  SAURASHTRA           occurs 81 times
     word  TETRAGRAM            occurs 81 times
     word  RUNIC                occurs 81 times
     word  NEW                  occurs 81 times
     word  DESERET              occurs 80 times
     word  SINHALA              occurs 80 times
     word  LUE                  occurs 80 times
     word  D                    occurs 79 times
     word  G                    occurs 79 times
     word  V                    occurs 79 times
     word  NOTATION             occurs 79 times
     word  SYRIAC               occurs 79 times
     word  CIRCUMFLEX           occurs 79 times
     word  PSILI                occurs 79 times
     word  GURMUKHI             occurs 79 times
     word  SEVEN                occurs 78 times
     word  NINE                 occurs 77 times
     word  VOCALIC              occurs 77 times
     word  LONG                 occurs 74 times
     word  LINE                 occurs 74 times
     word  LEPCHA               occurs 74 times
     word  K                    occurs 73 times
     word  DIAERESIS            occurs 73 times
     word  -STRUCK              occurs 72 times
     word  HAMZA                occurs 72 times
     word  TAMIL                occurs 72 times
     word  APL                  occurs 70 times
     word  FUNCTIONAL           occurs 70 times
     word  TELEGRAPH            occurs 69 times
     word  MAKSURA              occurs 69 times
     word  MACRON               occurs 68 times
     word  ALPHA                occurs 68 times
     word  GRAVE                occurs 68 times
     word  P                    occurs 67 times
     word  OMEGA                occurs 67 times
     word  ACCENT               occurs 67 times
     word  JUNGSEONG            occurs 67 times
     word  LIMBU                occurs 66 times
     word  BARB                 occurs 66 times
     word  TRIANGLE             occurs 66 times
     word  LOW                  occurs 66 times
     word  KHAROSHTHI           occurs 65 times
     word  BOPOMOFO             occurs 65 times
     word  LAO                  occurs 65 times
     word  NOT                  occurs 65 times
     word  RA                   occurs 64 times
     word  YA                   occurs 64 times
     word  HEXAGRAM             occurs 64 times
     word  HARPOON              occurs 64 times
     word  TA                   occurs 63 times
     word  REVERSED             occurs 63 times
     word  X                    occurs 62 times
     word  ANGLE                occurs 62 times
     word  MA                   occurs 62 times
     word  HIGH                 occurs 62 times
     word  MONOSPACE            occurs 62 times
     word  OXIA                 occurs 62 times
     word  VARIA                occurs 62 times
     word  GREATER              occurs 62 times
     word  J                    occurs 61 times
     word  PA                   occurs 61 times
     word  LI                   occurs 61 times
     word  KHAH                 occurs 61 times
     word  LAGAB                occurs 61 times
     word  LESS                 occurs 61 times
     word  W                    occurs 59 times
     word  LA                   occurs 59 times
     word  LOWER                occurs 59 times
     word  NKO                  occurs 59 times
     word  NUMERAL              occurs 58 times
     word  LAM                  occurs 58 times
     word  TURNED               occurs 58 times
     word  F                    occurs 57 times
     word  DA                   occurs 57 times
     word  AEGEAN               occurs 57 times
     word  SHORT                occurs 57 times
     word  GA2                  occurs 56 times
     word  PHAGS                occurs 56 times
     word  OPEN                 occurs 56 times
     word  NA                   occurs 56 times
     word  ETA                  occurs 56 times
     word  -PA                  occurs 56 times
     word  STOP                 occurs 56 times
     word  SUNDANESE            occurs 55 times
     word  CYPRIOT              occurs 55 times
     word  BREVE                occurs 55 times
     word  TIFINAGH             occurs 55 times
     word  IOTA                 occurs 54 times
     word  ACROPHONIC           occurs 53 times
     word  SA                   occurs 53 times
     word  PERSIAN              occurs 53 times
     word  ZERO                 occurs 53 times
     word  UPPER                occurs 53 times
     word  ROMAN                occurs 52 times
     word  SUBJOINED            occurs 52 times
     word  NOON                 occurs 52 times
BUILD SUCCESSFUL (total time: 14 seconds)


-Ulf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100430/ba319e9c/attachment.html>

From dmytro_sheyko at hotmail.com  Fri Apr 30 13:03:31 2010
From: dmytro_sheyko at hotmail.com (Dmytro Sheyko)
Date: Fri, 30 Apr 2010 20:03:31 +0700
Subject: New portion of improvements for Dual-Pivot Quicksort
In-Reply-To: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
References: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
Message-ID: <SNT137-w47313C397D90740DFEF9418A000@phx.gbl>


Hello Vladimir,

Could you remind me where can I find sources for the test suite?

Thanks,
Dmytro

> Date: Tue, 27 Apr 2010 01:50:08 +0400
> Subject: New portion of improvements for Dual-Pivot Quicksort
> From: vladimir.iaroslavski at googlemail.com
> To: core-libs-dev at openjdk.java.net
> 
> Hello, everyone!
> 
> I've investigated the implementation of the Dual-Pivot Quicksort
> which is used for sorting primitives and here is my result:
> 
> http://cr.openjdk.java.net/~alanb/6947216/webrev.00
> 
> New implementation of Dual-Pivot Quicksort is faster
> than previous one of 12% for client VM and few percents for
> server VM. Tests on Bentley's test suite (Win XP, JDK 7,
> build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
> for client VM and 0.98-1.00 for server VM.
> 
> In compare with sorting from JDK 6 by Jon L. Bentley and
> M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
> (client / server).
> 
> See the execution time for sorting array of 2`000`000 int elements
> 50 times, client / server VM, in milliseconds:
> 
> random
>  new: 16723 18776
> jdk7: 17571 18975
> jdk6: 22241 26524
> 
> ascendant
>  new:  3541  4702
> jdk7:  4486  4669
> jdk6:  8667  7978
> 
> descendant
>  new:  3854  4907
> jdk7:  4942  5034
> jdk6:  8787  8243
> 
> equal
>  new:   234   281
> jdk7:   291   230
> jdk6:   602  1018
> 
> organ pipes
>  new:  7673  8613
> jdk7:  8167  8993
> jdk6: 11902 14044
> 
> stagger 1
>  new:  7648  8591
> jdk7:  8161  8979
> jdk6: 11908 13810
> 
> stagger 2
>  new:  8349  9299
> jdk7: 10968 11916
> jdk6: 12194 14734
> 
> stagger 4
>  new:  8475  9622
> jdk7:  9221  9682
> jdk6: 10523 12006
> 
> stagger 8
>  new:  9321 10689
> jdk7: 11125 12387
> jdk6: 13829 16214
> 
> period 1..2
>  new:   758   751
> jdk7:   870   754
> jdk6:  1038  1227
> 
> period 1..4
>  new:  1004   963
> jdk7:  1365  1209
> jdk6:  1511  1847
> 
> period 1..8
>  new:  1588  1573
> jdk7:  1599  1790
> jdk6:  2602  3045
> 
> random 1..2
>  new:  1327  1125
> jdk7:  1362  1496
> jdk6:  1531  2182
> 
> random 1..4
>  new:  1830  2118
> jdk7:  1851  2236
> jdk6:  2292  3025
> 
> where stagger(m) is array like a[i] = i * (m + 1) % length.
> 
> The summary of changes is:
> 
> 1. For sorting small arrays is used insertion sort with sentinel
> instead of traditional, which has the structure:
> 
> for (int i = left + 1; i <= right; i++) {
>     for (j = i; j > left && a[j-1] > a[j]; j--) {
>         swap(a[i], a[j-1]);
>     }
> }
> 
> Note that range check j > left is performed on each iteration,
> but really helps very rare. To avoid this expensive range check,
> it was suggested to set minimum value (negative infinity) on the
> first position. This type of suggestion is used in new version:
> 
> if left bound > 0, we can put sentinel on a[left - 1], do insertion
> sort without expensive check range, and then restore a[left - 1]
> value. If left == 0, traditional insertion sort is used. Please,
> look at the webrev for details.
> 
> 2. In previous implementation 5 evenly spaced elements
> 
> sixth = length / 6;
> a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]
> 
> were used as candidates of pivots elements. This case is very
> sensitive for period inputs, especially by 6. The new suggestion
> is to take 5 center evenly spaced elements like this:
> 
> int seventh = length / 7;
> int midpoint = (left + right) >>> 1;
> 
> a[midpoint - 2 * seventh], a[midpoint -     seventh], a[midpoint],
> a[midpoint +     seventh], a[midpoint + 2 * seventh]
> 
> and moreover, the seventh is calculated inexpensively:
> seventh = (length >>> 3) + (length >>> 6) + 1;
> 
> This schema works the same on random, ascendant, descendant, equal
> inputs, but much better for period / stagger.
> 
> 3. The whole structure
> 
> <choose pivots>
> 
> if (pivotsDiffer) {
>     <do partitioning for two pivots>
> }
> else {
>     <do partitioning for one pivot>
> }
> 
> <sort left and right parts>
> 
> if (!pivotsDiffer) {
>     return;
> }
> <swap internal pivot values to ends>
> 
> <sort center part>
> 
> was modified to:
> ----------------
> 
> <choose pivots>
> 
> if (pivot1 < pivot2) {
>     <do partitioning for two pivots>
>     <swap pivots into their final positions>
>     <sort left and right parts>
>     <swap internal pivot values to ends>
>     <sort center part>
> }
> else {
>     <do partitioning for one pivot>
>     <sort left and right parts>
> }
> 
> 4. Partitioning for both cases have not been changed at all.
> 
> 5. Minor javadoc and format changes.
> 
> Please, review new implementation,
> any comments / suggestions are welcome!
> 
> Thank you,
> Vladimir
 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with Microsoft?s powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openjdk.java.net/pipermail/core-libs-dev/attachments/20100430/b0ea980f/attachment.html>

From vladimir.iaroslavski at googlemail.com  Fri Apr 30 13:11:10 2010
From: vladimir.iaroslavski at googlemail.com (Vladimir Iaroslavski)
Date: Fri, 30 Apr 2010 17:11:10 +0400
Subject: New portion of improvements for Dual-Pivot Quicksort
In-Reply-To: <SNT137-w47313C397D90740DFEF9418A000@phx.gbl>
References: <u2ke04523d1004261450na8f3c1f7z5ce7a55ec39d6a9b@mail.gmail.com>
	<SNT137-w47313C397D90740DFEF9418A000@phx.gbl>
Message-ID: <g2ze04523d1004300611hbb3f7352t1cc781935712bf1d@mail.gmail.com>

Hello Dmytro,

Please, see file test/java/util/Arrays/Sorting.java in the
JDK repository, It contains unit tests.

Thank you,
Vladimir

2010/4/30 Dmytro Sheyko <dmytro_sheyko at hotmail.com>:
> Hello Vladimir,
>
> Could you remind me where can I find sources for the test suite?
>
> Thanks,
> Dmytro
>
>> Date: Tue, 27 Apr 2010 01:50:08 +0400
>> Subject: New portion of improvements for Dual-Pivot Quicksort
>> From: vladimir.iaroslavski at googlemail.com
>> To: core-libs-dev at openjdk.java.net
>>
>> Hello, everyone!
>>
>> I've investigated the implementation of the Dual-Pivot Quicksort
>> which is used for sorting primitives and here is my result:
>>
>> http://cr.openjdk.java.net/~alanb/6947216/webrev.00
>>
>> New implementation of Dual-Pivot Quicksort is faster
>> than previous one of 12% for client VM and few percents for
>> server VM. Tests on Bentley's test suite (Win XP, JDK 7,
>> build 1.7.0-ea-b84, n = 1000000) show geometry mean 0.88
>> for client VM and 0.98-1.00 for server VM.
>>
>> In compare with sorting from JDK 6 by Jon L. Bentley and
>> M. Douglas McIlroy's (with one pivot) the ratio is 0.61 / 0.50
>> (client / server).
>>
>> See the execution time for sorting array of 2`000`000 int elements
>> 50 times, client / server VM, in milliseconds:
>>
>> random
>> new: 16723 18776
>> jdk7: 17571 18975
>> jdk6: 22241 26524
>>
>> ascendant
>> new: 3541 4702
>> jdk7: 4486 4669
>> jdk6: 8667 7978
>>
>> descendant
>> new: 3854 4907
>> jdk7: 4942 5034
>> jdk6: 8787 8243
>>
>> equal
>> new: 234 281
>> jdk7: 291 230
>> jdk6: 602 1018
>>
>> organ pipes
>> new: 7673 8613
>> jdk7: 8167 8993
>> jdk6: 11902 14044
>>
>> stagger 1
>> new: 7648 8591
>> jdk7: 8161 8979
>> jdk6: 11908 13810
>>
>> stagger 2
>> new: 8349 9299
>> jdk7: 10968 11916
>> jdk6: 12194 14734
>>
>> stagger 4
>> new: 8475 9622
>> jdk7: 9221 9682
>> jdk6: 10523 12006
>>
>> stagger 8
>> new: 9321 10689
>> jdk7: 11125 12387
>> jdk6: 13829 16214
>>
>> period 1..2
>> new: 758 751
>> jdk7: 870 754
>> jdk6: 1038 1227
>>
>> period 1..4
>> new: 1004 963
>> jdk7: 1365 1209
>> jdk6: 1511 1847
>>
>> period 1..8
>> new: 1588 1573
>> jdk7: 1599 1790
>> jdk6: 2602 3045
>>
>> random 1..2
>> new: 1327 1125
>> jdk7: 1362 1496
>> jdk6: 1531 2182
>>
>> random 1..4
>> new: 1830 2118
>> jdk7: 1851 2236
>> jdk6: 2292 3025
>>
>> where stagger(m) is array like a[i] = i * (m + 1) % length.
>>
>> The summary of changes is:
>>
>> 1. For sorting small arrays is used insertion sort with sentinel
>> instead of traditional, which has the structure:
>>
>> for (int i = left + 1; i <= right; i++) {
>> for (j = i; j > left && a[j-1] > a[j]; j--) {
>> swap(a[i], a[j-1]);
>> }
>> }
>>
>> Note that range check j > left is performed on each iteration,
>> but really helps very rare. To avoid this expensive range check,
>> it was suggested to set minimum value (negative infinity) on the
>> first position. This type of suggestion is used in new version:
>>
>> if left bound > 0, we can put sentinel on a[left - 1], do insertion
>> sort without expensive check range, and then restore a[left - 1]
>> value. If left == 0, traditional insertion sort is used. Please,
>> look at the webrev for details.
>>
>> 2. In previous implementation 5 evenly spaced elements
>>
>> sixth = length / 6;
>> a[sixth], a[2 * sixth], a[3 * sixth], a[4 * sixth], a[5 * sixth]
>>
>> were used as candidates of pivots elements. This case is very
>> sensitive for period inputs, especially by 6. The new suggestion
>> is to take 5 center evenly spaced elements like this:
>>
>> int seventh = length / 7;
>> int midpoint = (left + right) >>> 1;
>>
>> a[midpoint - 2 * seventh], a[midpoint - seventh], a[midpoint],
>> a[midpoint + seventh], a[midpoint + 2 * seventh]
>>
>> and moreover, the seventh is calculated inexpensively:
>> seventh = (length >>> 3) + (length >>> 6) + 1;
>>
>> This schema works the same on random, ascendant, descendant, equal
>> inputs, but much better for period / stagger.
>>
>> 3. The whole structure
>>
>> <choose pivots>
>>
>> if (pivotsDiffer) {
>> <do partitioning for two pivots>
>> }
>> else {
>> <do partitioning for one pivot>
>> }
>>
>> <sort left and right parts>
>>
>> if (!pivotsDiffer) {
>> return;
>> }
>> <swap internal pivot values to ends>
>>
>> <sort center part>
>>
>> was modified to:
>> ----------------
>>
>> <choose pivots>
>>
>> if (pivot1 < pivot2) {
>> <do partitioning for two pivots>
>> <swap pivots into their final positions>
>> <sort left and right parts>
>> <swap internal pivot values to ends>
>> <sort center part>
>> }
>> else {
>> <do partitioning for one pivot>
>> <sort left and right parts>
>> }
>>
>> 4. Partitioning for both cases have not been changed at all.
>>
>> 5. Minor javadoc and format changes.
>>
>> Please, review new implementation,
>> any comments / suggestions are welcome!
>>
>> Thank you,
>> Vladimir


From xueming.shen at oracle.com  Fri Apr 30 19:48:13 2010
From: xueming.shen at oracle.com (Xueming Shen)
Date: Fri, 30 Apr 2010 12:48:13 -0700
Subject: Unicode script support in Regex and Character class
In-Reply-To: <4BD52498.2090006@oracle.com>
References: <4BD00250.3020206@oracle.com>
	<o2w1ccfd1c11004241121lb9bb4814o2bdae377888fa82d@mail.gmail.com>
	<4BD3E7AC.2020801@oracle.com> <4BD52498.2090006@oracle.com>
Message-ID: <4BDB33FD.3030809@oracle.com>

Hi,

#4860714 has been closed as a dup (to workaround an internal process 
problem) of my newly created
#6948903 for the regex script support.

So here are the CCC drafts for

6945564: Unicode script support in Character class
6948903: Make Unicode scripts available for use in regular expressions

http://cr.openjdk.java.net/~sherman/script/6948903.htm
http://cr.openjdk.java.net/~sherman/script/6945564.htm

The blenderrevs are

http://cr.openjdk.java.net/~sherman/script/blenderrev_pattern.html
http://cr.openjdk.java.net/~sherman/script/blenderrev_ch.html

Thanks,
Sherman


Xueming Shen wrote:
>
> Can I assume we are all OK with at least the API part of the latest 
> webrev/blenderrev of
> the script support in j.l.Character and j.u.r.Pattern, including the 
> j.l.Chareacter.getName().
>
> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
> http://cr.openjdk.java.net/~sherman/script/webrev
>
> Okutsu-san, Yuka, can one of you help review the corresponding CCC at
> http://ccc.sfbay.sun.com/6945564?
>
> This is for the j.l.Character part only. I'm still trying to figure 
> out how to take over
> the ownership of 4860714 in CCC system, we have a placeholder for this 
> one in
> CCC back to 2003.
>
> Thanks,
> -Sherman
>
>
>
>
> Xueming Shen wrote:
>> Martin Buchholz wrote:
>>> Providing script support is obvious and non-controversial,
>>> because other regex programming environments provide it.
>>> Check that the behavior and syntax of the extension is
>>> consistent with e.g. ICU, python, and especially perl
>>> (5.12 just released!)
>>>
>>> http://perldoc.perl.org/perlunicode.html
>>>   
>>
>> \p{propName=propValue} is the unicode "compound form", which is 
>> supported in
>> perl 5.12. It also has a variant type \p{propName:propValue}. It was 
>> in my proposal,
>> but I removed it the last minutes. Two forms (\p{In/IsProp} and 
>> \p{propName=propValue}
>> should be good enough for now. Three is a little too much. We can 
>> always add it
>> in later, if desirable.
>>
>> \p{IsScript}, \p{Isgc}, \p{InBlock} are perl compatible as well.
>>
>>> I would add some documentation to the three special script values;
>>> their meaning is not obvious.
>>>
>>>   
>> I think it might be better to justt leave the detailed explain doc to 
>> the TR#24. The "script"
>> here in j.l.Character serves only the purpose of id, the API here 
>> should not be the place
>> to explain "what they really are".
>>
>>> For implementation, the character matching problem is in general
>>> equivalent to the problem of compiling a switch statement, which is
>>> known to be non-trivial.  Guava contains a CharMatcher class that
>>> tries to solve related problems.
>>>
>>> http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/CharMatcher.html 
>>>
>>>
>>> I'm thinking scripts and blocks should know about which ranges they 
>>> contain.
>>> In particular, \p{BlockName} should not need binary search at
>>> regex compile time or runtime.
>>>   
>> It definitely is desirable if we can avoid the binary-search lookup 
>> during at least the runtime. The
>> cost will be to keep a separate/redundant block/script->ranges table 
>> in regex itself.
>>
>>> ---
>>> There is one place you need to change
>>> key word => keyword
>>> ---
>>> InMongolian => {@code InMongolian}
>>> ---
>>>   
>>
>> Good catch, thanks!
>>
>>> I notice current Unicode block support in JDK is not updated to the
>>> latest standard.
>>> E.g. Samaritan is missing.
>>>
>>>   
>> The Character class has not been updated to the latest 5.20 yet. Yuka 
>> has a CCC pending for
>> that. My script data is from the 5.20.
>>
>>
>>> Martin
>>>
>>> On Thu, Apr 22, 2010 at 01:01, Xueming Shen 
>>> <xueming.shen at oracle.com> wrote:
>>>  
>>>> Hi,
>>>>
>>>> Here is the webrev of the proposal to add Unicode script support in 
>>>> regex
>>>> and j.l.Character.
>>>>
>>>> http://cr.openjdk.java.net/~sherman/script/webrev
>>>>
>>>> and the corresponding blenderrev
>>>>
>>>> http://cr.openjdk.java.net/~sherman/script/blenderrev.html
>>>>
>>>> Please comment on the APIs before I submit the CCC, especially
>>>>
>>>> (1) to use enum for the j.l.Character.UnicodeScript (compared to the
>>>> traditional j.l.c.Subset)
>>>> (2) the piggyback method j.l.c.getName() :-)
>>>> (3) the syntax for script constructs. In addition to the "normal"
>>>>    \p{InScriptName} and \P{InScriptName} for the script support
>>>>    I'm also adding
>>>>   \p{script=ScriptName} \P{script=ScriptName}  for the new script 
>>>> support
>>>>   \p{block=BlockName} \P{block=BlockName}  for the "existing" block 
>>>> support
>>>>   \p{general_category=CategoryName} 
>>>> \P{general_category=CategoryName} for
>>>> the "existing" gc
>>>>   Perl recently also started to accept this  \p{propName=propValue} 
>>>> Unicode
>>>> style.
>>>>   It opens the door for future "expanding", for example 
>>>> \p{name=XYZ} :-)
>>>> (4)and of course, the wording.
>>>>
>>>> Thanks,
>>>> Sherman
>>>>
>>>>
>>>>
>>>>     
>>
>>
>
>