<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am worried about the variability of the performance of Vector<E>. Worse, I am worried about how to explain to users the variability of the performance of Vector<E>.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I started from the second code example from https://docs.oracle.com/en/java/javase/25/docs/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html. I changed it to compute dot-product (reading two arrays and computing a scalar result) rather than
simple multiply (reading two arrays and writing a third array). When I look at the generated native instructions, I see that performance is dominated by the loop-carried dependence updating the result variable. The solution to that is to have each loop compute
several independent sum variables, using more vector registers. (That is what they are there for! That also gives the out-of-order engine in the core something to work with.) Using two sum variables runs almost two times as fast as using a single sum variable;
three sum variables runs faster than two sum variables; four sum variables runs faster than three sum variables, but there are diminishing returns. More interestingly, and more worryingly, the run time for four sum variables starts to show run-to-run variation.
Sometimes a run is fast, sometimes it is only a few percent slower than the fastest time, but some of the time (~30%) a run can be more than 25% slower than the fastest time. Worse, the run times are not a smooth curve between the fastest and slowest runs:
the distribution is step function. With no way to predict why some runs are slow. (JMH benchmarks available upon request.)</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have tracked the slowdown to the alignment of the input arrays to the L1 cache lines. For example, a `FloatVector.fromArray(FloatVector.SPECIES_PREFERRED, a, offset)` on a machine with 4 vector lanes will try to load 4 floats from the array `a`. If those
4 floats are all in the same cache line, life is good. If the 4 floats are split across 2 cache lines, the load takes longer. How much longer a load takes depends on the microarchitecture of the machine, where in the loads you cross the cache line boundary,
and probably other things a Java programmer cannot control. A C programmer can control the alignment of a float arroy by requesting an aligned allocation of the array. A Java programmer cannot control the alignment. The Java case is also complicated by
the header on the array, which can vary in size depending on run-time flags for the sizes of the fields in the array header.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Since the run-to-run variation is so large and hard to explain, I would like to open a discussion of what the JVM can do about aligning array data so that it can be used reliably by the `Vector<E>` mechanisms.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
An out-of-the-box experiment I tried was to run with `-XX:ObjectAlignmentInBytes=64` (on a machine with a 64-byte L1 cache line size). The hope was that every array of type E would be aligned the same way relative to the cache lines, so there would be no run-to-run
variation in performance. That worked. But increasing the object alignment for all objects wastes a lot of space.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Given that confirmation, I thought to try changing just the alignment of arrays of scalars. For example, by allocating filler objects before the array object to get the array base aligned on a cache line boundary. That only solves the problem for newly allocated
arrays but does not preserve the alignment when a garbage collector relocates the array. (I have no demographic data on the lifetime of arrays that are used with `Vector<E>`, but it would be disappointing to have good performance only until a collector relocates
your array and then have no way to recover the good performance.) Wave hands over large arrays and collectors that allocate large objects in humongous regions and then does not relocate those regions. Not all arrays of scalars should be in humongous regions.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The only positive (!) thing I have thought of that might help is that the high-order bit of the length field of an array is always 0, because the length field is an `int` and the maximum length of an array is `Integer.MAX_VALUE`. Maybe there is some way to
use the sign-bit of the length field to indicate that an array has an aligned base address. Establish that alignment at allocation time and re-establish it when an array is relocated. At the cost of computing the absolute value of the length field every
time it is read.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Aligning arrays to avoid occasional 25% performance loss seems like a worthwhile goal. I would like to open a discussion about how that end might be achieved.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
... peter</div>
</body>
</html>