<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,<br>
auto vector does not work particularly well in that loop, I tested
it before...</p>
<p>I was able to optimize it more and now I have better performance
than the scalar code with both AVX512 and AVX256.<br>
The code is now correct because both scalar and vector code
produces the same result.<br>
</p>
<div style="background-color:#1e1f22;color:#bcbec4">
<pre
style="font-family:'JetBrains Mono',monospace;font-size:9.8pt;"><span
style="color:#cf8e6d;">for </span>(<span style="color:#cf8e6d;">int </span>y = <span
style="color:#2aacb8;">0</span>; y < pixelInUseY; y++) {
<span style="color:#cf8e6d;">int </span>offsetY = yCoordinate + y;
<span style="color:#cf8e6d;">int </span>baseBufferOffset = (offsetY < <span
style="color:#c77dbb;">height</span>) ? (offsetY * <span
style="color:#c77dbb;">widthPlusStride</span>) : (<span
style="color:#c77dbb;">height </span>* <span
style="color:#c77dbb;">widthPlusStride</span>);
<span style="color:#cf8e6d;">for </span>(<span
style="color:#cf8e6d;">int </span>x = <span style="color:#2aacb8;">0</span>; x < pixelInUseX; x += <span
style="color:#c77dbb;">SPECIES</span>.length() * <span
style="color:#2aacb8;">3</span>) {
<span style="color:#cf8e6d;">int </span>offsetX = xCoordinate + x;
VectorMask<Integer> mask1 = <span style="color:#c77dbb;">SPECIES</span>.indexInRange(x, pixelInUseX);
VectorMask<Integer> mask2 = <span style="color:#c77dbb;">SPECIES</span>.indexInRange(x + <span
style="color:#c77dbb;">SPECIES</span>.length(), pixelInUseX);
VectorMask<Integer> mask3 = <span style="color:#c77dbb;">SPECIES</span>.indexInRange(x + <span
style="color:#2aacb8;">2 </span>* <span style="color:#c77dbb;">SPECIES</span>.length(), pixelInUseX);
IntVector rgbVector1 = IntVector.<span
style="font-style:italic;">fromMemorySegment</span>(<span
style="color:#c77dbb;">SPECIES</span>, <span
style="color:#c77dbb;">memorySegment</span>,
(<span style="color:#cf8e6d;">long</span>) (Math.<span
style="font-style:italic;">min</span>(offsetX, <span
style="color:#c77dbb;">widthPlusStride</span>) + baseBufferOffset) * Integer.<span
style="color:#c77dbb;font-style:italic;">BYTES</span>, ByteOrder.<span
style="font-style:italic;">nativeOrder</span>(), mask1);
IntVector rgbVector2 = IntVector.<span
style="font-style:italic;">fromMemorySegment</span>(<span
style="color:#c77dbb;">SPECIES</span>, <span
style="color:#c77dbb;">memorySegment</span>,
(<span style="color:#cf8e6d;">long</span>) (Math.<span
style="font-style:italic;">min</span>(offsetX + <span
style="color:#c77dbb;">SPECIES</span>.length(), <span
style="color:#c77dbb;">widthPlusStride</span>) + baseBufferOffset) * Integer.<span
style="color:#c77dbb;font-style:italic;">BYTES</span>, ByteOrder.<span
style="font-style:italic;">nativeOrder</span>(), mask2);
IntVector rgbVector3 = IntVector.<span
style="font-style:italic;">fromMemorySegment</span>(<span
style="color:#c77dbb;">SPECIES</span>, <span
style="color:#c77dbb;">memorySegment</span>,
(<span style="color:#cf8e6d;">long</span>) (Math.<span
style="font-style:italic;">min</span>(offsetX + <span
style="color:#2aacb8;">2 </span>* <span style="color:#c77dbb;">SPECIES</span>.length(), <span
style="color:#c77dbb;">widthPlusStride</span>) + baseBufferOffset) * Integer.<span
style="color:#c77dbb;font-style:italic;">BYTES</span>, ByteOrder.<span
style="font-style:italic;">nativeOrder</span>(), mask3);
IntVector rVector = rgbVector1.and(<span style="color:#2aacb8;">0xFF0000</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">16</span>)
.add(rgbVector2.and(<span style="color:#2aacb8;">0xFF0000</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">16</span>))
.add(rgbVector3.and(<span style="color:#2aacb8;">0xFF0000</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">16</span>));
IntVector gVector = rgbVector1.and(<span style="color:#2aacb8;">0x00FF00</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">8</span>)
.add(rgbVector2.and(<span style="color:#2aacb8;">0x00FF00</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">8</span>))
.add(rgbVector3.and(<span style="color:#2aacb8;">0x00FF00</span>).lanewise(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">LSHR</span>, <span
style="color:#2aacb8;">8</span>));
IntVector bVector = rgbVector1.and(<span style="color:#2aacb8;">0x0000FF</span>)
.add(rgbVector2.and(<span style="color:#2aacb8;">0x0000FF</span>))
.add(rgbVector3.and(<span style="color:#2aacb8;">0x0000FF</span>));
r += rVector.reduceLanes(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">ADD</span>);
g += gVector.reduceLanes(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">ADD</span>);
b += bVector.reduceLanes(VectorOperators.<span
style="color:#c77dbb;font-style:italic;">ADD</span>);
pickNumber += mask1.trueCount() + mask2.trueCount() + mask3.trueCount();
}
}</pre>
</div>
<p></p>
<p>Let's say 5 times more than the scalar version using AVX512. On
Linux differences are even bigger and the bigger is the initial
array to loop,<br>
the bigger is the performance increase.<br>
<br>
What I noticed is that using a mask helps a lot, using three masks
helps allllot more.<br>
From 3 to 4 masks there is nearly no difference, 5 masks brings
nothing more.<br>
<br>
I have tested it on an old 12th gen Intel laptop, on a more modern
14th gen one and on an AMD 7950X3D desktop (this one is AVX512
enabled).<br>
AVX256 is the one that get the biggest advantage from using 3
masks.</p>
<p>I understand that this may vary based on the specific use case
but I would say that I have found a sweet spot for my use case by
using three masks.</p>
<p>This API is a very nice addition to the Java ecosystem and it can
only improve over time because both Intel and AMD is investing in
improving their SIMD extensions.</p>
<p>Thank you for all the answers :)<br>
<br>
Regards,<br>
Davide<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 14/08/24 19:20, Daniel Lemire wrote:<br>
</div>
<blockquote type="cite"
cite="mid:51625e21-2785-4156-a8d5-c2609d289486@app.fastmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
<div>Have you checked whether Java autovectorizes your scalar
code?<br>
</div>
<div><br>
</div>
<div>- Daniel </div>
<div><br>
</div>
<div>On Wed, Aug 14, 2024, at 11:52, Davide Perini wrote:<br>
</div>
<blockquote type="cite" id="qt" style="">
<div>Pretty sad to admit that I was completely wrong.<br>
</div>
<div> My Vector code was faster because it was calculating less
numbers.<br>
</div>
<div> <br>
</div>
<div> The original code without vector api is this:<br>
</div>
<div
style="background-color:rgb(30, 31, 34);color:rgb(188, 190, 196);">
<pre
style="font-family:"JetBrains Mono", monospace;"><span
style="color:rgb(207, 142, 109);">for </span>(<span
style="color:rgb(207, 142, 109);">int </span>y = <span
style="color:rgb(42, 172, 184);">0</span>; y < pixelInUseY; y++) {
<span style="color:rgb(207, 142, 109);">for </span>(<span
style="color:rgb(207, 142, 109);">int </span>x = <span
style="color:rgb(42, 172, 184);">0</span>; x < pixelInUseX; x++) {
<span style="color:rgb(207, 142, 109);">int </span>offsetX = (xCoordinate + x);
<span style="color:rgb(207, 142, 109);">int </span>offsetY = (yCoordinate + y);
<span style="color:rgb(207, 142, 109);">int </span>bufferOffset = (Math.<i>min</i>(offsetX, <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>)) + ((offsetY < <span
style="color:rgb(199, 125, 187);">height</span>) ? (offsetY * <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>) : (<span
style="color:rgb(199, 125, 187);">height </span>* <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>));
<span style="color:rgb(207, 142, 109);">int </span>rgb = <span
style="color:rgb(199, 125, 187);">rgbBuffer</span>.get(Math.<i>min</i>(<span
style="color:rgb(199, 125, 187);">rgbBuffer</span>.capacity() - <span
style="color:rgb(42, 172, 184);">1</span>, bufferOffset));
r += rgb >> <span style="color:rgb(42, 172, 184);">16 </span>& <span
style="color:rgb(42, 172, 184);">0xFF</span>;
g += rgb >> <span style="color:rgb(42, 172, 184);">8 </span>& <span
style="color:rgb(42, 172, 184);">0xFF</span>;
b += rgb & <span style="color:rgb(42, 172, 184);">0xFF</span>;
pickNumber++;
}
}
</pre>
</div>
<div>the one with vector api that produces the same same result
is this:<br>
</div>
<div
style="background-color:rgb(30, 31, 34);color:rgb(188, 190, 196);">
<pre
style="font-family:"JetBrains Mono", monospace;">MemorySegment memorySegment = MemorySegment.<i>ofBuffer</i>(<span
style="color:rgb(199, 125, 187);">rgbBuffer</span>);
IntVector rVector = IntVector.<i>zero</i>(<span
style="color:rgb(199, 125, 187);">SPECIES</span>);
IntVector gVector = IntVector.<i>zero</i>(<span
style="color:rgb(199, 125, 187);">SPECIES</span>);
IntVector bVector = IntVector.<i>zero</i>(<span
style="color:rgb(199, 125, 187);">SPECIES</span>);
<span style="color:rgb(207, 142, 109);">long </span>maxOffset = (memorySegment.byteSize() - ((<span
style="color:rgb(207, 142, 109);">long</span>) <span
style="color:rgb(199, 125, 187);">SPECIES</span>.length() * Integer.<span
style="color:rgb(199, 125, 187);"><i>BYTES</i></span>));
<span style="color:rgb(207, 142, 109);">for </span>(<span
style="color:rgb(207, 142, 109);">int </span>y = <span
style="color:rgb(42, 172, 184);">0</span>; y < pixelInUseY; y++) {
<span style="color:rgb(207, 142, 109);">for </span>(<span
style="color:rgb(207, 142, 109);">int </span>x = <span
style="color:rgb(42, 172, 184);">0</span>; x < pixelInUseX; x+= <span
style="color:rgb(199, 125, 187);">SPECIES</span>.length()) {
<span style="color:rgb(207, 142, 109);">int </span>offsetX = (xCoordinate + x);
<span style="color:rgb(207, 142, 109);">int </span>offsetY = (yCoordinate + y);
<span style="color:rgb(207, 142, 109);">int </span>bufferOffset = (Math.<i>min</i>(offsetX, <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>)) + ((offsetY < <span
style="color:rgb(199, 125, 187);">height</span>) ? (offsetY * <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>) : (<span
style="color:rgb(199, 125, 187);">height </span>* <span
style="color:rgb(199, 125, 187);">widthPlusStride</span>));
bufferOffset = (<span style="color:rgb(207, 142, 109);">int</span>) Math.<i>min</i>(((<span
style="color:rgb(207, 142, 109);">long</span>) bufferOffset * Integer.<span
style="color:rgb(199, 125, 187);"><i>BYTES</i></span>), maxOffset);
IntVector rgbVector = IntVector.<i>fromMemorySegment</i>(<span
style="color:rgb(199, 125, 187);">SPECIES</span>, memorySegment, bufferOffset, ByteOrder.<i>nativeOrder</i>());
rVector = rVector.add(rgbVector.and(<span
style="color:rgb(42, 172, 184);">0xFF0000</span>).lanewise(VectorOperators.<span
style="color:rgb(199, 125, 187);"><i>LSHR</i></span>, <span
style="color:rgb(42, 172, 184);">16</span>));
gVector = gVector.add(rgbVector.and(<span
style="color:rgb(42, 172, 184);">0x00FF00</span>).lanewise(VectorOperators.<span
style="color:rgb(199, 125, 187);"><i>LSHR</i></span>, <span
style="color:rgb(42, 172, 184);">8</span>));
bVector = bVector.add(rgbVector.and(<span
style="color:rgb(42, 172, 184);">0x0000FF</span>));
pickNumber += <span style="color:rgb(199, 125, 187);">SPECIES</span>.length();
}
}
r = rVector.reduceLanes(VectorOperators.<span
style="color:rgb(199, 125, 187);"><i>ADD</i></span>);
g = gVector.reduceLanes(VectorOperators.<span
style="color:rgb(199, 125, 187);"><i>ADD</i></span>);
b = bVector.reduceLanes(VectorOperators.<span
style="color:rgb(199, 125, 187);"><i>ADD</i></span>);
</pre>
</div>
<div>and it's slower than the one without the vector API even
when using AVX512.<br>
</div>
<div> For big big big numbers AVX512 became a little bit faster
than standard code<br>
</div>
<div> but numbers must be bigger than what I need to use...<br>
</div>
<div> <br>
</div>
<div> Probably there are other optimizations that I can do to
that code but surpassing scalar performance isn't easy.<br>
</div>
<div> <br>
</div>
<div> Davide<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div class="qt-moz-cite-prefix">Il 13/08/2024 23:40, Paul Sandoz
ha scritto:<br>
</div>
<blockquote type="cite"
cite="mid:42E068A0-5D37-4A5A-A4D3-FF316498D4F2@oracle.com">
<pre class="qt-moz-quote-pre">Hi,
Glad the memory segment access worked.
I realize I did not fully understand your problem and led you astray! The second snippet will generate more instructions.
IIUC you are sampling the red, green, blue components of consecutive pixels and summing them.
Can you do the following (I have not thought it through in great detail)?
IntVector rgbSum = … // zero
IntVector lshrVector = … // 16, 8, 0, 0
for (…) {
IntVector rgbVector = IntVector.fromMemorySegment(MainSingleton.getInstance().SPECIES, memorySegment,
(long) bufferOffset * Integer.BYTES, ByteOrder.nativeOrder());
rgbSum = rgbSum.add(rgbVector.lanewise(VectorOperators.LSHR, lshrVector).lanewise(VectorOperators.AND, 0xFF));
}
int r = rgbSum.lane(0);
int g = rgbSum.lane(1);
int b = rgbSum.lane(2);
Paul.
</pre>
<blockquote type="cite">
<pre class="qt-moz-quote-pre">On Aug 13, 2024, at 1:38 PM, Davide Perini <a
class="qt-moz-txt-link-rfc2396E"
href="mailto:perini.davide@dpsoftware.org"
moz-do-not-send="true"><perini.davide@dpsoftware.org></a> wrote:
Hi Paul,
thanks for the answer, I really, really appreciate it.
I'm excited about these new APIs, they simply rocks.
After some trial and error I'll be able to use your suggestions of using MemorySegment and the compute time is now many times faster than using an on heap array.
This change done the tricks:
IntVector rgbVector = IntVector.fromMemorySegment(MainSingleton.getInstance().SPECIES, memorySegment,
(long) bufferOffset * Integer.BYTES, ByteOrder.nativeOrder());
r += rgbVector.lane(0) >> 16 & 0xFF;
g += rgbVector.lane(1) >> 8 & 0xFF;
b += rgbVector.lane(2) & 0xFF;
pickNumber++;
I have also tried the second suggestion (if I understood it well)
IntVector rVector = rgbVector.lanewise(VectorOperators.AND, 0xFF0000).lanewise(VectorOperators.LSHR, 16);
IntVector gVector = rgbVector.lanewise(VectorOperators.AND, 0x00FF00).lanewise(VectorOperators.LSHR, 8);
IntVector bVector = rgbVector.lanewise(VectorOperators.AND,0x00FF00);
r += rVector.reduceLanes(VectorOperators.ADD);
g += gVector.reduceLanes(VectorOperators.ADD);
b += bVector.reduceLanes(VectorOperators.ADD);
pickNumber += IntVector.SPECIES_PREFERRED.length();
but the second optimization produces slightly worse performance on my use case (let's say 30% slower on AVX512 and 300% slower on AVX256 more or less).
I'm more than happy with the current result but I would like to understand why the second snippet produces worse result if possible.
Any idea?
Thank you very much,
can't wait that this APIs will exit the incubator phase.
Davide
Il 09/08/2024 23:27, Paul Sandoz ha scritto:
</pre>
<blockquote type="cite">
<pre class="qt-moz-quote-pre">Hi Davide,
Moving this over to the panama-dev list.
A fun use-case.
If instead of an IntBuffer can you use a MemorySegment? Ideally you would use a MemorySegment for interchange with the host and GPU but for your case it should be possible to do the following:
IntBuffer rgbBuffer = …
// Create outside of loop
MemorySegment sRgbBuffer = MemorySegment.ofBuffer(rgbBuffer);
Then sRgbBuffer can be used to load int vectors within the loop (taking care to avoid attempting to read beyond the end of the array, if the buffer consists of sequences of triples of r, g, b)
I also suspect that in your inner loop you can avoid accessing the lanes of the vector and perform lanewise shifts and accumulation into a sum vector, after which you access outside of the loop, processing any tail.
Paul.
</pre>
<blockquote type="cite">
<pre class="qt-moz-quote-pre">On Aug 9, 2024, at 8:59 AM, Davide Perini <a
class="qt-moz-txt-link-rfc2396E"
href="mailto:perini.davide@dpsoftware.org"
moz-do-not-send="true"><perini.davide@dpsoftware.org></a> wrote:
Hi there,
thanks for the opportunity that you give us to write on this mailing-list.
I'm am playing with the Vector API bundled in Java 22 and wow, they are amazing.
I have some serious benefits using them even for simple tasks on my AMD Ryzen 9 7950X3D CPU that uses Zen4 architecture.
Can't wait to see how bigger the benefits will be on the upcoming processors that has some serious optimized AVX512 instructions (AMD Zen5 architecture and Intel AV10 instructions).
I'll try to give you some context.
I am writing an open source software that is basically a free clone of the Philips Ambilight effect.
What is it?
Basically you put a LED strip behind your monitor/TV, the software capture the screen,
it calculates the average colors of your screen, and sends those average values to a microcontroller (arduino) that drives the strip and light up the LEDs accordingly.
This effect is also known as dynamic bias light.
More info here if you are curious:
<a class="qt-moz-txt-link-freetext moz-txt-link-freetext"
href="https://github.com/sblantipodi/firefly_luciferin"
moz-do-not-send="true">https://github.com/sblantipodi/firefly_luciferin</a>
Most of the computations involved are on the GPU side but some intensive ones are on the CPU side.
Let's go deeper on the Vector API.
GPU acquire the screen image 60 times per seconds (or even more), every frame is a Buffer that contains colors information for each pixel of the frame.
This buffer is a Java Direct IntBuffer that doesn't have a corresponding array inside the heap for performance reason.
Once I have this IntBuffer I need to calculate the average colors of the screen and this thing can be made on the fly on the IntBuffer without copying the IntBuffer inside an Array. This kind of copy is really really heavy and degrade performance.
Just a snippet that shows it without using the Vector API...
for (int y = 0; y < pixelInUseY; y++) {
for (int x = 0; x < pixelInUseX; x++) {
int offsetX = (xCoordinate + x);
int offsetY = (yCoordinate + y);
int bufferOffset = (Math.min(offsetX, widthPlusStride)) + ((offsetY < height) ? (offsetY * widthPlusStride) : (height * widthPlusStride));
int rgb = rgbBuffer.get(Math.min(rgbBuffer.capacity() - 1, bufferOffset));
r += rgb >> 16 & 0xFF;
g += rgb >> 8 & 0xFF;
b += rgb & 0xFF;
pickNumber++;
}
}
leds[key - 1] = ImageProcessor.correctColors(r, g, b, pickNumber);
Now I'm trying to use the Vector API to accelerate this computations even more and hey, it worked awesome.
Using AVX512 (Species512) the computations is 40%-80% faster than without the Vector API.
int firstLimit;
int secondLimit;
// Processing the buffer in the correct order is crucial for SIMD performance
if (pixelInUseX < pixelInUseY) {
firstLimit = pixelInUseX;
secondLimit = pixelInUseY;
} else {
firstLimit = pixelInUseY;
secondLimit = pixelInUseX;
}
// SIMD iteration
for (int x = 0; x < firstLimit; x++) {
for (int y = 0; y < secondLimit; y += MainSingleton.getInstance().SPECIES.length()) {
int offsetX;
int offsetY;
if (pixelInUseX < pixelInUseY) {
offsetX = (xCoordinate + x);
offsetY = (yCoordinate + y);
} else {
offsetX = (xCoordinate + y);
offsetY = (yCoordinate + x);
}
int bufferOffset = (Math.min(offsetX, widthPlusStride)) + ((offsetY < height) ? (offsetY * widthPlusStride) : (height * widthPlusStride));
// Load RGB values using SIMD
int[] rgbArray = new int[MainSingleton.getInstance().SPECIES.length()];
rgbBuffer.position(bufferOffset);
rgbBuffer.get(rgbArray, 0, Math.min(MainSingleton.getInstance().SPECIES.length(), rgbBuffer.remaining()));
IntVector rgbVector = IntVector.fromArray(MainSingleton.getInstance().SPECIES, rgbArray, 0);
r += rgbVector.lane(0) >> 16 & 0xFF;
g += rgbVector.lane(1) >> 8 & 0xFF;
b += rgbVector.lane(2) & 0xFF;
pickNumber++;
}
}
leds[key - 1] = ImageProcessor.correctColors(r, g, b, pickNumber);
The computation itself is at least ten times faster but at the end it's only 40%-80% faster because I'm not able to process the IntBuffer on the fly using Vector API.
As you can see in the previous snippet I need to copy part of the IntBuffer into an int[] array and then process it using the Vector API.
This copy alone is the thing that requires more time.
Is it possible to process a direct IntBuffer with the Vector API without loosing time in an array copy?
Thank you for this wonderful API.
Kind regards
Davide
</pre>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<div><br>
</div>
</blockquote>
</body>
</html>