RFR: 8330936: [ubsan] signed integer overflows in libawt java2d

[Christoph Langer]

On Thu, 23 Jan 2025 09:23:52 GMT, Matthias Baesken <mbaesken at openjdk.org> wrote:

> In java2d coding there are a few overflows (those are shown when running jtreg tests with ubsan enabled binaries) 
> jtreg test java/awt/Scrollbar/AquaLFScrollbarTest/ScrollBarBorderTest.java shows
> 
> jdk/src/java.desktop/share/native/libawt/java2d/loops/TransformHelper.c:683:16: runtime error: signed integer overflow: 1651910497 + 660764199 cannot be represented in type 'int'
>     #0 0x7efe59e6ece8 in BilinearInterp src/java.desktop/share/native/libawt/java2d/loops/TransformHelper.c:683
>     #1 0x7efe59e75e21 in Java_sun_java2d_loops_TransformHelper_Transform src/java.desktop/share/native/libawt/java2d/loops/TransformHelper.c:499
>     #2 0x7efe9b8dee7b (<unknown module>)
> 
> java/awt/BasicStroke/DashStrokeTest.java shows
> 
> src/java.desktop/share/native/libawt/java2d/pipe/ShapeSpanIterator.c:1366:21: runtime error: signed integer overflow: 128253951 + 2118518271 cannot be represented in type 'int'
>     #0 0x7fb97d7daf21 in ShapeSINextSpan src/java.desktop/share/native/libawt/java2d/pipe/ShapeSpanIterator.c:1366
>     #1 0x7fb97d62fa7e in AnyIntSetSpans src/java.desktop/share/native/libawt/java2d/loops/AnyInt.c:75
>     #2 0x7fb97d6a8816 in Java_sun_java2d_loops_FillSpans_FillSpans src/java.desktop/share/native/libawt/java2d/loops/FillSpans.c:92
>     #3 0x7fba12d07e7b (<unknown module>)
> 
> 
> There is currently no need seen to adjust this coding, so exclude the methods from ubsan checking to avoid unneeded warnings.

The change looks ok and trivial. However I find nothing about an investigation what would need to be changed to eliminate the UB warning in code.

-------------

PR Comment: https://git.openjdk.org/jdk/pull/23255#issuecomment-2619393742