Java 17 bug (?) on Windows with MemorySegment.asSlice and offset =

[Uwe Schindler]

Hi Maurizio,

> mapAddress = MapViewOfFile(
>          mapping,             /* Handle of file mapping object */
>          mapAccess,           /* Read and write access */
>          highOffset,          /* High word of offset */
>          lowOffset,           /* Low word of offset */
>          (DWORD)len);         /* Number of bytes to map */ <------
> 
> ```
> 
> That cast seems questionable, as DWORD is 32-bit, so casting an 8-byte
> value bigger than 2^32 into that will surely fail.
> 
> Of course, this was never an issue with ByteBuffer, as you could never
> create such a big memory mapped region. But with memory segments this
> becomes problematic.
> 
> Fixing the cast to use SIZE_T instead solves the issue you are seeing.

Do you have a PR / bug report for this already? That's actually a good thing that Erel found that problem! This reminds me about the other issue I found with ignoring the offset on windows (exactly the code you also suspected to be buggy; the code where the rounding of base address was applied at two different places). Luckily those bugs were quickly found by unit tests on Lucene's side (in JDK 16 already).

Lucene is testing with MMapDirectory, too - but not on Windows with huge indexes like 8 Gigabytes. Actually for fragmentation reasons we still split larger files into chunks, but those chunk sizes are always multiplies of a power of two - good luck! And during testing we create much smaller chunks to test the chunking, so we have no huge mappings during testing. One note about your workaround: The power of 2 workaround does not work well, if you have an unaligned start offset (like mapping at offset 7 of file for 2 gigabytes), because then the total mapped size is larger (a few bytes added to get the file offset aligned).

I am glad that we did not yet have release Lucene with support for mapped memory segments! �� My problem with those bugs is that you cannot test it easily with our unit/integration tests in our testing infrastructure, because building indexes of that size is not an everyday task.

Uwe