Memory Mapped file segment / always appending

[Maurizio Cimadamore]

On 27/07/2020 18:18, Maurizio Cimadamore wrote:
>
> On 27/07/2020 18:12, Johannes Lichtenberger wrote:
>> Hi,
>>
>> as I'm always appending data to the end of a file. I came up with the 
>> idea
>> to always map a segment in chunks of Integer.MAX, that is the segment
>> always get's closed and a new segment gets created with twice the 
>> size (for
>> reading/writing). When I'm closing the writer I'm simply truncating the
>> file(s) to the real length.
>>
>> However, I just saw that it's horribly slow for a lot of small 
>> writes. That
>> is I'm always creating a read/write page reader, and closing it again 
>> after
>> a write. However, truncating a lot of unused space, over and over 
>> again is
>> a real problem. I've attached a flame graph from YourKit showing that 
>> half
>> of the time over a large amount of CPU samples is spend with truncating.
>
> Is the time spent in the Panama API, or do you mean that the code is 
> slow just natively (e.g. at the mmap level)
>
> We can't see the attachment (the server truncates them :-) )

If the latter, note that there's also another way to truncate a file, 
via FileChannel::truncate - although, under the hood both bottom out at 
ftruncate64 on linux.

I guess the important question here is - is there anything that the 
memory access API does which is making this slower than it needs to be, 
or is this particular use case of repeated truncation just inherently 
slow? Have you tried writing some C code for it and get a feel for what 
level of performances you can get there?

Maurizio

>
> Maurizio
>
>>
>> Any ideas, how to improve performance?
>>
>> The packe is this and very small:
>>
>> https://github.com/sirixdb/sirix/tree/master/bundles/sirix-core/src/main/java/org/sirix/io/memorymapped 
>>
>>
>> kind regards
>> Johannes