Announcing Elembuf

[Cyroxin]

On Tuesday, 18 December 2018 at 01:34:00 UTC, H. S. Teoh wrote:
> I'm not sure I understand what you mean by "globals"; AFAICT 
> MmFile just has a bunch of member variables, most of which are 
> only important on the initial mapping and later unmapping.  
> Once you get a T[] out of MmFile, there's little reason to use 
> the MmFile object directly anymore until you're done with the 
> mapping.

Even if you were to use it only once, you would still have to 
create all those variables and keep them around untill you no 
longer use the slice. While with Elembuf, you only need to keep 
the slice.


>> In std.mmfile OpSlice returns a void[] instead of a T[], 
>> making it difficult to work with as it requires a cast, there 
>> would also be a need to do costly conversions should "T.sizeof 
>> != void.sizeof" be true.
>
> Are you sure? Casting void[] to T[] only needs to be done once, 
> and the only cost is recomputing .length. (Casting an array 
> does *not* make a copy of the elements or anything of that 
> sort, btw.) Once you have a T[], it's pointless to call 
> Mmfile.opSlice again; just slice the T[] directly.

I was assuming that you were using mmfile directly. Yes, it is 
possible to just use the output, but the benefit I see from 
Elembuf is that you can use it directly without too much 
overhead, but you can also just take a slice and refill when 
needed as well. While the focus of this library is in socket 
receival, reading from a file doesn't seem to be bad either. 
Although if you are intending on sending the file through a 
socket to another computer, use splice 
instead.(http://man7.org/linux/man-pages/man2/splice.2.html)


>> However, from purely a code perspective Elembuf attempts to 
>> have minimal runtime arguments and variables, with heavy 
>> reliance on compile time arguments. It also uses a newer 
>> system call for Linux (Glibc) that is currently not in 
>> druntime, the reason for this system call is that it allows 
>> for faster buffer construction. Read more about it here: 
>> https://dvdhrm.wordpress.com/2014/06/10/memfd_create2/
>
> Hmm. Isn't that orthogonal to mmap(), though?  You could just 
> map a
> memfd descriptor using mmap() to achieve essentially equivalent
> functionality.  Am I missing something obvious?

The main point from the post linked earlier is that using 
memfd_create instead of shm_open has many benefits. While it may 
not directly affect how files are read into memory,as mmfile is 
using open, it is still interesting to know when benchmarking 
your options. If you are writing the whole file into memory, then 
this library may be a better option for you just purely for the 
sake of saving memory. If you are not doing that and instead 
using file mmap with an offset, then benchmarking would give you 
the best answer.