randomIO, std.file, core.stdc.stdio

[Charles Hixson via Digitalmars-d-learn]

On 07/26/2016 12:53 PM, Adam D. Ruppe via Digitalmars-d-learn wrote:
> On Tuesday, 26 July 2016 at 19:30:35 UTC, Charles Hixson wrote:
>> It looks as if the entire file is stored in memory, which is not at 
>> all what I want, but I also can't really believe that's what's going on.
>
>
> It is just mapped to virtual memory without actually being loaded into 
> physical memory, so when you access the array it returns, the kernel 
> loads a page of the file into memory, but it doesn't do that until it 
> actually has to.
>
> Think of it as being like this:
>
> struct MagicFile {
>     ubyte[] opIndex(size_t idx) {
>           auto buffer = new ubyte[](some_block_length);
>           fseek(fp, idx, SEEK_SET);
>           fread(buffer.ptr, buffer.length, 1);
>           return buffer;
>     }
> }
>
>
> And something analogous for writing, but instead of being done with 
> overloaded operators in D, it is done with the MMU hardware by the 
> kernel (and the kernel also does smarter buffering than this little 
> example).
>
>
>> A part of the problem is that I don't want this to be a process with 
>> an arbitrarily high memory use.
>
> The kernel will automatically handle physical memory usage too, 
> similarly to a page file. If you haven't read a portion of the file 
> recently, it will discard that page, since it can always read it again 
> off disk if needed, but if you do have memory to spare, it will keep 
> the data in memory for faster access later.
>
>
> So basically the operating system handles a lot of the details which 
> makes it efficient.
>
>
> Growing a memory mapped file is a bit tricky though, you need to unmap 
> and remap. Since it is an OS concept, you can always look for C or C++ 
> examples too, like herE: 
> http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462
O, dear.  It was sounding like such an excellent approach until this 
last paragraph, but growing the file is going to be one of the common 
operations.  (Certainly at first.)  It sounds as if that means the file 
needs to be closed and re-opened for extensions.  And I quote from 
https://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html: 
<END
Function: /void */ *mremap* /(void *address, size_t length, size_t 
new_length, int flag)/

    Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety
    Concepts
    <https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html#POSIX-Safety-Concepts>.


    This function can be used to change the size of an existing memory
    area. address and length must cover a region entirely mapped in the
    same |mmap| statement. A new mapping with the same characteristics
    will be returned with the length new_length.

...
This function is only available on a few systems. Except for performing 
optional optimizations one should not rely on this function.

END
So I'm probably better off sticking to using a seek based i/o system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.puremagic.com/pipermail/digitalmars-d-learn/attachments/20160726/985bbc93/attachment.html>