<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 07/26/2016 12:53 PM, Adam D. Ruppe via Digitalmars-d-learn wrote:<br>
<blockquote cite="mid:mdxgpievhdtmesndrwje@forum.dlang.org"
type="cite">On Tuesday, 26 July 2016 at 19:30:35 UTC, Charles
Hixson wrote:
<br>
<blockquote type="cite">It looks as if the entire file is stored
in memory, which is not at all what I want, but I also can't
really believe that's what's going on.
<br>
</blockquote>
<br>
<br>
It is just mapped to virtual memory without actually being loaded
into physical memory, so when you access the array it returns, the
kernel loads a page of the file into memory, but it doesn't do
that until it actually has to.
<br>
<br>
Think of it as being like this:
<br>
<br>
struct MagicFile {
<br>
ubyte[] opIndex(size_t idx) {
<br>
auto buffer = new ubyte[](some_block_length);
<br>
fseek(fp, idx, SEEK_SET);
<br>
fread(buffer.ptr, buffer.length, 1);
<br>
return buffer;
<br>
}
<br>
}
<br>
<br>
<br>
And something analogous for writing, but instead of being done
with overloaded operators in D, it is done with the MMU hardware
by the kernel (and the kernel also does smarter buffering than
this little example).
<br>
<br>
<br>
<blockquote type="cite">A part of the problem is that I don't want
this to be a process with an arbitrarily high memory use.
<br>
</blockquote>
<br>
The kernel will automatically handle physical memory usage too,
similarly to a page file. If you haven't read a portion of the
file recently, it will discard that page, since it can always read
it again off disk if needed, but if you do have memory to spare,
it will keep the data in memory for faster access later.
<br>
<br>
<br>
So basically the operating system handles a lot of the details
which makes it efficient.
<br>
<br>
<br>
Growing a memory mapped file is a bit tricky though, you need to
unmap and remap. Since it is an OS concept, you can always look
for C or C++ examples too, like herE:
<a class="moz-txt-link-freetext" href="http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462">http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462</a><br>
</blockquote>
O, dear. It was sounding like such an excellent approach until this
last paragraph, but growing the file is going to be one of the
common operations. (Certainly at first.) It sounds as if that
means the file needs to be closed and re-opened for extensions. And
I quote from
<a class="moz-txt-link-freetext" href="https://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html">https://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html</a>:
<END<br>
<dt>Function: <em>void *</em> <strong>mremap</strong> <em>(void *<var>address</var>,
size_t <var>length</var>, size_t <var>new_length</var>, int <var>flag</var>)</em></dt>
<dd>
<p>Preliminary:
| MT-Safe | AS-Safe | AC-Safe | See <a
href="https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html#POSIX-Safety-Concepts">POSIX
Safety Concepts</a>.
</p>
<p>This function can be used to change the size of an existing
memory
area. <var>address</var> and <var>length</var> must cover a
region entirely mapped
in the same <code>mmap</code> statement. A new mapping with the
same
characteristics will be returned with the length <var>new_length</var>.
</p>
</dd>
...<br>
This function is only available on a few systems. Except for
performing
optional optimizations one should not rely on this function.
<br>
<br>
END<br>
So I'm probably better off sticking to using a seek based i/o
system.<br>
</body>
</html>