Trying to reduce memory usage
frame
frame86 at live.com
Fri Feb 12 07:23:12 UTC 2021
On Friday, 12 February 2021 at 02:22:35 UTC, H. S. Teoh wrote:
> This turns the OP's O(n log n) algorithm into an O(n)
> algorithm, doesn't
> need to copy the entire content of the file into memory, and
> also uses
> much less memory by storing only hashes.
But this kind of hash is maybe insufficient to avoid hash
collisions. For such big data slower but stronger algorithms like
SHA are advisable.
Also associative arrays uses the same weak algorithm where you
can run into collision issues. Thus using the hash from string
data as key can be a problem. I always use a quick hash as key
but hold actually a collection of hashes in them and do a lookup
to be on the safe side.
More information about the Digitalmars-d-learn
mailing list