Trying to reduce memory usage

[Daniel N]

On Saturday, 13 February 2021 at 04:19:17 UTC, Ali Çehreli wrote:
> On 2/11/21 6:22 PM, H. S. Teoh wrote:
>
> > 	bool[size_t] hashes;
>
> I would start with an even simpler solution until it's proven 
> that there still is a memory issue:
>
> import std.stdio;
>
> void main() {
> 	bool[string] lines;
> 	foreach (line; stdin.byLine) {
> 		if (line !in lines) {
> 			stdout.writeln(line);
> 			lines[line.idup] = true;
> 		}
> 		// else this line already seen before; just skip it
> 	}
> }
>
> (Grr... Thanks for the tab characters! :p)
>
> Ali

Try this Boost Licensed tool.
https://github.com/eBay/tsv-utils/tree/master/tsv-uniq