Improving IO Speed

[TJB]

On Friday, 14 March 2014 at 19:11:12 UTC, Craig Dillabaugh wrote:
> On Friday, 14 March 2014 at 18:00:58 UTC, TJB wrote:
>> I have a program in C++ that I am translating to D as a way to 
>> investigate and learn D. The program is used to process 
>> potentially hundreds of TB's of financial transactions data so 
>> it is crucial that it be performant. Right now the C++ version 
>> is orders of magnitude faster.
>>
>> Here is a simple example of what I am doing in D:
>>
>> import std.stdio : writefln;
>> import std.stream;
>>
>> align(1) struct TaqIdx
>> {
>>  align(1) char[10] symbol;
>>  align(1) int tdate;
>>  align(1) int begrec;
>>  align(1) int endrec;
>> }
>>
>> void main()
>> {
>>  auto input = new File("T201212A.IDX");
>>  TaqIdx tmp;
>>  int count;
>>
>>  while(!input.eof())
>>  {
>>    input.readExact(&tmp, TaqIdx.sizeof);
>>   // Do something with the data
>>  }
>> }
>>
>> Do you have any suggestions for improving the speed in this 
>> situation?
>>
>> Thank you!
>>
>> TJB
>
> I am not sure how std.stream buffers data (the library has been 
> marked for removal, so perhaps not very efficiently), but what 
> happens if you read in a large array of your TaqIdx structs 
> with each read.

Well, one thing that I found out by experimentation was that if I 
replace

auto input = new File("T201212A.IDX");

with

auto input = new BufferedFile("T201212A.IDX");

The performance gap vanishes.  Now I have nearly identical 
execution times between the two codes.  But perhaps if std.stream 
is scheduled for removal I shouldn't be using it?