iopipe v0.0.4 - RingBuffers!

[biocyberman]

On Monday, 14 May 2018 at 14:23:43 UTC, Steven Schveighoffer 
wrote:
> On 5/14/18 6:02 AM, bioinfornatics wrote:
>> On Thursday, 10 May 2018 at 23:22:02 UTC, Steven Schveighoffer 
>> wrote:
>>>[...]
>> 
>> Hi Steve,
>> 
>> It is an exciting works, that could help in bioinformatics 
>> area.
>> Indeed in bioinformatics we are I/O bounding and we process 
>> lot of big files the amount of data can be in gigabytes, 
>> terabytes and even some times in petabytes.
>> 
>> So processing efficiently these amount of data is critic. Some 
>> years ago I got a request 'How to parse fastq file format in 
>> D?' and monarch_dodra wrote a really fast parser (code: 
>> http://dpaste.dzfl.pl/37b893ed )
>> 
>> It could be interesting to show how iopipe is fast.
>
> Yeah, I have been working on and off with Vang Le (biocyberman) 
> on using iopipe to parse such formats. He gave a good 
> presentation at dconf this year on using D in bioinformatics, 
> and I think it is a great fit for D!
>
> At dconf, I threw together a crude fasta parser (with the 
> intention of having it be the base for parsing fastq as well) 
> to demonstrate how iopipe can perform while parsing such 
> things. I have no idea how fast or slow it is, as I just barely 
> got it to work (pass unit tests I made up based on wikipedia 
> entry for fasta), but IMO, the direct buffer access makes fast 
> parsing much more pleasant than having to deal with your own 
> buffering (using phobos makes parsing a bit difficult, however, 
> I still see a need for some parsing tools for iopipe).
>
> You can find that library here: 
> https://github.com/schveiguy/fastaq
>
> Not being in the field of bioinformatics, I can't really say 
> that I am likely to continue development of it, but I'm 
> certainly willing to help with iopipe for anyone who wants to 
> use it in this field.
>
> -Steve

Hi Steve

Great work continuing to improve iopipe. Thank you for the 
example implementation of fasta/q parser with iopipe. I will 
definitely continue to work on this. It still requires some more 
time for me to get over beginner barriers in D. I am currently 
trying out some work over here https://github.com/bioslaD.

@Johnathan(bioinformatics) It will be great if you can join 
bioslaD and offer some help to make things move faster.

Vang