Yahoo Finance Scraper

[jmh530]

On Friday, 12 June 2020 at 18:22:28 UTC, Selim wrote:
> I wrote a small Yahoo finance scraper and wanted to share with 
> the community. I have been using D for a while and I think 
> contributing something to the community is good. There is an 
> example main script and a unit test. Those should get you 
> going. It currently saves the scraped data as a json file under 
> the executable's folder. I might add a public method to access 
> individual data columns inside json in the following days too.
>
> All mistakes are my own and I appreciate any feedback.
>
> https://github.com/SelimOzel/YahooMinerD
>
> Best,
> Selim

Thanks! There was a period there where you couldn't use the yahoo 
API, glad to see that people can use it again.

I haven't run it myself yet, but I have a few comments for 
potential changes and enhancements.

Why do you use a class for YahooMinerD (also the name 
YahooFinanceD might get more people to use)? It doesn't look like 
you are using any inheritance. I don't see any reason not to 
change to a struct and avoid new.

It looks like you have a lot of writeln statements. While these 
could be helpful, they will also prevent those functions from 
ever being @nogc. You could use an approach like below and give 
the user the opportunity to avoid the writelns and allow for 
attribute inference elsewhere.

@nogc void fooImpl(bool val)()
     if (val)
{

}

void fooImpl(bool val)()
     if (!val)
{
     import std.stdio: writeln;
     writeln("here");
}

void foo(bool val = false)()
{
     fooImpl!val;
}

@nogc void main() {
     foo!true;
}

It looks like the primary way to get the data is from the 
WriteToJson, correct? What if you want to use the data without 
writing the JSON to file? For instance, I want to get the data 
and put it in my own database, or I just want to get the data, do 
some calculations, and then not save it. There should be a way to 
get the data out of there without writing to file. std.json can 
be used for parsing the JSON and there are other libraries out 
there.

The WriteToJSON method should also allow writing events or prices 
without needing to write both.

You might consider adding the ability to control the frequency 
(instead of just daily).