language feature usage statistics

Dennis dkorpel at gmail.com
Sun Oct 20 22:03:22 UTC 2019


On Sunday, 20 October 2019 at 21:29:24 UTC, aliak wrote:
> Which APIs did you use to get all d project links? Does dub 
> provide something?

There might be an API, but I simply parsed the html pages.
First I get the identifiers of all packages:
```
import std.net.curl;
string page = 
get("http://code.dlang.org/?sort=added&category=&skip=0&limit=2000").idup;
string[] result;
foreach(m; page.matchAll(regex(`packages/([a-zA-Z0-9_-]+)`)))
     result ~= m[1];
```

Then I parse the package pages for the repository link with htmld:

```
import html; // http://code.dlang.org/packages/htmld
string getRepo(string packageName) {
     string page = 
get("http://code.dlang.org/packages/"~packageName).idup;
     auto doc = createDocument(page);
     if (auto p = doc.querySelector("#repository")) {
         if (auto m = p.html.matchFirst(`href="([^"]+)`)) {
             return m[1].text;
         }
     }
}
```

> And curious, were you rate limited by github (i'm assuming this 
> was the work of a for loop?).

I wouldn't have been surprised if I got a timeout for cloning 
1600 repositories in succession, but I didn't. (I suppose the 
same happens when installing your average NPM package, lol)


More information about the Digitalmars-d mailing list