DUB - call to arms

[H. S. Teoh]

On Sat, Apr 27, 2019 at 05:22:40AM +0000, Seb via Digitalmars-d wrote:
> On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
[...]
> > OK, so I upgraded my dub to the latest git master, and did a quick
> > and dirty test.  Init a fresh new project with `-t vibe.d`, accept
> > all default values (name, license, etc.), then run `time dub -v` to
> > make the first build.
> > [...]
> 
> 10 seconds for a rebuild is still too much.
> 
> One quick solution is to upgrade to ld.gold - it will half your build
> time.

Just to be clear: the 10 seconds rebuild time is only if dub.selections
is deleted before the rebuild.  I only did that as a way of getting a
rough measure of how fast/slow the dependency resolution algorithm is,
when dependencies have already been downloaded into the dub cache.  It's
not something I do habitually, or recommend. :-D

Without deleting dub.selections, the turnaround time is about 4-5
seconds.  Which is not fast, but at least marginally acceptable.
Certainly, 10 seconds turnaround would be unacceptable edit-compile-test
turnaround time on an empty project.


On Sat, Apr 27, 2019 at 05:45:50AM +0000, Seb via Digitalmars-d wrote:
> On Saturday, 27 April 2019 at 01:13:33 UTC, H. S. Teoh wrote:
> > In the meantime, it would seem that we need to look into why the
> > "searching for versions" phase takes so long.  Is it just a
> > network-dependent thing (my network has bogonously slow DNS
> > resolution, no thanks to my ISP), or is it something that can be
> > fixed in dub itself?
> 
> Thanks a lot for your interest in dub and investigating this!
> 
> First off you should never delete your dub.selections.json as it locks
> your project dependencies (it won't be used if your project is used as
> a library).

Yes, I only did that to get some idea of how efficient the dependency
resolution is.


> Anyhow, that being said there are still a ton of things that can be
> done:
> 
> - The new single API request feature doesn't work for all 100% with
>   optional dependencies (see the respective GitHub PR that introduced
>   it for details)
> - Dependencies could be checked in parallel

Parallel checking is a must, IMO, since it doesn't make sense to
bottleneck on individual network requests when there's plenty of
bandwidth to run multiple queries in parallel.


> - The registry itself could be optimized more for caching (maybe even
>   with a CDN proxy)

I'm not sure we need to use a CDN yet, unless code.dlang.org is really
getting that much traffic.

But what might help is if the registry allows more complex queries, like
"fetch me all candidate packages satisfying contraints P, Q, R... ."
Single network roundtrip for the entire query, rather than separate
network requests, once per package.  Of course, this puts more load on
the server, which may or may not be a good thing, I'm not sure.


[[...]
> There are more experimental things we could try like e.g. a fully
> local JSON index that is only updated when needed and supports partial
> updates (think apt), but I believe the bigger gains in user-experience
> will be:
> 
> - initial fetch (important for fast CI turnaround times. First point
>   of attack: parallelizing the fetching process)
> - build times with existing dependencies (important as the default
>   case.  First points of attack: build independent dependencies in
>   parallel, warn if ld.gold isn't the default on Linux, ...)

Doesn't --parallel already do this??  If not, that certainly needs to be
fixed.

Sigh... what I wouldn't give for a generic topological walk framework
that allows maximal parallelization...


T

-- 
Customer support: the art of getting your clients to pay for your own incompetence.