code.dlang.org is down

[Sönke Ludwig]

Am 12.05.2025 um 13:49 schrieb kdevel:
> I would like to ask you if you thought about some of the following
> measures (not necessarily in their order of appearance):
> 
> On Friday, 11 April 2025 at 09:06:01 UTC, Sönke Ludwig wrote:
>> [...]
>> I'm not sure what exactly causes it, but the process is in a state 
>> where it's mostly busy with garbage collection. It seems like there is 
>> maybe a self-reinforcing effect, where incoming connections that time 
>> out lead to higher GC pressure.
> 
> - Turn off the GC.

In the situation that lead to the issues that would blow up memory usage 
within a very short amount of time, replacing the bad responsiveness 
with a non-responsive system or in the OOM killer terminating some process.

> 
> - Run vibe.d under hard memory/cpu limit (which terminate
>    the process if exceeded).

The problem here was not hardware resources, but the fact that the GC 
was stopping the process for a large amount of time, as well as 
allocation overhead. The concurrent GC might improve this, but the 
question is whether that would then lead to excessive memory usage.

> 
> - Deploy a monitoring process which checks if vibe.d responds
>    in say 20 ms. If not kill vibe.d with SIGKILL.

There is a certain startup overhead and there are some requests that can 
take longer (registry dump being the longest, but also requesting 
information about a dependency graph, which is done by dub). I think 
this should really only be a last-resort approach (e.g. response time > 
5s), because it replaces bad response times with failed requests.

> - flock with LOCK_EX on an open fd fopend on the vibe.d binary
>    right before vibe.d issues the bind call.
> 
> - Have a second vibe.d process running blocking on it's flock call
>    (= auto restart).

This would still mean that active connections will fail, which is not 
ideal in a situation where the restart would be frequently necessary.

> 
>> The other thing is that there appear to be some very aggressive 
>> crawlers that go through the whole site at maximum speed with what 
>> looks like 8 parallel requests.
> 
> To me it seems that code.dlang.org is mostly not a web app but
> a web site with static content. Have you been thinking of serving
> that static content with apache/nginx using rate limiting
> (mod_throttle etc.)? Or putting this content directly to CF who is
> also a web hoster?

This is not really true, the truly static content has been moved to 
dub.pm a while ago and the rest is made up of dynamic views on the 
package database. Of course it would be possible to cache pages, but 
that wouldn't help against crawlers.

Writing out all package and package list pages and then serving them as 
static content would result in a huge amount of files and occupied 
memory and would be time consuming. This would only really make sense 
when the number of pages would get reduced massively (e.g. no 
per-version package pages, limiting the number of result pages for the 
popular/new/updated package lists).

Optimizing the memory allocation patterns I think is the most efficient 
approach to improve the situation in the short term. Redesigning the 
package update process so that it runs in a separate process that 
communicates with one or more web frontends would enable scaling and 
load balancing, but would be more involved.