Httparsed - fast native dlang HTTP 1.x message header parser

tchaloupka chalucha at gmail.com
Mon Dec 14 21:59:02 UTC 2020


Hi,
I was missing some commonly usable HTTP parser on code.dlang.org 
and after some research and work I've published httparsed[1].

It's inspired by picohttpparser[2] which is great, but instead of 
a binding, I wanted something native to D. Go has it's own 
parsers, Rust has it's own parsers, why not D?

I think we're missing other small libraries like this on the 
code.dlang.org to be commonly used in larger ones like it's so 
common in other languages - while improving the ecosystem. Vibe-d 
is just huuuge.

It is nothrow, @nogc and can work with betterC. It just parses 
the message header and calls provided callbacks with slices to 
the original buffer to be handled as needed by the caller.

Same as picohttpparser it uses SSE4.2 `_mm_cmpestri` instruction 
to speedup the invalid characters lookup (when built with ldc2 
and target that supports it).

It has pretty thorough test suite.
Can parse incomplete message headers.
Can continue parsing from the last completely parsed line.
Doesn't enforce method or protocol version on itself to be usable 
with other internet message like protocols as is for example RTSP.

Performance wise it's pretty on par with picohttpparser [3]. 
Without SSE4.2 it's a bit faster, with SSE4.2 it's a bit slower 
and I can't figure out why :/.
But overall, I'm pretty happy with the outcome.

I've tried to check and compare with two popular libraries and:

* vibe-d - performs nearly the same as http_parser[4] (but that 
itself is pretty slow and now obsolete), but as it looks, doesn't 
do much in regard of RFC conformance - some test's from [2] won't 
pass for sure

* arsd's cgi.d - I haven't expected it to be so much slower than 
vibe-d parser, it's almost 3 times slower, but on the other hand 
it's super simple idiomatic D (again doesn't check or allow what 
RFC says it should and many tests will fail)
   * I guess the main problem would be `idup` on every line and 
autodecode
   * Stripped down minimalistic version of the original [5] is 
here [6]

[1] https://code.dlang.org/packages/httparsed
[2] https://github.com/h2o/picohttpparser
[3] https://i.imgur.com/iRCDGVo.png
[4] https://github.com/nodejs/http-parser
[5] 
https://github.com/adamdruppe/arsd/blob/402ea062b81197410b05df7f75c299e5e3eef0d8/cgi.d#L1737
[6] 
https://github.com/tchaloupka/httparsed/blob/230ba9a4a280ba91267a22e97137be12269b5574/bench/bench.d#L194


More information about the Digitalmars-d-announce mailing list