vibe.d: How to get the conent of a file upload ?

[aberba]

On Friday, 18 September 2020 at 00:07:12 UTC, wjoe wrote:
> On Thursday, 17 September 2020 at 22:33:46 UTC, Steven 
> Schveighoffer wrote:
>> On 9/17/20 6:13 PM, aberba wrote:
>>> On Thursday, 17 September 2020 at 21:57:37 UTC, Steven 
>>> Schveighoffer wrote:
>>>> On 9/17/20 1:08 PM, wjoe wrote:
>>>>> [...]
>>>>
>>>> the `files` property actually does the processing only when 
>>>> you call it.
>>>>
>>>> If you access the `bodyReader` property directly, you can 
>>>> process that data yourself. You can even register a web 
>>>> interface function with an `InputStream` parameter type, and 
>>>> it will be bound to the body data.
>>> 
>>> I'm not sure I understand how to do this and parser the files 
>>> in memory.
>>
>> So an HTTP request with form data will come in with the 
>> headers parsed, but the data is still on the network stream.
>>
>> The first time you access `files`, it processes the stream 
>> data, and splits it into form data and file data, saves the 
>> files, and then gives you back the file dictionary so you can 
>> use them.
>>
>> If instead, you access `bodyReader`, YOU get to process the 
>> form data and file data.
>>
>>>>
>>>> I've done this with my REST interface, though that's not 
>>>> form data.
>>>>
>>>> That's not a great API, though. I would love to see vibe.d 
>>>> allow a direct call to vibe.inet.webform.parseFormData with 
>>>> a specific handler for files and form data.
>>> Can we file an issue for this? Because I'm very interested in 
>>> having this resolved
>>
>> You can always file an issue! 
>> https://github.com/vibe-d/vibe.d/issues
>>
>> There may already be one in there.
>>
>>> There's potential to results in out of memory condition. Its 
>>> a know issues. A complete parser (like multer in nodejs) 
>>> allowance you to limit file size as well for error handling.
>>
>> Meh, this is D :) we should be able to just process the data 
>> and do whatever we want with it. What I would like to see is 
>> vibe provide the parsing of form data, and just give me the 
>> data as it comes (kind of like a SAX parser). Maybe just a 
>> property in the HTTPServerRequest that I can set that says 
>> "use this callback when you get Form File data".
>>
>>>> I've done this with my REST interface, though that's not 
>>>> form data.
>>> 
>>> Can you share your code for this?
>>
>> Heh, this is not form data, it's just file data, raw on the 
>> stream. So I have a function like:
>>
>> ```
>> class FileRestImpl
>> {
>>     @path(":cat/:id/:uuid/upload")
>>     @getAuth
>>     void postUpload(HTTPServerResponse res, string _cat, int 
>> _id, string _uuid, InputStream stream, Nullable!string md5sum, 
>> NRMAuthInfo _authInfo)
>>     {
>>         ...
>>     }
>> }
>> ```
>>
>> You can see, I take an InputStream as a parameter -- the data 
>> comes in there. I just read it and save it to a file (in the 
>> correct location) anyway, verifying the md5sum is valid.
>>
>> -Steve
>
> Not a reply to this post in particular but to all the ones I've 
> read so far.
>
> If I understand correctly. Vibe parses the form data and writes 
> all files to disk. Where to ?
> Can I configure it ? I don't want libraries to just write data 
> to my file systems without me setting this up. Nowhere did I 
> find this behavior described in the docs.
> And if not, how is data processed with a 10mb file upload 
> followed by a few number fields ?
> It needs to read all of the file data to get to the other data 
> fields, doesn't it ?
>
> I'm sorry this is completely counter intuitive. I can 
> understand the memory/security risks and all but I have no 
> intention to hack, DOS or however else disrupt my private 
> server in my private network with garbage data. I just want to 
> get the data in a byte[].

That's what I was trying to answer. When Steve said meh, he 
probably didn't get what I said. Probably its because of my typos.

This sort of convenience and productivity benefit is part of why 
I use Node.Js in the job when I need to get things done....and 
not D yet. There are several pieces and bits you can't write 
yourself when working on projects.

In this case you want to get the file(s) in memory...in the form 
of bytes (or buffer) and probably set a file size limit. Its all 
doable through a library but such a library doesn't exist in D 
yet. At least not that I know of.

Its why I mentioned that multer[1] in Node.Js able to do 
that...hence the advantage. Its built for the express 
framework...meaning such library can be built to work with 
vibe.d. Not everything can be built into vibe.d..and I think 
that'll even make it bloated for other uses case. Its need an 
ecosystem of third-party libraries.

In the case of the vibe.d form, data and files are handled using 
this implementation[2] so its a reference to such a form parser 
implementation...with support for a storage parameter for either 
a MemoryStore or SessionStore. Multer does it pretty cleanly.

1. Multer: https://www.npmjs.com/package/multer
2. 
https://github.com/vibe-d/vibe.d/blob/ebebfa827f568cc9bced4bec2b66edc043a8adf7/inet/vibe/inet/webform.d


>
> Why does the lib not simply reject files that are unreasonably 
> (configurable) big ?
> Writing files to disk in order to then needing to copy them 
> somewhere else or to read them back into memory for further 
> processing sounds, above all else, incredibly inefficient.
> I might not even want to keep the file and drop it.

Not implemented yet.

>
> I guess it's no problem to parse the data myself, but then 
> what's the point in using a framework ?

Vibe.d still lacks many things I personally need... there's 
simply not enough ecosystem third-party libraries.

>
> Are there other frameworks besides vibe that can do what I want 
> ?

I don't think so. Vibe.d is the one with most features. But Does 
it need to be part of the framework itself?

>
> I'm sorry for the rant, developing this kind of software is a 
> pain in the drain and stresses me out to no end. It sucked hard 
> in the past with php and decades later with python, ruby, D, 
> you name it it still sucks ;)

Its sucks but the absence of libraries makes it suck more. Its 
only a matter of time...as long as more people try vibe.d and 
develop tools around it.

I love D but I also know its not ready for every task...unless 
you're willing to write some things yourself.