Strange Bug

[Rene Zwanenburg via Digitalmars-d-learn]

On Friday, 20 January 2017 at 08:19:57 UTC, Chris M. wrote:
> I have no idea if this is an issue with D, or OpenSSL, or if 
> I'm just doing something completely wrong. I'm writing a 
> program that will either encrypt or decrypt a string using AES 
> in ECB mode (for a school assignment) and it's giving me a very 
> strange bug.
> 
> [...]
> 
>     auto encOut = new ubyte[inputStr.length];
>
>     // Encrypt and convert to base64
>     AES_set_encrypt_key(aesKey.ptr, aesKey.sizeof * 8, &encKey);
>     AES_ecb_encrypt(inputStr.ptr, encOut.ptr, &encKey, 
> AES_ENCRYPT);
> 
> Here's the problem. I tried running this without the if-else 
> statements (i.e. encrypting and decrypting all in one run of 
> the program, code below). If I leave in the base64 encoding and 
> decoding, and use decB64 as the input to decrypt, it still 
> doesn't work. However, if I decrypt with encOut directly, or 
> assign encOut to decB64, it somehow works.

My guess:

The encrypted output will be a bit longer than your input. You're 
not getting an out of bounds exception during encryption since 
OpenSSL only has the pointer to write to, not a buffer length it 
can check. The memory behind your buffer is apparently committed, 
and will be written to by OpenSSL. This is why using the same 
buffer to decrypt works: it continues to read after the end of 
the buffer. (In case it's not clear, writing and reading past the 
end of the buffer is really bad)

I expect OpenSSL to have a helper function to calculate the 
required buffer size for a given input length. Use that to 
allocate the buffer.