malloc and buffer overflow attacks

[Patrick Schluter]

On Friday, 31 December 2021 at 22:10:10 UTC, H. S. Teoh wrote:
> On Fri, Dec 31, 2021 at 12:12:45PM -0800, Walter Bright via 
> Digitalmars-d wrote:
>> On 12/30/2021 4:37 PM, sarn wrote:
>> > Good thing to do, but Walter's talking about integer 
>> > overflow with the `len * T.sizeof` calculation itself.
>> > 
>> > calloc() doesn't have this problem.
>> 
>> The calculation of `len` can also have overflow problems. 
>> `calloc` is not sufficient. The provenance of `len` needs to 
>> be carefully checked.
>
> At my day job we use Coverity to identify potential issues with 
> our C codebase.  One of the issues it reports is using external 
> inputs as the length of a memory allocation, the typical case 
> being reading an int or long from a file/socket and then 
> passing that to malloc, et al (potentially with a sizeof 
> multipler).  So imagine a carefully-crafted malicious input 
> designed to overflow a 64-bit integer just a little -- the 
> malloc call would end up allocating just a tiny amount of 
> memory while the rest of the code thinks that the buffer has 
> more memory than can be addressed.
>
> Of course, that tempts the following check (obviously wrong, 
> but I've seen this scarily often in "professional" code):
>
> 	size_t n = ... /* read from file/socket */;
> 	size_t nbytes = n * sizeof(Element); // oops
> 	if (nbytes > INT64_MAX)	// never true

    if(__builtin_clz(nbytes)+__builtin_clz(sizeof Element)<64)

this is the best way to find overflow as it avoids all the 
undefined behaviour shenanigans of the compilers optimizers. The 
inconvenience is that not all CPU implement CLZ and some do but 
quite slowly.

> 		error();	// dead code
> 	void *p = malloc(n * sizeof(Element)); // uh-oh
> 	for (i=0; i < n; i++) {
> 		... /* use p: kaboom */