cast(int) getting an unexpected number

rmcguire rjmcguire at gmail.com
Wed Nov 4 23:16:01 PST 2009


"Lars T. Kyllingstad" <public at kyllingen.NOSPAMnet> wrote:
 
> Charles Hixson wrote:
>> Lars T. Kyllingstad wrote:
>>> Michal Minich wrote:
>>>> Hello rmcguire,
>>>>
>>>>> why is this not a compiler bug?
>>>>> because:
>>>>> import std.stdio;
>>>>> void main() {
>>>>> float f=0.01;
>>>>> writefln("%0.2f->%d",f,cast(int)(f*100f));
>>>>> writefln("%0.2f->%d",f,cast(int)(.01*100f));
>>>>> writefln("%0.2f->%f",f,(f*100f));
>>>>> }
>>>>> results in:
>>>>> 0.01->0
>>>>> 0.01->1
>>>>> 0.01->1.000000
>>>>> I would say something is dodgy.
>>>>>
>>>>> -Rory
>>>>>
>>>>
>>>> I think this may be case of:
>>>> At comple time floating point computations may be done at a higher
>>>> precision than run time.
>>>
>>>
>>> Yes, if you do this:
>>>
>>> float f = 0.01;
>>> float g = f * 100f;
>>> real r = f * 100f;
>>> writeln("%s, %s, %s", f, cast(int) g, cast(int) r);
>>>
>>> you get:
>>>
>>> 0.01, 0, 1
>>>
>>> I believe just writing cast(int)(f*100f) is more or less the same as the
>>> 'real' case above.
>>>
>>> -Lars
>> Can that *really* be the explanation??  I know that float doesn't have 
>> all that much precision, but I thought it was more than 5 or 6 
>> places...and this is, essentially, two places.
> 
> 
> Yes it does, but that's not what matters.
> 
> Say that for floats, the representable number closest to 0.01 is 
> 0.01000000000001. (This is just an example, I don't know the true 
> number.) Then you have a lot more precision than the two digits you 
> mention, and multiplying with 100 gives 1.000000000001. Round this 
> towards zero (which is what cast(int) does) and you get 1. This is the 
> 'float g = f*100f;' case in my example.
> 
> Now, say that for reals, which (I think) is what the compiler uses 
> internally, the number closest to 0.01 is
>      0.0099999999999999999999999999999999999999.
> Again, just an example, the point is that the precision is higher than 
> the above, but the closest number is now smaller than 0.01. Multiply 
> this by 100, and you get
>      0.99999999999999999999999999999999999999.
> This number will be cast to the integer 0, which happens in the OP's case.
> 
> I admit I'm no expert in these things, but I suspect this is how it 
> goes. By the way, I recommend Don's excellent article on floating-point 
> numbers. It has really cleared things up for me:
> 
>    http://www.digitalmars.com/d/2.0/d-floating-point.html
> 
> -Lars
> 

Hmm, thanks.

pity its not consistent. Would be nice if you didn't have to learn everything 
about floating point just to make a calculator.

-Rory



More information about the Digitalmars-d-learn mailing list