Precision Error?

Sergey kornburn at yandex.ru
Sun Oct 6 22:27:18 UTC 2024


On Sunday, 6 October 2024 at 21:09:22 UTC, WhatMeWorry` wrote:
>     float computeToFitInWindow(uint rows, uint cols)
>     {
>         // code removed ...
>         writeln(hexWidth);
>         return hexWidth;
>     }
>
>
>     unittest
>     {
>         assert(computeToFitInWindow(1, 1) == 2.0f, "This assert 
> failed");
>         assert(computeToFitInWindow(3, 3) == 0.8f, "This assert 
> failed");
>         assert(computeToFitInWindow(2, 2) == 1.14286f, "This 
> assert failed");   // Line 74
>     }
>
>
> 2
> 0.8
> 1.14286
> main.d(74): [unittest] This assert failed
>
> Can someone tell me why line 74 fails?   Aren't they both 
> 1.14286?  I presume it has to do with precision.  Is there a 
> technique to address this issue.

https://dlang.org/library/std/math/operations/is_close.html


More information about the Digitalmars-d-learn mailing list