Precision Error?
Sergey
kornburn at yandex.ru
Sun Oct 6 22:27:18 UTC 2024
On Sunday, 6 October 2024 at 21:09:22 UTC, WhatMeWorry` wrote:
> float computeToFitInWindow(uint rows, uint cols)
> {
> // code removed ...
> writeln(hexWidth);
> return hexWidth;
> }
>
>
> unittest
> {
> assert(computeToFitInWindow(1, 1) == 2.0f, "This assert
> failed");
> assert(computeToFitInWindow(3, 3) == 0.8f, "This assert
> failed");
> assert(computeToFitInWindow(2, 2) == 1.14286f, "This
> assert failed"); // Line 74
> }
>
>
> 2
> 0.8
> 1.14286
> main.d(74): [unittest] This assert failed
>
> Can someone tell me why line 74 fails? Aren't they both
> 1.14286? I presume it has to do with precision. Is there a
> technique to address this issue.
https://dlang.org/library/std/math/operations/is_close.html
More information about the Digitalmars-d-learn
mailing list