Precision Error?
WhatMeWorry`
kheaser at gmail.com
Sun Oct 6 21:09:22 UTC 2024
float computeToFitInWindow(uint rows, uint cols)
{
// code removed ...
writeln(hexWidth);
return hexWidth;
}
unittest
{
assert(computeToFitInWindow(1, 1) == 2.0f, "This assert
failed");
assert(computeToFitInWindow(3, 3) == 0.8f, "This assert
failed");
assert(computeToFitInWindow(2, 2) == 1.14286f, "This
assert failed"); // Line 74
}
2
0.8
1.14286
main.d(74): [unittest] This assert failed
Can someone tell me why line 74 fails? Aren't they both
1.14286? I presume it has to do with precision. Is there a
technique to address this issue.
More information about the Digitalmars-d-learn
mailing list