Precision Error?

WhatMeWorry` kheaser at gmail.com
Sun Oct 6 21:09:22 UTC 2024


     float computeToFitInWindow(uint rows, uint cols)
     {
         // code removed ...
         writeln(hexWidth);
         return hexWidth;
     }


     unittest
     {
         assert(computeToFitInWindow(1, 1) == 2.0f, "This assert 
failed");
         assert(computeToFitInWindow(3, 3) == 0.8f, "This assert 
failed");
         assert(computeToFitInWindow(2, 2) == 1.14286f, "This 
assert failed");   // Line 74
     }


2
0.8
1.14286
main.d(74): [unittest] This assert failed

Can someone tell me why line 74 fails?   Aren't they both 
1.14286?  I presume it has to do with precision.  Is there a 
technique to address this issue.



More information about the Digitalmars-d-learn mailing list