Double value is rounded to unexpected value: 2.5 -> 2 instead of 3
alex_ca via Digitalmars-d-learn
digitalmars-d-learn at puremagic.com
Fri Jul 7 11:29:57 PDT 2017
Hi,
I'm having trouble understanding why in some cases a double value
will be rounded up and other times down, for the same code.
Here's a snippet with code I tried to debug:
int getNumberOfStitchesForRowLength(double rowLength)
{
writeln("input ", rowLength, " ", currentGauge.stitch_gauge,
" ", currentGauge.gauge_length);
writeln("stitches: ", (rowLength * currentGauge.stitch_gauge)
/ currentGauge.gauge_length, " -> " , ((rowLength *
currentGauge.stitch_gauge) /
currentGauge.gauge_length).roundTo!int);
double end = 2.5;
double start = 0;
writeln("I expect: ", ((abs(end-start)*10)/10).roundTo!int);
return ((rowLength * currentGauge.stitch_gauge) /
currentGauge.gauge_length).roundTo!int;
}
And here's some output from that:
input 2.5 10 10
stitches: 2.5 -> 2
I expect: 3
OR, similarly, I get
input 3.5 10 10
stitches: 3.5 -> 3
I expect: 4
However, it works as I would expect for one value:
input 1.5 10 10
stitches: 1.5 -> 2
I expect: 2
I would appreciate some ideas for why I see this seeming
inconsistency. Or if someone can share how I can further debug
this.
Thanks!
I'm using DMD32 D Compiler v2.074.1 on Windows
More information about the Digitalmars-d-learn
mailing list