VC++6 + VS SP6 = COMPUTATION BUG!!
-
Hi guys, PC : Intel PIII 700MHz with W2k SP6 I was using VC++ 6 with VisualStudio SP4 and the result of a calculation like this one..
double res = 25.5/1000; // would give you 0.025500000000, as you can expect..Now, here’s something quite weird: I’ve just upgraded VS to SP6 and the same calculation now gives me..double res = 25.5/1000; // 0.025499999999999998 ?????:doh: even worse if you write:double t2 = 5.f*0.001f; // 0.0050000002374872565 :wtf:And it seems to be like that every time you try to compute 2 doubles or a double and an int for ex. (multiplication or division !) as anyone noticed something similar? Wonder if there is any fix for that one. Tx Fred -
Hi guys, PC : Intel PIII 700MHz with W2k SP6 I was using VC++ 6 with VisualStudio SP4 and the result of a calculation like this one..
double res = 25.5/1000; // would give you 0.025500000000, as you can expect..Now, here’s something quite weird: I’ve just upgraded VS to SP6 and the same calculation now gives me..double res = 25.5/1000; // 0.025499999999999998 ?????:doh: even worse if you write:double t2 = 5.f*0.001f; // 0.0050000002374872565 :wtf:And it seems to be like that every time you try to compute 2 doubles or a double and an int for ex. (multiplication or division !) as anyone noticed something similar? Wonder if there is any fix for that one. Tx Fredhttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_Why_Floating_Point_Numbers_May_Lose_Precision.asp[^]
"You're obviously a superstar." - Christian Graus about me - 12 Feb '03 "Obviously ??? You're definitely a superstar!!!" mYkel - 21 Jun '04 Within you lies the power for good - Use it!
-
Hi guys, PC : Intel PIII 700MHz with W2k SP6 I was using VC++ 6 with VisualStudio SP4 and the result of a calculation like this one..
double res = 25.5/1000; // would give you 0.025500000000, as you can expect..Now, here’s something quite weird: I’ve just upgraded VS to SP6 and the same calculation now gives me..double res = 25.5/1000; // 0.025499999999999998 ?????:doh: even worse if you write:double t2 = 5.f*0.001f; // 0.0050000002374872565 :wtf:And it seems to be like that every time you try to compute 2 doubles or a double and an int for ex. (multiplication or division !) as anyone noticed something similar? Wonder if there is any fix for that one. Tx FredFred D. wrote: Wonder if there is any fix for that one I doubt it. What you have stumbled across is inevitable with floating point arithmetic. You need to decide what precision you want and round up to that precision after your calculation. You can then show the floating point number to the precision required. e.g.
double res = 25.5/1000 + 0.0005;for your example. Ant. I'm hard, yet soft.
I'm coloured, yet clear.
I'm fruity and sweet.
I'm jelly, what am I? Muse on it further, I shall return! - David Walliams (Little Britain) -
Fred D. wrote: Wonder if there is any fix for that one I doubt it. What you have stumbled across is inevitable with floating point arithmetic. You need to decide what precision you want and round up to that precision after your calculation. You can then show the floating point number to the precision required. e.g.
double res = 25.5/1000 + 0.0005;for your example. Ant. I'm hard, yet soft.
I'm coloured, yet clear.
I'm fruity and sweet.
I'm jelly, what am I? Muse on it further, I shall return! - David Walliams (Little Britain)tx for your reply, in your example it would actually be like that:
Double t = (25.5+0.0005)/1000; // but that still gives 0.025500499999999999 :doh:but actually, it seems it be a display issue in the debugger viewer.. 'cos even when the debugger shows a value d = 0.025500000001 where it should be 0.025500000000 when compare d with 0.025500000000, the result is true. and a further check at the @ of the variable in the memory window, shows a correct value. TK GOD! :-D Fred -
tx for your reply, in your example it would actually be like that:
Double t = (25.5+0.0005)/1000; // but that still gives 0.025500499999999999 :doh:but actually, it seems it be a display issue in the debugger viewer.. 'cos even when the debugger shows a value d = 0.025500000001 where it should be 0.025500000000 when compare d with 0.025500000000, the result is true. and a further check at the @ of the variable in the memory window, shows a correct value. TK GOD! :-D FredYes you still will not have the correct answer to all those decimal places. The idea is that the answer is correct to the precision that you choose. In this case 4 decimal places. You are quite correct in the matter of it being a display issue thereafter. Ant. I'm hard, yet soft.
I'm coloured, yet clear.
I'm fruity and sweet.
I'm jelly, what am I? Muse on it further, I shall return! - David Walliams (Little Britain)