Adding a Positive Double to Negative Double
-
Hi Guys, I have this weird problem in a VB6 application i am maintaining where an unexpected value is returned from addition of two doubles. One of those doubles is negative. Eg.
Dim tst As Double Dim tst1 As Double tst = -8.4 tst1 = 8.45 tst = tst + tst1 MsgBox tstThe Value of tst is 4.99999999999989E-02 :omg: when the MsgBox appears. Anyone know whats going on here? Thanks In Advance
-
Hi Guys, I have this weird problem in a VB6 application i am maintaining where an unexpected value is returned from addition of two doubles. One of those doubles is negative. Eg.
Dim tst As Double Dim tst1 As Double tst = -8.4 tst1 = 8.45 tst = tst + tst1 MsgBox tstThe Value of tst is 4.99999999999989E-02 :omg: when the MsgBox appears. Anyone know whats going on here? Thanks In Advance
For standard questions like this I have a standard answer: The way floats/doubles are stored in binary makes it impossible to exactly represent most values, especially the ones that seem like round numbers to humans thinking using base ten. So 4, 2, 1, 0.5, 3.5, 3.75 and many others are representable exactly (basically since they all equal an integer divided by some power of 2); but 3.6 and 3.8 and an infinite number of other values are not. Now whatever float/double function you call, if it returns a float/double will suffer from the same phenomenon. So the only way to really get "3.6" or "3.8" is by using a function that not only rounds but returns a string. I trust there are some formatting methods that do just that in every programming language; In .NET languages ToString() should be one of them, given an appropriate format specifier. If you want to know much more about this topic, here it is: http://docs.sun.com/source/806-3568/ncg\_goldberg.html BTW: to circumvent the floating-point rounding problem, especially for monetary numbers, they introduced the "decimal" data type (which basically stores decimal digits). :)
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get. Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
-
For standard questions like this I have a standard answer: The way floats/doubles are stored in binary makes it impossible to exactly represent most values, especially the ones that seem like round numbers to humans thinking using base ten. So 4, 2, 1, 0.5, 3.5, 3.75 and many others are representable exactly (basically since they all equal an integer divided by some power of 2); but 3.6 and 3.8 and an infinite number of other values are not. Now whatever float/double function you call, if it returns a float/double will suffer from the same phenomenon. So the only way to really get "3.6" or "3.8" is by using a function that not only rounds but returns a string. I trust there are some formatting methods that do just that in every programming language; In .NET languages ToString() should be one of them, given an appropriate format specifier. If you want to know much more about this topic, here it is: http://docs.sun.com/source/806-3568/ncg\_goldberg.html BTW: to circumvent the floating-point rounding problem, especially for monetary numbers, they introduced the "decimal" data type (which basically stores decimal digits). :)
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get. Show formatted code inside PRE tags, and give clear symptoms when describing a problem.