Numeric Calc Proplem
-
All, 1) Why does Dim mResult as double mResult = 0.1 * 0.1 return mResult = 0.010000000000000002? What can I do about it? Thanks, Joe
You cannot trust double nor float results on any language (to be picky, those who use IEEE floating point arithmetic): they are subject to rounding errors. A full description of the causes can be found on several articles on the web, like here Search google for, e.g., IEEE rounding errors. There are quick ways of working this around: 1. round the results 2. don't compare with '=', use a 'between'-like comparison. 3. Use the Decimal data type. Trying to make bits uncopyable is like trying to make water not wet. -- Bruce Schneier
-
You cannot trust double nor float results on any language (to be picky, those who use IEEE floating point arithmetic): they are subject to rounding errors. A full description of the causes can be found on several articles on the web, like here Search google for, e.g., IEEE rounding errors. There are quick ways of working this around: 1. round the results 2. don't compare with '=', use a 'between'-like comparison. 3. Use the Decimal data type. Trying to make bits uncopyable is like trying to make water not wet. -- Bruce Schneier