floating point precision issue
-
Dear All, I have an issue with floating point precision with Visual Studio 2005 (2008 as well) See the sample code.
const float f = 0.2f * 3;
CString str;
str.Format(_T( "%f"), f );When you check the watch window, the variable will be having value 0.60000002(I expected 0.6). To confirm this again, I tried running with WinDBG, it has given a value 0.6000000238 in it's watch window I doubt this may affect my calculation which should be very precise. I tried different floating point models using /fp compiler option. But always it has the same output. But the formatted string below gives 0.600000 as the result. What could be the reason for this issue? Any solutions?
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
-
Dear All, I have an issue with floating point precision with Visual Studio 2005 (2008 as well) See the sample code.
const float f = 0.2f * 3;
CString str;
str.Format(_T( "%f"), f );When you check the watch window, the variable will be having value 0.60000002(I expected 0.6). To confirm this again, I tried running with WinDBG, it has given a value 0.6000000238 in it's watch window I doubt this may affect my calculation which should be very precise. I tried different floating point models using /fp compiler option. But always it has the same output. But the formatted string below gives 0.600000 as the result. What could be the reason for this issue? Any solutions?
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
I don't understand where is your problem exactly: only with the precision or when formating the string ? For the precision problem, you can use a double instead of a float, you'll gain much more precision. For the formating, you can supply the number of decimal points you want to show in the format string (just add a number before the 'f').
Cédric Moonen Software developer
Charting control [v1.5] OpenGL game tutorial in C++ -
Dear All, I have an issue with floating point precision with Visual Studio 2005 (2008 as well) See the sample code.
const float f = 0.2f * 3;
CString str;
str.Format(_T( "%f"), f );When you check the watch window, the variable will be having value 0.60000002(I expected 0.6). To confirm this again, I tried running with WinDBG, it has given a value 0.6000000238 in it's watch window I doubt this may affect my calculation which should be very precise. I tried different floating point models using /fp compiler option. But always it has the same output. But the formatted string below gives 0.600000 as the result. What could be the reason for this issue? Any solutions?
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
With
doubledata type you'll get a better approximation (you know you can't get exact values) :)If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler. -- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong. -- Iain Clarke
[My articles] -
Dear All, I have an issue with floating point precision with Visual Studio 2005 (2008 as well) See the sample code.
const float f = 0.2f * 3;
CString str;
str.Format(_T( "%f"), f );When you check the watch window, the variable will be having value 0.60000002(I expected 0.6). To confirm this again, I tried running with WinDBG, it has given a value 0.6000000238 in it's watch window I doubt this may affect my calculation which should be very precise. I tried different floating point models using /fp compiler option. But always it has the same output. But the formatted string below gives 0.600000 as the result. What could be the reason for this issue? Any solutions?
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
It's because standard floats and doubles cannot exactly represent 0.6. The representation of 0.6 as a float can be determined by working out what sum of binary fractions (1/2n) best approximates it - you can see it's going to be something like 0.5 + 0.0625 + 0.03125 + .... - SpeedCrunch[^] (a nice calculator) gives it as this: 0b0.1001100110011001100110011001100110011001100110011001100110011001101 Infinitely recurring binary time... Read this document[^] for the full story.
-
I don't understand where is your problem exactly: only with the precision or when formating the string ? For the precision problem, you can use a double instead of a float, you'll gain much more precision. For the formating, you can supply the number of decimal points you want to show in the format string (just add a number before the 'f').
Cédric Moonen Software developer
Charting control [v1.5] OpenGL game tutorial in C++OK try this code. then things will be more clear.
float f = 0.2f 0.6f;
f -= 0.2f;
f -= 0.2f;
f -= 0.2f;if( f == 0.0f ) // this condition will be failed { // f == 0; }In this case, after subtracting 0.2 three times, the value should reach to 0.0f. But it's fails. because of the trailing additional digits in the floating point representation. The same case exists for double type as well.
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
modified on Tuesday, February 3, 2009 6:59 AM
-
OK try this code. then things will be more clear.
float f = 0.2f 0.6f;
f -= 0.2f;
f -= 0.2f;
f -= 0.2f;if( f == 0.0f ) // this condition will be failed { // f == 0; }In this case, after subtracting 0.2 three times, the value should reach to 0.0f. But it's fails. because of the trailing additional digits in the floating point representation. The same case exists for double type as well.
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
modified on Tuesday, February 3, 2009 6:59 AM
When working with doubles or floats, you should never compare numbers directly. This won't work. Instead, you have to use precision range when comparing them (for instance if the value is between -0.000001 and 0.000001, then you can say it is equal to 0. That's the only way to work with floating point values, because they will never be 100% precise and you have to keep that in mind.
Cédric Moonen Software developer
Charting control [v1.5] OpenGL game tutorial in C++ -
When working with doubles or floats, you should never compare numbers directly. This won't work. Instead, you have to use precision range when comparing them (for instance if the value is between -0.000001 and 0.000001, then you can say it is equal to 0. That's the only way to work with floating point values, because they will never be 100% precise and you have to keep that in mind.
Cédric Moonen Software developer
Charting control [v1.5] OpenGL game tutorial in C++Cedric Moonen wrote:
for instance if the value is between -0.000001 and 0.000001, then you can say it is equal to 0. That's the only way to work with floating point values, because they will never be 100% precise and you have to keep that in mind
Yeap the sign associated with it was the real issue. it may remain negative (even I expect 0.0f)
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
-
Cedric Moonen wrote:
for instance if the value is between -0.000001 and 0.000001, then you can say it is equal to 0. That's the only way to work with floating point values, because they will never be 100% precise and you have to keep that in mind
Yeap the sign associated with it was the real issue. it may remain negative (even I expect 0.0f)
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
Sarath. wrote:
the sign associated with it was the real issue
Shouldn't be - the IEEE standard requires that positive or negative zeros compare equal.
-
Dear All, I have an issue with floating point precision with Visual Studio 2005 (2008 as well) See the sample code.
const float f = 0.2f * 3;
CString str;
str.Format(_T( "%f"), f );When you check the watch window, the variable will be having value 0.60000002(I expected 0.6). To confirm this again, I tried running with WinDBG, it has given a value 0.6000000238 in it's watch window I doubt this may affect my calculation which should be very precise. I tried different floating point models using /fp compiler option. But always it has the same output. But the formatted string below gives 0.600000 as the result. What could be the reason for this issue? Any solutions?
-Sarath. "Great hopes make everything great possible" - Benjamin Franklin
My blog - Sharing My Thoughts
See here and here. In short, not all floating-point numbers can be stored in binary without some rounding.
"Old age is like a bank account. You withdraw later in life what you have deposited along the way." - Unknown
"The brick walls are there for a reason...to stop the people who don't want it badly enough." - Randy Pausch