Solved: Floating point rounding question
-
NB: As indicated in a reply the issue has been solved. The DirectX module was changing the floating point precision. A colleague is calling the constructor of a class which takes two doubles as arguments. Each argument he passes is the sum of two doubles: CGridPoint( a + b, c + d) The class object stores the two values internally. When he views the result he sees that the values have been rounded to 1 decimal place. The values are not so large that this would be expected. Here are example: a = 44.445472717 b = 724631.800000000 a + b = 724676.250000000 In other words he passes in 44.445472717 + 724631.800000000 as the first argument to the constructor and the result is stored internally and is seen to be 724676.250000000. Does anyone know the cause of this rounding? BTW he uses VS2008. This works in VS2010.
-
NB: As indicated in a reply the issue has been solved. The DirectX module was changing the floating point precision. A colleague is calling the constructor of a class which takes two doubles as arguments. Each argument he passes is the sum of two doubles: CGridPoint( a + b, c + d) The class object stores the two values internally. When he views the result he sees that the values have been rounded to 1 decimal place. The values are not so large that this would be expected. Here are example: a = 44.445472717 b = 724631.800000000 a + b = 724676.250000000 In other words he passes in 44.445472717 + 724631.800000000 as the first argument to the constructor and the result is stored internally and is seen to be 724676.250000000. Does anyone know the cause of this rounding? BTW he uses VS2008. This works in VS2010.
What is the code he is using to store the values and also to display them?
Unrequited desire is character building. OriginalGriff
-
What is the code he is using to store the values and also to display them?
Unrequited desire is character building. OriginalGriff
He is examing the property of the CGridPoint class using the debugger. He also confirms that the results look wrong when used. He displays a 2D representation of a Geophysical survey vessel and streamers on his screen, and position error ellipses are wrong, which is how he found this issue. The code is C++ of course. He has rewritten like this: CGridPoint gpt2 = gpt[i].Rotate(m_gptEllipseAngle); gpt2.m_fEasting += gptLocation.m_fEasting; gpt2.m_fNorthing += gptLocation.m_fNorthing; pts[i] = ViewPoint(gpt2); And it still fails. The original would have been something like: CGridPoint gpt2 = gpt[i].Rotate(m_gptEllipseAngle); pts[i] = ViewPoint(CGridPoint(gpt2.fEasting + gptLocation.m_fEasting, gpt2.fNorthing + gptLocation.m_fNorthing )); The above might have a typo or two, not sure, I rewrote it here without compiling it.
-
He is examing the property of the CGridPoint class using the debugger. He also confirms that the results look wrong when used. He displays a 2D representation of a Geophysical survey vessel and streamers on his screen, and position error ellipses are wrong, which is how he found this issue. The code is C++ of course. He has rewritten like this: CGridPoint gpt2 = gpt[i].Rotate(m_gptEllipseAngle); gpt2.m_fEasting += gptLocation.m_fEasting; gpt2.m_fNorthing += gptLocation.m_fNorthing; pts[i] = ViewPoint(gpt2); And it still fails. The original would have been something like: CGridPoint gpt2 = gpt[i].Rotate(m_gptEllipseAngle); pts[i] = ViewPoint(CGridPoint(gpt2.fEasting + gptLocation.m_fEasting, gpt2.fNorthing + gptLocation.m_fNorthing )); The above might have a typo or two, not sure, I rewrote it here without compiling it.
I can't see the
CGridPointclass in ATL or MFC; is this a Microsoft or home produced class?Unrequited desire is character building. OriginalGriff
-
NB: As indicated in a reply the issue has been solved. The DirectX module was changing the floating point precision. A colleague is calling the constructor of a class which takes two doubles as arguments. Each argument he passes is the sum of two doubles: CGridPoint( a + b, c + d) The class object stores the two values internally. When he views the result he sees that the values have been rounded to 1 decimal place. The values are not so large that this would be expected. Here are example: a = 44.445472717 b = 724631.800000000 a + b = 724676.250000000 In other words he passes in 44.445472717 + 724631.800000000 as the first argument to the constructor and the result is stored internally and is seen to be 724676.250000000. Does anyone know the cause of this rounding? BTW he uses VS2008. This works in VS2010.
-
NB: As indicated in a reply the issue has been solved. The DirectX module was changing the floating point precision. A colleague is calling the constructor of a class which takes two doubles as arguments. Each argument he passes is the sum of two doubles: CGridPoint( a + b, c + d) The class object stores the two values internally. When he views the result he sees that the values have been rounded to 1 decimal place. The values are not so large that this would be expected. Here are example: a = 44.445472717 b = 724631.800000000 a + b = 724676.250000000 In other words he passes in 44.445472717 + 724631.800000000 as the first argument to the constructor and the result is stored internally and is seen to be 724676.250000000. Does anyone know the cause of this rounding? BTW he uses VS2008. This works in VS2010.
A quick look at the rounded result (7xxx.25) shows it to have about the precision we'd expect from
float. I'd wager that there's an explicit or implicit conversion to float and back again hidden in there somewhere. Looks like you'll have to pick through it line by line. PeterSoftware rusts. Simon Stephenson, ca 1994.
-
I can't see the
CGridPointclass in ATL or MFC; is this a Microsoft or home produced class?Unrequited desire is character building. OriginalGriff
He has found the cause. A call to DirectX was changing the floating point (double) precision ..