divide by 10 problem
-
This has been a problem of mine for years. It appears to be compiler specific which is why I put it in this section. Run this piece of code double test = 3.0/10.0; The result of test is .299999999999999 I know why this happens. I was just wondering how some of you have solved this problem. :-D Thanks for any input
George W Software Developer www.zsystems.ca
-
This has been a problem of mine for years. It appears to be compiler specific which is why I put it in this section. Run this piece of code double test = 3.0/10.0; The result of test is .299999999999999 I know why this happens. I was just wondering how some of you have solved this problem. :-D Thanks for any input
George W Software Developer www.zsystems.ca
IGeorgeI wrote:
I know why this happens. I was just wondering how some of you have solved this problem.
If you know why it happens, then you must also know it has no solution. Floating-point numbers behave that way on binary machines.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
-
IGeorgeI wrote:
I know why this happens. I was just wondering how some of you have solved this problem.
If you know why it happens, then you must also know it has no solution. Floating-point numbers behave that way on binary machines.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
Of course but when you are dealing with high precision numbers, you have to implement workarounds to solve the problem. I was just wondering how others have handled it. You simply cannot leave it like that when dealing with mathematical engines written in c++. The error will carry through and get worse. So I just wanted to see what others are doing. For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
George W Software Developer www.zsystems.ca
-
Of course but when you are dealing with high precision numbers, you have to implement workarounds to solve the problem. I was just wondering how others have handled it. You simply cannot leave it like that when dealing with mathematical engines written in c++. The error will carry through and get worse. So I just wanted to see what others are doing. For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
George W Software Developer www.zsystems.ca
IGeorgeI wrote:
For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
Perhaps it makes use of some sort of epsilon value. See here for more.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
-
This has been a problem of mine for years. It appears to be compiler specific which is why I put it in this section. Run this piece of code double test = 3.0/10.0; The result of test is .299999999999999 I know why this happens. I was just wondering how some of you have solved this problem. :-D Thanks for any input
George W Software Developer www.zsystems.ca
IGeorgeI wrote:
I was just wondering how some of you have solved this problem.
add a tiny amount to the result of the division, then truncate. aka. 'round up' yes, it biases the results in one direction, but them's the breaks.
image processing toolkits | batch image processing | blogging
-
Of course but when you are dealing with high precision numbers, you have to implement workarounds to solve the problem. I was just wondering how others have handled it. You simply cannot leave it like that when dealing with mathematical engines written in c++. The error will carry through and get worse. So I just wanted to see what others are doing. For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
George W Software Developer www.zsystems.ca
IGeorgeI wrote:
For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
First, this isn't a C# forum. Second, If you're doing this in C++, you're going to have to write some code to "fix" it. Write yourself a rounding function that will round to the nearest whole number. It ain't rocket science. Third, you're arguing with someone that's trying to help you. Not very smart, even for a C# programmer.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001 -
IGeorgeI wrote:
For your information, the c# compiler does solve the problem and does give you to correct base 10 answer of .30
First, this isn't a C# forum. Second, If you're doing this in C++, you're going to have to write some code to "fix" it. Write yourself a rounding function that will round to the nearest whole number. It ain't rocket science. Third, you're arguing with someone that's trying to help you. Not very smart, even for a C# programmer.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001I am not a C# programmer by trade. I am a C/C++/Java/MFC developer. 1) I posted my C++ question in the C++ section of this forum. I merely added the fact about the C# compiler to help other people. You must have misunderstood my post. 2)I have written work arounds for this problem, I just wanted to see how other people have handled this problem out of pure interest. We can all become better programmers if we want to. You must have misunderstood my post. 3) At no point was I arguing with him. Why would I argue with somebody who is clearly trying to help me?? You must have misunderstood my post. I would suggest that in the future you should try reading the post more closely to avoid misunderstandings like this. If you want, you can contact me anytime and I will help you with that. Kindest Regards
George W Software Developer www.zsystems.ca
-
This has been a problem of mine for years. It appears to be compiler specific which is why I put it in this section. Run this piece of code double test = 3.0/10.0; The result of test is .299999999999999 I know why this happens. I was just wondering how some of you have solved this problem. :-D Thanks for any input
George W Software Developer www.zsystems.ca
First of all I want to state that I have read the other posts and would have ignored that one about C#. This is not a compiler specific problem, it is a hardware problem and I do not know any solution. Although there is an excellent book called “Numerical Recipes in …” (very old now) that probably addresses these issues. If Microsoft has introduced a way to solve it, via software, they have not been vocal about it. The only thing I can think of is rounding up when you see all those ‘9’s’.
INTP "Program testing can be used to show the presence of bugs, but never to show their absence."Edsger Dijkstra
