Math.Floor problem [modified]
-
How come Math.Floor(0.3 / 0.1) returns 2.0??? I assume it is some sort of rounding issue. How can I fix this? Edit: This is what I came up with: Math.Floor(Math.Round(0.3 / 0.1, 2)); Is there an easier way? Thanks
modified on Tuesday, December 15, 2009 1:03 PM
-
How come Math.Floor(0.3 / 0.1) returns 2.0??? I assume it is some sort of rounding issue. How can I fix this? Edit: This is what I came up with: Math.Floor(Math.Round(0.3 / 0.1, 2)); Is there an easier way? Thanks
modified on Tuesday, December 15, 2009 1:03 PM
It's because
0.3/0.1returns a double, and there are all sorts of rounding problems associated with doubles.Console.WriteLine(Math.Floor(0.3M / 0.1M));uses decimal arithmatic, and you get the expected answer.CCC solved so far: 2 (including a Hard One!) 37!?!! - Randall, Clerks
-
How come Math.Floor(0.3 / 0.1) returns 2.0??? I assume it is some sort of rounding issue. How can I fix this? Edit: This is what I came up with: Math.Floor(Math.Round(0.3 / 0.1, 2)); Is there an easier way? Thanks
modified on Tuesday, December 15, 2009 1:03 PM
keefb is 100% correct. If you type "0.3 / 0.1" into the immediate window, it spits out 2.9999999999999996... Now, we all know that's pretty much "3", but if you do a Floor(), it rounds down to 2. As keefb said, use decimals and you'll get the right answer. But if you need to work with doubles, or have other constraints, you could "hack" it by, say, adding 0.0000001 before going through the Floor()... Kind of like the stone-age trick of adding 0.5 before a truncation to simulate rounding up or down to an integer.
Proud to have finally moved to the A-Ark. Which one are you in? Author of Guardians of Xen (Sci-Fi/Fantasy novel)