precision problems with double type
-
I'm having precision problems with the double type, the help files say I should get 15 digits of precision... BUT...
double d1 = 12589230.0342; double d2 = -10.335927963256836; //good precision, but when added, problems occur double newd = d1+d2; // newd now equals 12589220.0Where's all my precision???? I've tried decimal type, but is's incredibly slow.... I have to use it when multiplying, but addition of 10??? Any suggestions? should I use my own type? Possibly a binary coded decimal Thanks for any help. -Sam -
I'm having precision problems with the double type, the help files say I should get 15 digits of precision... BUT...
double d1 = 12589230.0342; double d2 = -10.335927963256836; //good precision, but when added, problems occur double newd = d1+d2; // newd now equals 12589220.0Where's all my precision???? I've tried decimal type, but is's incredibly slow.... I have to use it when multiplying, but addition of 10??? Any suggestions? should I use my own type? Possibly a binary coded decimal Thanks for any help. -SamThe result I see in the watch window is 12589219.698272036, not 12589220.0. (the real result is 12589219.698272036743164). The decimal type can be used (28 digit precision) : decimal d1 = 12589230.0342M; decimal d2 = -10.335927963256836M; decimal d = d1 + d2; and I get the real result in the watch window.
-
The result I see in the watch window is 12589219.698272036, not 12589220.0. (the real result is 12589219.698272036743164). The decimal type can be used (28 digit precision) : decimal d1 = 12589230.0342M; decimal d2 = -10.335927963256836M; decimal d = d1 + d2; and I get the real result in the watch window.
YOU'RE right!!!!! Now that makes things even weirder.... My actual code.... public void Translate(double x, double y, double z) { foreach(DPoint p in points) { p.x += x; p.y += y; p.z += z; } } a Dpoint is just 3 doubles x,y,z on the first iteration of the foreach: x=12589230.0342 y = 693719.8863 z = 0.0 p.x = -10.335927963256836 p.y = 9.6102085113525391 p.z = 5.7499995231628418 after first iteration p.x = 12589220.0 p.y = 693729.5 p.z = 5.7499995231628418 I'm trying to replicate problem is a simple app now...
-
YOU'RE right!!!!! Now that makes things even weirder.... My actual code.... public void Translate(double x, double y, double z) { foreach(DPoint p in points) { p.x += x; p.y += y; p.z += z; } } a Dpoint is just 3 doubles x,y,z on the first iteration of the foreach: x=12589230.0342 y = 693719.8863 z = 0.0 p.x = -10.335927963256836 p.y = 9.6102085113525391 p.z = 5.7499995231628418 after first iteration p.x = 12589220.0 p.y = 693729.5 p.z = 5.7499995231628418 I'm trying to replicate problem is a simple app now...
This snippet:
decimal px = -10.335927963256836M; decimal ix = 12589230.0342M; decimal nx= px + ix; double d = Convert.ToDouble(nx); double c = 2;Works as long as it happens before this line: device = new Device(graphicsSettings.AdapterOrdinal, graphicsSettings.DevType, windowed ? ourRenderTarget : this , createFlags, presentParams); If it happens the line after I get my problem.... now that is messed up, how can it change the behavior of a built in type???? -
This snippet:
decimal px = -10.335927963256836M; decimal ix = 12589230.0342M; decimal nx= px + ix; double d = Convert.ToDouble(nx); double c = 2;Works as long as it happens before this line: device = new Device(graphicsSettings.AdapterOrdinal, graphicsSettings.DevType, windowed ? ourRenderTarget : this , createFlags, presentParams); If it happens the line after I get my problem.... now that is messed up, how can it change the behavior of a built in type????