Single Precision not precise
-
I noticed some very stange values coming back from some calculations in a datatable and decided to investigate. Has anyone come across the problem below ?CSng(.51) + 0.01 0.51999999046325684 ?CSng(.57) + CSng(1) 1.56999993 I assume this is something to do with the way that the floating point value is held, but I've never seen it before. I'm using VB.NET to do this, but has anyone else noticed similar behaviour in other flavours of .NET? :confused:
-
I noticed some very stange values coming back from some calculations in a datatable and decided to investigate. Has anyone come across the problem below ?CSng(.51) + 0.01 0.51999999046325684 ?CSng(.57) + CSng(1) 1.56999993 I assume this is something to do with the way that the floating point value is held, but I've never seen it before. I'm using VB.NET to do this, but has anyone else noticed similar behaviour in other flavours of .NET? :confused:
What Every Computer Scientist Should Know About Floating Point Arithmetic[^]. Stability. What an interesting concept. -- Chris Maunder
-
I noticed some very stange values coming back from some calculations in a datatable and decided to investigate. Has anyone come across the problem below ?CSng(.51) + 0.01 0.51999999046325684 ?CSng(.57) + CSng(1) 1.56999993 I assume this is something to do with the way that the floating point value is held, but I've never seen it before. I'm using VB.NET to do this, but has anyone else noticed similar behaviour in other flavours of .NET? :confused:
Yes, anyone who has done any floating-point calculations on a computer has run into this. Welcome to floating-point. PeterRitchie.com