Float problems

[dbytz]

I'm using a float to keep track of a scale factor -- sounds easy enough. When the user clicks a button I either add or subtract 0.10F from the scale value (it starts at 1.0F). After several iterations I start to see the value varing by a small amount i.e instead of 0.80 I get .7999. This seems like a large error for a simple addition or subtraction operation... Any insights?

[Dave Kreskowiak]

This is because of the internal representation of a float type. When you store a number in a 32-bit float, you're not storing the absolute precise version of the number. You actually storing a mantissa and exponent. Try this[^] for a very short explanation as to why... 0.1 in decimal does NOT = 0.1 represented in IEEE754 format. RageInTheMachine9532 "...a pungent, gastly, stinky piece of cheese!" -- The Roaming Gnome

[Michael Potter]

Use an integer for the scale factor. Start at 100:

int iScale = 100;

Move up or down by 10.

iScale += 10;

Convert to float before using:

float fScale = (float)iScale/100f;

This should keep your scaling the way you want.

[dbytz]

Thanks. I had considered the integer solution, it will work for the case outlined but if the scalinng capability is expanded to include "To width" etc... it may not not work as well, though I may be able to adjust the precision to account for those situtations.

[dbytz]

I'd have to read the IEEE754 spec to see why it's different than normal FP calculations, i.e. the same calcutions in C/C++ work fine with floats, but are way off with both floats and doubles in C#. I understand the internal FP representation, but I haven't seen this bad/apperant/annoying of a problem for some years, hmmm maybe somewhat over a decade or more....

[Baris Kurtlutepe]

Have you considered using the decimal data type? It does not lose precision since it is intended for decimal arithmetic.

[Dave Kreskowiak]

Decimal is also 128-bits in size compared to the 32-bit for a float... A little overkill for the application... RageInTheMachine9532 "...a pungent, gastly, stinky piece of cheese!" -- The Roaming Gnome

[Dave Kreskowiak]

I've had to deal with it for years. I've always used the 'integer' technique simply because math with int's are much quicker than with floats. Only when I must use floats/doubles do they get converted back to their floating point counterparts. RageInTheMachine9532 "...a pungent, gastly, stinky piece of cheese!" -- The Roaming Gnome

[Baris Kurtlutepe]

Dave Kreskowiak wrote: Decimal is also 128-bits in size compared to the 32-bit for a float... A little overkill for the application... Yes, decimals are stored as 16 bytes (but only 13 of them are used, pity) In any case they're much slower than a float, since they are structs and do not have hardware support. But I think it's OK unless you're programming a game or an embedded system in a spaceship :)

[dbytz]

No, I don't think that I'll use decimal, espcially since I was worried that converting the int to a float every thime the window was invalidate would be too expensive (a few second later I figured out that I could store both -- do the math on the int and convert to float whe the scale changes.) I guess the major question is why when you write a prog in C/C++ do you get the "right"/expexted answer and in C# you don't. Ah well, if it still bothers me tommorrow I may take a journey down assembly lane and look at the disassembly for the C code. Another thing I was considering checking is to simply assign various values to a float an view the results...