Floating point ambiguous errors
-
Okay, here's the deal. Let's just say I have a variable, amtChange, which is a single floating point number. Let's say that it's 1.23 in this example. If I subtract amtDol, which is an integer with a value of 1, I end up with .2299995 as a difference. I find that this happens often. How can I prevent and/or fix this? Oh, here's my code:
Private amtOwed, amtPaid, amtChange As Single Private amtDol, amtQuar, amtDime, amtNick, amtPen As Integer *** amtOwed = Val(txtOwed.Text) amtPaid = Val(txtPaid.Text) amtChange = amtPaid - amtOwed amtDol = Int(amtChange) amtChange = amtChange - amtDol MsgBox(amtChange)And the Message Box gives me .2299995! -
Okay, here's the deal. Let's just say I have a variable, amtChange, which is a single floating point number. Let's say that it's 1.23 in this example. If I subtract amtDol, which is an integer with a value of 1, I end up with .2299995 as a difference. I find that this happens often. How can I prevent and/or fix this? Oh, here's my code:
Private amtOwed, amtPaid, amtChange As Single Private amtDol, amtQuar, amtDime, amtNick, amtPen As Integer *** amtOwed = Val(txtOwed.Text) amtPaid = Val(txtPaid.Text) amtChange = amtPaid - amtOwed amtDol = Int(amtChange) amtChange = amtChange - amtDol MsgBox(amtChange)And the Message Box gives me .2299995!i'm not expert in this but try to change the data type into double :)
-
Okay, here's the deal. Let's just say I have a variable, amtChange, which is a single floating point number. Let's say that it's 1.23 in this example. If I subtract amtDol, which is an integer with a value of 1, I end up with .2299995 as a difference. I find that this happens often. How can I prevent and/or fix this? Oh, here's my code:
Private amtOwed, amtPaid, amtChange As Single Private amtDol, amtQuar, amtDime, amtNick, amtPen As Integer *** amtOwed = Val(txtOwed.Text) amtPaid = Val(txtPaid.Text) amtChange = amtPaid - amtOwed amtDol = Int(amtChange) amtChange = amtChange - amtDol MsgBox(amtChange)And the Message Box gives me .2299995! -
Hi, You can try this. Define all the variables as double or type cast it into double and then subtract it. Take the result into a variable which is double. Tasnim
As a matter of fact, it turns out the Decimal variable type doesn't generate ambiguous errors. Thanks for the input, though. I really appreciate it! :D