rounding problem

In my code I have to calculate the difference between two
numeric values stored as strings and then store the result
as a string.

Using cstr(cdbl(strA) - cdbl(strB)) works fine most of the
time, but I encountered at least one exception:

when strA = 0.92 and strB = 0.85 the result always is:

cdbl(0.92) - cdbl(0.85) = 0.070000000000000062

Any ideas on why this is happening?

I think I have a workaround by using convert.todecimal
inplace of cdbl, but I'm afraid this can result in some
precision loss when converting from decimal back to string.

Tom
Tue Dec 02 15:56:38 CST 2003

Have you considered using the Decimal type instead?
Tom Clement
Apptero, Inc.


"Isaak" <anonymous@discussions.microsoft.com> wrote in message
news:043301c3b91d$3d577dc0$a301280a@phx.gbl...
> In my code I have to calculate the difference between two
> numeric values stored as strings and then store the result
> as a string.
>
> Using cstr(cdbl(strA) - cdbl(strB)) works fine most of the
> time, but I encountered at least one exception:
>
> when strA = 0.92 and strB = 0.85 the result always is:
>
> cdbl(0.92) - cdbl(0.85) = 0.070000000000000062
>
> Any ideas on why this is happening?
>
> I think I have a workaround by using convert.todecimal
> inplace of cdbl, but I'm afraid this can result in some
> precision loss when converting from decimal back to string.

Jon
Wed Dec 03 05:26:11 CST 2003

Isaak <anonymous@discussions.microsoft.com> wrote:
> In my code I have to calculate the difference between two
> numeric values stored as strings and then store the result
> as a string.
>
> Using cstr(cdbl(strA) - cdbl(strB)) works fine most of the
> time, but I encountered at least one exception:
>
> when strA = 0.92 and strB = 0.85 the result always is:
>
> cdbl(0.92) - cdbl(0.85) = 0.070000000000000062
>
> Any ideas on why this is happening?

Yes - it's because neither 0.92 nor 0.85 are exactly representable in
floating binary point.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html for more
information.

--
Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too