cast and precision

I'm doing a series of principal * rate calculations and the rate lookup
itself is calculated in a sql statement. When I first wrote the query, vfp
showed me a cursor with 4 decimal places in the rates col .. which isn't
enough for my users, so I redid the query using cast( <calc> as N(10,6)).
While this has certainly improved the appearance of the col in the cursor,
floating point calc's being what they are... has it also increased its
accuracy?
-Lew

Anders
Thu Dec 06 14:38:39 PST 2007

If you really calculate 6 decimals the integer par cannot exceed 999
'999.000001' is the max length of your number, 9 digits and a decimal point.
If you use fewer decimals the length if the integer can increase-
-Anders


"Lew" <Lew@discussions.microsoft.com> wrote in message
news:3521B415-D199-465B-AED8-3BFD3D1CA7A5@microsoft.com...
> I'm doing a series of principal * rate calculations and the rate lookup
> itself is calculated in a sql statement. When I first wrote the query, vfp
> showed me a cursor with 4 decimal places in the rates col .. which isn't
> enough for my users, so I redid the query using cast( <calc> as N(10,6)).
> While this has certainly improved the appearance of the col in the cursor,
> floating point calc's being what they are... has it also increased its
> accuracy?
> -Lew

Bernhard
Fri Dec 07 03:27:57 PST 2007

Hi Lew,

> I'm doing a series of principal * rate calculations and the rate lookup
> itself is calculated in a sql statement. When I first wrote the query, vfp
> showed me a cursor with 4 decimal places in the rates col .. which isn't
> enough for my users, so I redid the query using cast( <calc> as N(10,6)).
> While this has certainly improved the appearance of the col in the cursor,
> floating point calc's being what they are... has it also increased its
> accuracy?

What is the type of your base fields principal and rate. From the automatic
result type of 4 decimal places, I guess something like N(...,2). Then you never
need more decimal places for the result of a single multiplication of those two
values.
You need more decimal places, if there is division or more than one multiplication.

If the type of your base fields is double, then with your cast( ... as N(...))
you will almost always cut away some of the calculated decimal places. The
automatic result type should be double.

Regards
Bernhard Sander