Re: RPG math

On Thu, Apr 12, 2012 at 10:08 AM, Charles Wilt <charles.wilt@xxxxxxxxx> wrote:

Why worry about it RPG when you don't appear
to be worried about it Excel?

I'm sure the issue is just satisfying the customer (i.e. the
spec-maker) and most customers trust Excel. So the spec he's been
given is most likely "get it to match Excel", regardless of which is
more correct or more appropriate.

Given that Excel apparently stores numbers as floating
point and that there are differences between floating
point arithmetic and RPG's fixed decimal arithmetic...
expecting the same results is probably a fantasy.

It's even more complicated than that, because Excel doesn't *strictly*
adhere to the IEEE floating point standard. (It tries to do little
bits of finagling so that it appears to behave a little bit more like
what normal people expect when doing calculations; the result being a
slight *loss* of precision when compared to the IEEE standard.)

Personally, I'd consider RPG's fixed decimal answer "right" over
Excel's floating point answer when dealing with real world values
especially for currency.

The *real* real world is mostly messy numbers that are just as poorly
represented by finite base-10 approximations as by finite base-2
approximations. For virtually all scientists and engineers, both are
bad, so you might as well go with the one that computers like better.
But I certainly agree with using base 10 in the business world.

However, if you want to consider Excel's floating point
answer "right", it would behoove you to use floating
point variables in RPG to match.

This will get you closer, but as I mentioned above, it's still not
quite the same.

The true "right" solution in my mind
1) use RPG fixed point
2) Understand RPG's fixed point precision rules
3) use %round()  and/or %dec() as needed to control intermediate
results as you want.

I think this is probably better than using RPG floats, even if you are
trying to emulate Excel.

John