Float numbers in RPG IV (was: RE: Procedue with *varsize parm)

Chanh Le wrote:
>... I also encounter another similar
>issue of using %float and %dec
>
>FLD01  (21,0)  value   000000000000033355555
>FLD02  (15,0)  value   000000000100000
>FLD03  (30,9)
>
>FLD03 = %float(FLD01 / FLD02) = 000000000000000000333.555549999
>FLD03 = %dec(FLD01 / FLD02)  = 000000000000000000333.555550000

What you're seeing is normal behaviour for floating point
numbers.  Using float numbers, you have a vast range of
possible values, but the numbers are never completely
precise.  At best, using 8-byte values, you never have
better than about 16 digits of precision.  And very often,
like you see here, inaccuracy in the least significant
digit skews the value downwards, resulting in that string
of nines.

Floating point is ideally suited for modelling physical
processes, such as computing satellite orbits.  But since
physicists deal with measured values, their numbers are
never fully precise anyways.

Generally, the RPG programmer really shouldn't worry too
much about floating point.  However, there are a couple
of places where you need to understand floating point.
For example, the ** operator returns a floating point
value.  Furthermore, SQRT is done in long float format.

If you're dealing with numeric amounts that absolutely
must be accurate to the last digit, such as amounts of
money, avoid floating point.  If your monetary calculation
involves using the ** operator (useful for things like
calculating interest), put the ** computation in a
separate expression, and store the result in a decimal
variable.

Cheers!  Hans

Hans Boldt, ILE RPG Development, IBM Toronto Lab, boldt@ca.ibm.com


+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---