RE: SQL ROUND function on a FLOAT

IANAM  (I Am Not A Mathematician ;-)

But I'd guess that it is working as designed.

Floating point arithmetic has some weird rules, mathematical operations
on floats often lead to "unexpected" results. ( see: "What Every
Computer Scientist Should Know About Floating-Point Arithmetic
http://docs.sun.com/source/806-3568/ncg_goldberg.html#693 )

I'd say if you want 3.41739e0 to be rounded to 2 decimals like the
fix-decimal 3.41739 would be, the first thing you should do is convert
it to fixed decimal.

Do you really have to use float here? Is it really a floating point
value?

Charles Wilt
--
iSeries Systems Administrator / Developer
Mitsubishi Electric Automotive America
ph: 513-573-4343
fax: 513-398-1121
  

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx 
[mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Mark Adkins
Sent: Thursday, August 24, 2006 11:34 AM
To: rpg400-l@xxxxxxxxxxxx
Subject: SQL ROUND function on a FLOAT

This isn't so much a question but more like a "what the?!".

If I have the float value "3.4173900000000001E+000" and issue 
a "ROUND( 
3.4173900000000001E+000, 2 )", I get (drum roll) 
3.4199999999999999E+000. 
Cast as a dec with 2 decimal places and viola!  You get 
3.41739 rounded to 
3.41.

This statement shows it all. 

SELECT FLOAT(3.41739) AS NUM, 
       ROUND( FLOAT(3.41739), 2 ) AS ROUNDED_NUM, 
       CAST( ROUND( FLOAT(3.41739), 2 ) AS DEC(5,2) )
        AS CASTED_ROUNDED_NUM 
FROM SYSIBM/SYSDUMMY1 

Output:
                    NUM               ROUNDED_NUM  CASTED_ROUNDED_NUM
3.4173900000000001E+000   3.4199999999999999E+000          3.41 

Of course I know to cast it to 3 decimal places before the round, but 
still...

Mark
-- 
This is the RPG programming on the AS400 / iSeries (RPG400-L) 
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.