So what you're saying is that truncation is mandatory, so DB2 is doing high order truncation instead of low order.
I went back to where I'm originally calculating A, and forced it to use a shorter length. That worked.
Thanks
-----Original Message-----
From: Birgitta Hauser [mailto:Hauser@xxxxxxxxxxxxxxx]
Sent: Friday, February 12, 2016 12:09 AM
To: 'Midrange Systems Technical Discussion' <midrange-l@xxxxxxxxxxxx>
Subject: AW: SQL precision problem
The problem occurs because the calculation is performed under the cover as float.
One rule when working with float is, if the (calculated) maximum length is reached, decimal positions are cut at the end.
I assume the length for your numeric values in your job is 31 (can be changed up to 63 in STRSQL and IBM i Navigator's or ACS's RUN AN SQL SCRIPT)
If you change the cast function to a precicion of 24 or , it calculates correctly.
Select x.*,
(A - B) / B * 100.00 as RESULT
from (Values(cast(1.5267 as decimal(24, 4)), cast(1,6 as decimal(12, 4)))) x (a, b);;
If you change the maximum precision in your job/program from 31 to 63 it also return the expected result.
Mit freundlichen Grüßen / Best regards
Birgitta Hauser
"Shoot for the moon, even if you miss, you'll land among the stars." (Les
Brown)
"If you think education is expensive, try ignorance." (Derek Bok) "What is worse than training your staff and losing them? Not training them and keeping them!"
-----Ursprüngliche Nachricht-----
Von: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] Im Auftrag von Justin Taylor
Gesendet: Thursday, 11.2 2016 22:50
An: MIDRANGE-L (midrange-l@xxxxxxxxxxxx)
Betreff: SQL precision problem
I'm trying to figure out how SQL arithmetic involving decimals is giving an integer result. I've put together a sample query that replicates my question.
with Values as (
select cast(1.5267 as decimal(31,4)) as A, cast(1.6 as decimal(12,4)) as B
from SYSIBM.SYSDUMMY1
)
select Values.*, (A - B) / B * 100.00 as RESULT
from Values;
This gives: 1.5267 1.6 0
The RESULT column has a type of decimal(31,2), but the value is zero. With a precision of 2, RESULT should actually be -4.58.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit:
http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at
http://archive.midrange.com/midrange-l.
Please contact support@xxxxxxxxxxxx for any subscription related questions.
midrange.com
