× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.


  • Subject: RE: Character to Numeric
  • From: boldt@xxxxxxxxxx
  • Date: Fri, 1 Dec 2000 08:45:31 -0500
  • Importance: Normal


Brad wrote:
>I haven't had any problems.  This is basically the IBM one.  The original
>IBM one produced wrong results 99% of the time (ie 1.9999999 instead of
2).
>So I just added a half adjust and so far it's worked flawlessly for 3
years
>or more.

Sorry Brad, Barbara's right.  (She usually is!)  Just feed your procedure
a large value, say '123456789123456789.123456789'.  I just tried it out
and it gives a result of 123456789123456784.000000000!  That's what using
floating point can do to you.

Bummer to get news like this on a friday, isn't it?  Anyways, I'm sure we
can expect to find the fixed version of the procedure on your web page
later today, right?


Hans Boldt, ILE RPG Development, IBM Toronto Lab, boldt@ca.ibm.com

+---
| This is the RPG/400 Mailing List!
| To submit a new message, send your mail to RPG400-L@midrange.com.
| To subscribe to this list send email to RPG400-L-SUB@midrange.com.
| To unsubscribe from this list send email to RPG400-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---

As an Amazon Associate we earn from qualifying purchases.

This thread ...

Follow-Ups:

Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.