× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.


  1.  Home
  2. RPG400-L
  3. July 2023

Re: Odd, inconsistent rounding error

On 7/27/23 9:26 PM, Daniel Gross wrote:

But in your case, one of the problems seems to be U01168 which is
11,2 where all other fields are nn,5. Whenever you assign a value to
this field, the precision is reduced to 11,2 - and in you case
without rounding.

The issue is not that there's a rounding error, but that the rounding error is *inconsistent* between the existing version and my version. And the drop from nn.5 down to 11.2 is a constant across both versions.

--
JHHL

As an Amazon Associate we earn from qualifying purchases.

This thread ...
Replies:
Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.