Re: CSVR4 and UTF-8

Hello,

It has been nearly 20 years since the Mac stopped using CR as an end-of-line character.  Today's MacOS is fully POSIX compliant, and uses LF.  The older Apple computers such as the Apple //e also used CR, as did many of its contemporaries, such as the Commodore 64.

I did not plan to support CR-delimited lines in CSVR4.

Do you have a need to handle CR delimited lines?

-SK

On 5/7/2020 8:44 AM, Tools/400 wrote:

Hi Scott,

People may call that "irrelevant" or "hypothetical", but now it is the "testCrUnicode" test case that fails with a "A character representation of a numeric value is in error.".

The reason is that stmfReadLine() does not honor CR as the end-of-line character. It reads the entire file for the first out of three records. As far as I know, CR is used on Mac as the linefeed character.

For example: https://www.oreilly.com/library/view/mac-os-x/0596004605/ch01s06.html

"The Mac, by default, uses a single carriage return (<CR>), represented as \r. Unix, on the other hand, uses a single linefeed (<LF>), \n. Windows goes one step further and uses both, creating a (<CRLF>) combination, \r\n."

The unit test ends successfully for the other two test cases with LF and CRLF linefeed characters.

Regards,

Thomas.

Am 07.05.2020 um 11:36 schrieb Scott Klement:

Hello Thomas and everyone,

I've updated the copy of the CSV utilities on my web site to fix the problems Thomas mentions, below.

Let me know if there's anything else I can do.

-SK

On 5/6/2020 11:58 PM, Tools/400 wrote:

Hi Scott,

For 1) -- That is what I expressed earlier this day (March, 6th):

"In this example there is an unexpected x'00' before x'00A0', so that we would get an extra x'00' byte if we scanned for x'00A0'. Looks like a bug in fgets()."

In addition to that it might be worth to know that there is a PTF for the Unix-type read() function, which fixes a problem for UTF8 files with a BOM. We spotted that problem earlier this year:

https://www.ibm.com/support/pages/ptf/MF67069

For 2) -- You are correct. I did not look at your code carefully enough. When it comes to an "If" I always think about the "else". Hence I automatically cleared "peFldData".

Regards,

Thomas.