Re: CSV files with Byte Over Marks

On 6/19/2018 11:42 AM, Soucy, Michael wrote:

I've been working with both the DATA-INTO and the free CSV utility that Scott Klement wrote. I have a couple of CSV files that I generate daily out of our payroll system. If I try and do a simple client access data transfer I end up with some weird characters in the first column of the first record. I also get similar results when I use the DATA-INTO and the CSV utility from Scott Klement. If I run the program in debug mode I see that the first field of the first column is padded with what looks like this.


It was suggested to me that the issue could be that the CCSID is not UTF-8. It was also mentioned that I may be dealing with files that have BOM (Byte over mark) in them. I'm not sure if I'm doing this correctly, but if I do a WRKLNK, locate the file, and check the properties of the file on the IFS I see the coded character set ID is 1252. I'm not sure if that means the file is UTF-8 or not.

CCSID 1252 is Windows Latin 1
http://www-01.ibm.com/support/docview.wss?uid=nas8N1012954

UTF-8 would be CCSID 1208
http://www-01.ibm.com/software/globalization/ccsid/ccsid_registered.html

Wikipedia has a decent article on the Byte Order Mark
https://en.wikipedia.org/wiki/Byte_order_mark even noting what the BOM
looks like in Windows Latin 1

How do I get around the issue of the BOM? Any help would be greatly appreciated. What I have been doing in the mean time is downloading the file as an Excel file, opening it up in Excel, and then saving it as a CSV file. Doing that drops the BOM, but that seems like a lot of extra work.

The CCSID attribute for stream files are like the TEXT() parameter on
physical files. Advisory, informational, but not at all guaranteed to
be representative of the actual contents of the file.

I think that the first step is to make the CCSID of the stream file
match the CCSID of the data being poured into it. I'm guessing that
with a BOM present, the actual character encoding is UTF-8 (1208), so
that means you need to delete the existing stream file and create a new
one with CCSID 1208. Scott wrote a very informative post on this
https://archive.midrange.com/midrange-l/201301/msg01167.html

Carel noted CHGATR. There's nothing wrong with that, but I like to have
the RPG program set the CCSID itself. All the pieces in one bit of code.