Re: CSV files with Byte Over Marks -- RPG400-L

Michael,

Why would 1208 produce a conversion error? Is the data not properly encoded for 1208? Or, is the program not written to handle conversions properly?

You are referring to something that I wrote. Can you tell me which utility this is, and what the code looks like? I have written many things that could be called "CSV utilities", and if the problem is the way my code is opening the file, I'd need to to see the specific code in order to comment on how it can be fixed.

Of course, if you got it from me, you undoubtedly have the source code and could also fix it yourself. But, I'd be glad to give my thoughts, if I understood which tool we were working with.

-SK

On 6/19/2018 12:21 PM, Soucy, Michael wrote:

I seem to be stuck between a rock and a hard place. I tried changing the attributes using the CHGATR command. When I do that it removed the BOM from the file, but now the program I wrote using Scott Klement's CSV parsing utility won't work. It's throwing a conversion error. I can't create the CSV file with CCSID 1208 because it's the result of a report I run from our payroll company (ADP). I contacted ADP before about it, and they stated that the file is being generated in the proper format and there is nothing for them to change.

Mike

-----Original Message-----
From: RPG400-L [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Buck Calabro
Sent: Tuesday, June 19, 2018 12:31 PM
To: rpg400-l@xxxxxxxxxxxx
Subject: Re: CSV files with Byte Over Marks

On 6/19/2018 11:42 AM, Soucy, Michael wrote:

I've been working with both the DATA-INTO and the free CSV utility that Scott Klement wrote. I have a couple of CSV files that I generate daily out of our payroll system. If I try and do a simple client access data transfer I end up with some weird characters in the first column of the first record. I also get similar results when I use the DATA-INTO and the CSV utility from Scott Klement. If I run the program in debug mode I see that the first field of the first column is padded with what looks like this.
ï»¿

It was suggested to me that the issue could be that the CCSID is not UTF-8. It was also mentioned that I may be dealing with files that have BOM (Byte over mark) in them. I'm not sure if I'm doing this correctly, but if I do a WRKLNK, locate the file, and check the properties of the file on the IFS I see the coded character set ID is 1252. I'm not sure if that means the file is UTF-8 or not.

CCSID 1252 is Windows Latin 1
http://www-01.ibm.com/support/docview.wss?uid=nas8N1012954

UTF-8 would be CCSID 1208
http://www-01.ibm.com/software/globalization/ccsid/ccsid_registered.html

Wikipedia has a decent article on the Byte Order Mark https://en.wikipedia.org/wiki/Byte_order_mark even noting what the BOM looks like in Windows Latin 1

How do I get around the issue of the BOM? Any help would be greatly appreciated. What I have been doing in the mean time is downloading the file as an Excel file, opening it up in Excel, and then saving it as a CSV file. Doing that drops the BOM, but that seems like a lot of extra work.

The CCSID attribute for stream files are like the TEXT() parameter on physical files. Advisory, informational, but not at all guaranteed to be representative of the actual contents of the file.

I think that the first step is to make the CCSID of the stream file match the CCSID of the data being poured into it. I'm guessing that with a BOM present, the actual character encoding is UTF-8 (1208), so that means you need to delete the existing stream file and create a new one with CCSID 1208. Scott wrote a very informative post on this https://archive.midrange.com/midrange-l/201301/msg01167.html

Carel noted CHGATR. There's nothing wrong with that, but I like to have the RPG program set the CCSID itself. All the pieces in one bit of code.

--
--buck

http://wiki.midrange.com
Your updates make it better!

--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: http://amzn.to/2dEadiD