Re: Chinese characters in an interface file

On 2020-05-13 11:22 a.m., Vernon Hamberg wrote:

Hi

It is possible that the CSV file is in UTF-8 -- the hex 31 you describe is what the number 1 would be in UTF-8.
...

If it IS UTF-8, you might try marking the file with CCSID 1208 and do a text transfer, not a binary transfer.

Or mark the field in your PF as 1208 CCSID, do the binary transfer, then see what RPG does with it. In RPG, do and EVAL from the UTF-8 field to a regular 37 EBCDIC field.
...

I agree with Vern that it is likely that you have UTF-8 data.

If you get the data into a UTF-8 RPG field and assign it to a CCSID 37 using RPG, the characters that can't convert to CCSID(37) will be x'3F'. You could then scan for x'3F' and blank the field if you find one.

Or, if you're specifically only concerned with Chinese, you could assign to a CCSID(937) field, and then scan for x'0E' to see if there are any Chinese characters.

But ... following the Never Lose Data principle, I think it would be a zillion times better to save the UTF-8 data as is and not try to convert it to CCSID 37, and especially not blank the field if it has Chinese data.

Just guessing, but assuming you do need this in CCSID 37, wouldn't it be better to just replace the unconvertable characters with say '?' than to blank out the entire field?