Re: CSVR4 and UTF-8

Hi Vern,

As so often Scott is fast like the wind and has already fixed CSVR4.

I think that should help Greg as much as possible. CSVDEMO4 explains well how to do it. One could even filter invalid Unicode characters by a range of hexadecimal values in order to remove or replace them.

Regards,

Thomas.

Am 06.05.2020 um 00:24 schrieb Vernon Hamberg:

You're right, Thomas - I found SQL's XMLTABLE (and probably the same for the equivalent JSON UDTF) to yield the best (most reliable?) results - trying to replace a non-printable UTF-8 character would be nice, maybe, and I hope my colleague has some luck with that. As I see, it you'd need a table of the UTF-8 glyphs that are not in your version of EBCDIC, then provide an alternative. I hope that is not a fool's errand!

One alternative is to put everything in your tables into UTF-8 or UTF-16.

Vern

On 5/5/2020 1:15 PM, Tools/400 wrote:

I do not think that iconv() could help, because there are no counterparts in EBCDIC for a large number of characters that are available in UTF-8.

So all these missing characters would end up in x'3F', right?

Thomas.

Am 05.05.2020 um 19:58 schrieb Vernon Hamberg:

I think probably, Sam - I suspect that the i is using that in WRKLNK, option 5 (also known as DSPF). And probably equally the SQL engine does.

Unless we can specify that the XML-SAX and XML-INTO opcodes can have the CCSID set to 1208 - you CAN have it set to UCS2, but that's not enough to prevent errors when the UTF-8 character is not in the UCS2 character set - emojis certainly are in this category, things like ellipsis and em dash get set to that X'3F' again.

But it's too late to use iconv, I am guessing, when you use those opcodes.

Maybe for consuming XML (not using SQL's capabilities), the DATA-INTO can be used and then something that uses iconv?

A very challenging task, we have, don't we?

Vern

On 5/5/2020 12:38 PM, Sam_L wrote:

I admit my lack of experience or knowledge on this, but would the iconv()--Code Conversion API be of any use?

Sam


On 5/5/2020 12:10 PM, Vernon Hamberg wrote:

Oh yeah - and I have a presentation on this problem in the iNSIGHT virtual COMMON conference that started today - I believe everything will be available for 60 days, one can still register for it. Have to be a COMMON member, probably.

Vern

On 5/5/2020 9:17 AM, Greg Wilburn wrote:

So I changed the attributes of the file from 819 to 1208.  It displays better in WRKLNK.  The Right Single Quote is displayed as a blank instead of the other characters.

I'm wondering if this will help (at least some).