Re: CPYFRMIMPF and Unicode - UTF-16 in particular

[Scott Klement <midrange-l@xxxxxxxxxxxxxxxx>]

Hmmm...  on which version of Notepad is UTF-16 the default option?!  On 
my system (Windows 7) ANSI comes up as the default.   Seems to me that a 
more sensible default (more sensible than either ANSI or UTF-16) is 
UTF-8.  After all, that's what the Web uses for everything, so if you 
wanted to type up an HTML doc, XML, JSON, or just a plain text document, 
it seems to me that UTF-8 is the most sensible choice.

And UTF-8 has CCSID 1208, so it'd be easy to handle with CPYFRMIMPF as well.

To handle UTF-16 LE on IBM i, you'd probably have to write a program. 
IBM has an API called "Transform UCS data" that can transform UTF-16 LE 
to a more friendly option (such as UTF-16 big endian, which is ccsid 
1200) but you'd have to write a program to transform it before using 
CPYFRMIMPF -- not a great option, IMHO.

Seems to me it's a better option to tell your client to save as UTF-8.


On 2/27/2013 3:50 PM, Vernon Hamberg wrote:
> Hi
>
> We are on 7.1, I don't know the PTF level. I'm looking at importing
> tab-delimited data from text files that are in little-endian UTF-16
> (called Unicode in Notepad Save As). That is what Windows uses by
> default. Yes, there is an option in Notepad to save as big-endian UTF-16
> (called Unicode big-endian), and I may be recommending that. Or just use
> ANSI. But mistakes do happen, and it'd be nice to have the program
> handle things.
>
> I also tried saving the text file with TextPad, which also has the 2
> UTF-16 "endian" options. There is a difference in how Notepad and
> TextPad do it - the former puts in the BOM at the start - the Byte Order
> Marker, which is x'FEFF' for big-endian (normal order, the one IBM i
> uses) and x'FFFE' for little-endian (bytes are in reverse order).
>
> In any case, we want to be able to process Unicode text files. So I have
> 4 Unicode text files, tab-delimited.
>
> *** It seems that CPYFRMIMPF converts individual bytes, not characters,
> as I'll show below. ***
>
> CPY will do the conversion correctly for the big-endian ones, but for
> little-endian, the result is an empty stream file.
>
> So double-clutch using CPY, then CPYFRMIMPF from the copy works.
>
> 2 questions:
> 1. Can CPYFRMIMPF actually do this correctly? you see the parameters I
> tried.
> 2. Can IBM i handle UTF-16LE (little-endian)? Is there a CCSID for that,
> like 1200 is for UTF-16BE (big-endian).
>
> Thanks
> Vern
>
> uc-big-endian-notepad.txt      (x'FEFF' at start of file)
> uc-little-endian-notepad.txt      (x'FFFE' at start of file)
> uc-big-endian-textpad.txt      (nothing at start of file)
> uc-little-endian-textpad.txt      (nothing at start of file)
>
> The CPYFRMIMPF command I'm testing looks like this - sample result
> follows the first one, others are similar - a blank in clear text is
> actually x'00' - and note that x'FEFF' is converted to x'8EDF'.
>
> CPYFRMIMPF FROMSTMF('uc-big-endian-notepad.txt') TOFILE(VERN/FLAT3000)
> MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')
> þÿ b r a n d s t o r e _
> 8D0809080908000A0A09090806
> EF02090105040502030609050D
>
> CPYFRMIMPF FROMSTMF('uc-little-endian-notepad.txt')
> TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
> RCDDLM(*ALL) FLDDLM('~')
>
> CPYFRMIMPF FROMSTMF('uc-big-endian-textpad.txt') TOFILE(VERN/FLAT3000)
> MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')
>
> CPYFRMIMPF FROMSTMF('uc-little-endian-textpad.txt')
> TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
> RCDDLM(*ALL) FLDDLM('~')