Re: CPYFRMIMPF and Unicode - UTF-16 in particular

Hmmm... on which version of Notepad is UTF-16 the default option?! On my system (Windows 7) ANSI comes up as the default. Seems to me that a more sensible default (more sensible than either ANSI or UTF-16) is UTF-8. After all, that's what the Web uses for everything, so if you wanted to type up an HTML doc, XML, JSON, or just a plain text document, it seems to me that UTF-8 is the most sensible choice.

And UTF-8 has CCSID 1208, so it'd be easy to handle with CPYFRMIMPF as well.

To handle UTF-16 LE on IBM i, you'd probably have to write a program. IBM has an API called "Transform UCS data" that can transform UTF-16 LE to a more friendly option (such as UTF-16 big endian, which is ccsid 1200) but you'd have to write a program to transform it before using CPYFRMIMPF -- not a great option, IMHO.

Seems to me it's a better option to tell your client to save as UTF-8.


On 2/27/2013 3:50 PM, Vernon Hamberg wrote:

Hi

We are on 7.1, I don't know the PTF level. I'm looking at importing
tab-delimited data from text files that are in little-endian UTF-16
(called Unicode in Notepad Save As). That is what Windows uses by
default. Yes, there is an option in Notepad to save as big-endian UTF-16
(called Unicode big-endian), and I may be recommending that. Or just use
ANSI. But mistakes do happen, and it'd be nice to have the program
handle things.

I also tried saving the text file with TextPad, which also has the 2
UTF-16 "endian" options. There is a difference in how Notepad and
TextPad do it - the former puts in the BOM at the start - the Byte Order
Marker, which is x'FEFF' for big-endian (normal order, the one IBM i
uses) and x'FFFE' for little-endian (bytes are in reverse order).

In any case, we want to be able to process Unicode text files. So I have
4 Unicode text files, tab-delimited.

*** It seems that CPYFRMIMPF converts individual bytes, not characters,
as I'll show below. ***

CPY will do the conversion correctly for the big-endian ones, but for
little-endian, the result is an empty stream file.

So double-clutch using CPY, then CPYFRMIMPF from the copy works.

2 questions:
1. Can CPYFRMIMPF actually do this correctly? you see the parameters I
tried.
2. Can IBM i handle UTF-16LE (little-endian)? Is there a CCSID for that,
like 1200 is for UTF-16BE (big-endian).

Thanks
Vern

uc-big-endian-notepad.txt (x'FEFF' at start of file)
uc-little-endian-notepad.txt (x'FFFE' at start of file)
uc-big-endian-textpad.txt (nothing at start of file)
uc-little-endian-textpad.txt (nothing at start of file)

The CPYFRMIMPF command I'm testing looks like this - sample result
follows the first one, others are similar - a blank in clear text is
actually x'00' - and note that x'FEFF' is converted to x'8EDF'.

CPYFRMIMPF FROMSTMF('uc-big-endian-notepad.txt') TOFILE(VERN/FLAT3000)
MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')
þÿ b r a n d s t o r e _
8D0809080908000A0A09090806
EF02090105040502030609050D

CPYFRMIMPF FROMSTMF('uc-little-endian-notepad.txt')
TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
RCDDLM(*ALL) FLDDLM('~')

CPYFRMIMPF FROMSTMF('uc-big-endian-textpad.txt') TOFILE(VERN/FLAT3000)
MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')

CPYFRMIMPF FROMSTMF('uc-little-endian-textpad.txt')
TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
RCDDLM(*ALL) FLDDLM('~')