Re: CPYFRMIMPF and Unicode - UTF-16 in particular

[Vernon Hamberg <vhamberg@xxxxxxxxxxxxxxx>]

Steve

My tests show that CPYFRMIMPF doesn't even correctly process UTF-16 BE 
with/without BOM.

It seems to do a byte-to-byte conversion, not a character-to-character.

Looks like a couple requirements here!

Vern

On 2/27/2013 5:57 PM, TheBorg wrote:
> Bruce -
>
> Perhaps I was a bit vague in my response...
>
> I am aware of QlgTransformUCSData, but what I meant by 'directly handle' did
> not include the necessity of writing programs which utilize an API to
> convert data from/to Unicode (like UTF-16 LE or UTF-16 LE with BOM).  It
> seems to me that IBM could have built this support into the CPYFRMIMPF and
> CPYTOIMPF commands to make it easier for me to send/receive UTF-16 data to
> my customer (a BI group), who prefers to get Unicode UTF-16 LE).
>
> - sjl
>
> "Bruce Vining"  wrote in message
> news:mailman.17117.1362008341.10847.midrange-l@xxxxxxxxxxxx...
>
> The Transform UCS Data API, documented at
> http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qlgtrans.htm,
> supports UTF-16 LE. The API has been available since V4R5.
>
> There is not a specific CCSID for it, but you can certainly
> convert/transform the data to a format that does have a CCSID.
>
> Bruce Vining
>
> On Wed, Feb 27, 2013 at 4:51 PM, TheBorg <sjl_abc@xxxxxxxxxxx> wrote:
>
> > Vern -
> > >From my Unicode research, IBM I cannot directly handle UTF-16 LE...
> > -sjl
> >
> >
> > "Vernon Hamberg"  wrote in message
> > news:mailman.17096.1362001857.10847.midrange-l@xxxxxxxxxxxx...
> >
> > Hi
> >
> > We are on 7.1, I don't know the PTF level. I'm looking at importing
> > tab-delimited data from text files that are in little-endian UTF-16
> > (called Unicode in Notepad Save As). That is what Windows uses by
> > default. Yes, there is an option in Notepad to save as big-endian UTF-16
> > (called Unicode big-endian), and I may be recommending that. Or just use
> > ANSI. But mistakes do happen, and it'd be nice to have the program
> > handle things.
> >
> > I also tried saving the text file with TextPad, which also has the 2
> > UTF-16 "endian" options. There is a difference in how Notepad and
> > TextPad do it - the former puts in the BOM at the start - the Byte Order
> > Marker, which is x'FEFF' for big-endian (normal order, the one IBM i
> > uses) and x'FFFE' for little-endian (bytes are in reverse order).
> >
> > In any case, we want to be able to process Unicode text files. So I have
> > 4 Unicode text files, tab-delimited.
> >
> > *** It seems that CPYFRMIMPF converts individual bytes, not characters,
> > as I'll show below. ***
> >
> > CPY will do the conversion correctly for the big-endian ones, but for
> > little-endian, the result is an empty stream file.
> >
> > So double-clutch using CPY, then CPYFRMIMPF from the copy works.
> >
> > 2 questions:
> > 1. Can CPYFRMIMPF actually do this correctly? you see the parameters I
> > tried.
> > 2. Can IBM i handle UTF-16LE (little-endian)? Is there a CCSID for that,
> > like 1200 is for UTF-16BE (big-endian).
> >
> > Thanks
> > Vern
> >
> > uc-big-endian-notepad.txt      (x'FEFF' at start of file)
> > uc-little-endian-notepad.txt      (x'FFFE' at start of file)
> > uc-big-endian-textpad.txt      (nothing at start of file)
> > uc-little-endian-textpad.txt      (nothing at start of file)
> >
> > The CPYFRMIMPF command I'm testing looks like this - sample result
> > follows the first one, others are similar - a blank in clear text is
> > actually x'00' - and note that x'FEFF' is converted to x'8EDF'.
> >
> > CPYFRMIMPF FROMSTMF('uc-big-endian-notepad.txt') TOFILE(VERN/FLAT3000)
> > MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')
> > þÿ b r a n d  s t o r e _
> > 8D0809080908000A0A09090806
> > EF02090105040502030609050D
> >
> > CPYFRMIMPF FROMSTMF('uc-little-endian-notepad.txt')
> > TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
> > RCDDLM(*ALL) FLDDLM('~')
> >
> > CPYFRMIMPF FROMSTMF('uc-big-endian-textpad.txt') TOFILE(VERN/FLAT3000)
> > MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37) RCDDLM(*ALL) FLDDLM('~')
> >
> > CPYFRMIMPF FROMSTMF('uc-little-endian-textpad.txt')
> > TOFILE(VERN/FLAT3000) MBROPT(*REPLACE) FROMCCSID(1200) TOCCSID(37)
> > RCDDLM(*ALL) FLDDLM('~')
> >
> > --
> > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> > list
> > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > To subscribe, unsubscribe, or change list options,
> > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > Before posting, please take a moment to review the archives
> > at http://archive.midrange.com/midrange-l.