Re: CPYFRMIMPF and Unicode - UTF-16 in particular

How kludgy.

If you're going to write all of these workarounds (removing x'00', removing empty lines, etc), why not make it detect the encoding correctly, and convert it correctly?

What will you do if someone uses a character in 1252 that doesn't have the same codepoint in unicode?

What will you do if (gasp) the user uses a Unicode character that doesn't exist in ASCII? Your approach will result in multiple characters appearing the resulting EBCDIC document.


On 3/4/2013 11:07 AM, Vernon Hamberg wrote:

I believe I've an answer that ends up being very flexible. CPYFRMSTMF.
The following command results in rows of tab-delimited values in a flat
file -

CPYFRMSTMF
FROMSTMF('some-encoded-stream-file.txt')
TOMBR('/QSYS.LIB/QTEMP.LIB/FLATFILE.FILE/FLATFILE.MBR')
MBROPT(*REPLACE)
CVTDTA(*AUTO)
STMFCCSID(*STMF)
DBFCCSID(*FILE)
ENDLINFMT(*ALL)
TABEXPN(*NO)

This was preceded by a CRTPF QTEMP/FLATFILE RCDLEN(3000)

With UTF-16, there is an extra row interleaved - because there is a
Unicode CRLF, and the conversion sees the CR and the LF as separate. No
problem - this is easy to clean up!

BOM's can be removed nicely, if present, with and SQL UPDATE. The
interleaved rows can also easily be removed. And the nulls (UTF-16 only)
can be cleaned up with an SQL REPLACE function. This will all work fine
with RUNSQLSTM. Or successive RUNSQL, since we are on 7.1 - once I know
we have the requisite group PTF. Well, we do - the command is there!

Any cautions are much appreciated - still, this does look pretty cool -
no transform needed, similar effect to how we were using CPYFRMIMPF for
ANSI-encoded stream files.

Regards
Vern

On 3/3/2013 10:52 PM, CRPence wrote:

On 03 Mar 2013 15:58, Vernon Hamberg wrote:

I'm considering using CPY - not CPYFRMIMPF - to copy the STMF
into a user space - haven't succeeded so far, maybe due to the
things you describe here.

As I had noted, the stream-file functions try really hard to not let
you get non-EBCDIC data into the user space. Most invocations will fail
with a complaint about the CCSID of the target "file"; i.e. the target
as *USRSPC can not have a CCSID set "to match the CCSID of the source
file" [source as in source\target, not source PF] according to CPFA098.

-snip-

And if you could convince somebody to change the FROMCCSID to apply
to the FROMSTMF, then you could import directly from the user space
after the data has been transformed to UTF-16BE. But of course a limit
of 16MB would be rather limiting :-(

I believe I actually ran into that limit - I got a different message,
CPFA0A2, Information passed to this operation was not valid. And the
sample I was given was around 20 MB. So tricking CPY didn't work - it
did with a shorter stream file.

On v5r3 I was able to trick\convince the CPY feature into copying the
ASCII data into the User Space using the following script:

chgjob ccsid(*hex)
cpy obj(utf16) toobj('/qsys.lib/mylib.lib/space.usrspc')
fromccsid(*JOBCCSID) toccsid(*JOBCCSID) dtafmt(*binary) replace(*YES)
/* dtafmt(*text) seemed to effect the same */

Personally I can not understand why, if the format is *binary* why
the feature even cares about the CCSID of either source or target.? I
wonder if the effect may be an /accident/ versus the intended function
of that CPY invocation.?