CPIA083 on CPYFRMIMPF with RcdDlm(*all) (was: SFTP client via PASE, large files possibly timing out) -- MIDRANGE-L

On 14-Nov-2015 06:35 -0600, Steinmetz, Paul wrote:

On 13-Nov-2015 13:01 -0600, Steinmetz, Paul wrote:

<<SNIP>> the code used to copy output is truncating, only 16
records copied, which I believe is now giving I believe a false
error.

sftp> lcd /BRC/TOA
sftp> put BRCSAI_1105_1310 to_toa/inventory_upload/BRCSAI_1105_1310
Uploading BRCSAI_1105_1310 to /to_toa/inventory_upload/BRCSAI_1105_1310

BRCSAI_1105_1310 0% 0 0.0KB/s --:-- ETA
BRCSAI_1105_1310 1% 1504KB 1.5MB/s 01:13 ETA
BRCSAI_1105_1310 2% 3232KB 1.5MB/s 01:10 ETA
BRCSAI_1105_1310 4% 4896KB 1.5MB/s 01:09 ETA
BRCSAI_1105_1310 5% 6528KB 1.5MB/s 01:07 ETA
BRCSAI_1105_1310 7% 8064KB 1.5MB/s 01:06 ETA
BRCSAI_1105_1310 8% 9536KB 1.5MB/s 01:06 ETA
BRCSAI_1105_1310 9% 11MB 1.5MB/s 01:06 ETA
BRCSAI_1105_1310 11% 12MB 1.5MB/s 01:04 ETA
BRCSAI_1105_1310 12% 14MB 1.5MB/s 01:03 ETA
BRCSAI_1105_1310 14% 15MB 1.5MB/s 01:02 ETA
BRCSAI_1105_1310 15% 17MB 1.5MB/s 01:01 ETA
BRCSAI_1105_1310 16% 18MB 1.5MB/s 00:59 ETA
BRCSAI_1105_1310 18% 20MB 1.5MB/s 00:58 ETA
BRCSAI_1105_1310 19% 22MB 1.5MB/s 00:57 ETA
<<SNIPped many records like those above>>
BRCSAI_1105_1310 99% 108MB 1.5MB/s 00:00 ETA
BRCSAI_1105_1310 100% 109MB 1.5MB/s 01:12
sftp> 2015-11-13,13:42:56

*NONE Command 11/13/15 13:40:09.973076
QCADRV QSYS 0463 TOACLP03T BRCPGMSN 000E
Message: 800 - CRTPF FILE(QTEMP/SYFPHY01) RCDLEN(1024)
CPC7301 Completion 00 11/13/15 13:40:10.004855
QDDCPF QSYS 056D TOACLP03T BRCPGMSN 000E
Message: File SYFPHY01 created in library QTEMP.
CPC7305 Completion 00 11/13/15 13:40:10.030364
QDDCPFM QSYS 005B TOACLP03T BRCPGMSN 000E
Message: Member SYFPHY01 added to file SYFPHY01 in QTEMP.
*NONE Command 11/13/15 13:40:10.030481
QCLCLCPR QSYS 04CF TOACLP03T BRCPGMSN 0015
Message: 1100 - CALL PGM(QP2SHELL)
/* The CALL command contains parameters */
*NONE Command 11/13/15 13:42:57.835090
QCADRV QSYS 0463 TOACLP03T BRCPGMSN 001A
Message: 1900 - CPYFRMIMPF
FROMSTMF('/home/SFTPXFER/TOA/toageneric.output')
TOFILE(QTEMP/SYFPHY01) MBROPT(*REPLACE) TOCCSID(37)
RCDDLM(*ALL) FLDDLM('|') RPLNULLVAL(*FLDDFT)
CPC2206 Completion 00 11/13/15 13:42:57.861964
QSYCHONR QSYS 0695 QLIINSRT QSYS 051B
Message: Ownership of object QCPIMTEMPS in QTEMP type *USRSPC
changed.
CPC2206 Completion 00 11/13/15 13:42:57.874708
QSYCHONR QSYS 0695 QLIINSRT QSYS 051B
Message: Ownership of object QACPTEMP01 in QTEMP type *USRSPC
changed.
CPC2206 Completion 00 11/13/15 13:42:57.896532
QSYCHONR QSYS 0695 QLIINSRT QSYS 051B
Message: Ownership of object QACPTEMP01 in QTEMP type *USRSPC
changed.
CPC2206 Completion 00 11/13/15 13:42:57.912704
QSYCHONR QSYS 0695 QLIINSRT QSYS 051B
Message: Ownership of object QCFT648728 in QTEMP type *USRSPC
changed.
CPC2206 Completion 00 11/13/15 13:42:57.980743
QSYCHONR QSYS 0695 QLIINSRT QSYS 051B
Message: Ownership of object Q7F5FF02 in QTEMP type *FILE changed.
CPC3101 Completion 00 11/13/15 13:42:57.982990
QDBCLRPF QSYS 0241 QC2SYS QSYS *STMT
To module . . : QC2SYS
To procedure : system
Statement . . : 13
Message: Member SYFPHY01 file SYFPHY01 in QTEMP cleared.
CPIA083 Diagnostic 10 11/13/15 13:42:58.034439
QCPIMPRT QSYS *STMT QDBCTHREAD QSYS *STMT
From module . : QCPIMPRT
From procedure . .: Send_type_msg
Statement . . : 24
To module . . : QDBCTHTWRK
To procedure : start__17qdbcth_ThreadWorkFP17qdbcth_ThreadPool
Statement . . : 8
Message: Stream file copied to object with truncated records.
CPC2959 Completion 00 11/13/15 13:42:58.053774
QCPIMPRT QSYS *STMT TOACLP03T BRCPGMSN 001A
From module . : QCPIMPRT
From procedure . .: Send_type_msg
Statement . . : 24
Message: 16 records copied to member SYFPHY01.

I've identified the issue with the CPYFRMIMPF.
The output file contains More than one End of Line character found.
On the CPYFRMIMPF, I was using a Record delimiter of *all.
I changed record delimiter from *all to *CR.
This eliminated the truncation error, CPIA083, Stream file copied to
object with truncated records.
I now have a valid output file which is used for confirm if the SFTP
was successful.
I look for 100% in the output file.

I'm still questioning why *ALL is not working properly.

The chosen implementation for the special value of *ALL for the Record delimiter (RCDDLM) parameter of the CPYFRMIMPF feature was to use the "First occurrence of any single or double character combination of carriage-return and line-feed" as the consistent EOR. That contrasts with the implementation for the special value of *ALL for the End Of Line Characters (ENDLINFMT) parameter of the CPYFRMSTMF feature. That would seem a likely suspect, but difficult to know for sure why, without knowing more about the environment and an example of actual data that can reproduce the error; e.g. the CCSID of the FROMSTMF, if the override [via a DtaAra] is in effect to force the import to use the v5r2 code-path, any changed command defaults, et al.

Seems possibly that /using the wrong tool for the job/ might describe the issue; i.e. if the stream record-data shown [with some blanks compressed] in the earlier quoted message is what is being copied. The DBF import request suggests that pipe-delimited data is being imported from the stream data into a described database file; there is of course, nothing like that in that stream of data. And if that is the data being copied, then seemingly the desire is instead, merely to do a simple record-copy with translation of the text stream record-data into database fixed-length record flat file record data, for which Copy From Stream File (CPYFRMSTMF) is the expected choice. Essentially, why is not the following request being made, instead?:

CPYFRMSMTF
FROMSTMF('/home/SFTPXFER/TOA/toageneric.output')
TOMBR('/qsys.lib/qtemp.lib/SYFPHY01.file/SYFPHY01.mbr')
MBROPT(*REPLACE) TABEXPN(*NO) ENDLINFMT(*ALL)
CVTDTA(*AUTO) STMFCCSID(*STMF) DBFCCSID(37)