Re: Making the most, or over-thinking it - SQL for uploads

Hi Duane,

FWIW...

I do a lot of PC to DB2 on i data transfers due to the nature of my work (data migration and transformation).

Here is how I handle them:
1) I convert the PC data to bar delimited, since a bar character is rarely found inside data.
2) I transfer the data to DB2 on i into a source file using IBM's ACS product without converting it other than a basic conversion to EBCDIC. The data is uploaded as a single bar delimited text column. I assign a generous record length longer than likely to exist in the source data.
3) I then use a self-created SQL parsing function inside a CREATE TABLE AS ( SELECT ... ) SQL code pattern designed for this purpose to split the bar delimited data into columnar format. I modify this code as needed to apply any custom data transformation required.

By uploading the data as a single delimited column, it makes it very easy to get the data up to the IBM i system. If you try to apply the data conversion during the upload into separate columns on the host, it is a much more problematic process in my experience (especially if the source data is a spreadsheet with inconsistent formatting of the data).

By waiting to perform the primary data transformation until you have the data on the host, I think the process is much more efficient (assuming you have good data transformation skills).

I get these upload / transformation tasks completed very quickly, efficiently, and consistently using this process. If the data format is inconsistent, no matter how you slice it, you have to deal with the inconsistency. I prefer to deal with data quality issues up on the host, instead of during the transfer to the host, because I've created a large library of host based SQL transformation functions.

Hope this helps,

Mike

-----Original Message-----
From: Duane Scott [mailto:dscott@xxxxxxxxxxx]
Sent: Tuesday, August 30, 2016 8:16 AM
To: WDSCI-L@xxxxxxxxxxxx
Subject: [WDSCI-L] Making the most, or over-thinking it - SQL for uploads


I had the awesome idea to use SQL in RDi to import data into a DB2 file from a text file on a network drive.

I'm mostly familiar with SQL but not so much in the RDi environment. (Getting there is all the fun.)

I can access my DB2 files on the IBM I and display data, but haven't figured out the set up to access text files on a PC (using SQL).

I need to be able to access only three areas of the text file, one for the WHERE clause, and the other two for the data to INSERT.

I'm relatively certain that it's in the Database Connections setup, but need a little help in figuring out the wizard.

Any ideas on an existing QAD primer? Especially one on the more recent versions of RDi (9.5.0.3)

And even more for the PC-disabled. I really need things spelled out for my simple mind.

Duane

--
________________________________
________________________________
CONFIDENTIALITY NOTICE: This electronic message transmission is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. If you have received this transmission, but are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this information is strictly prohibited. If you have received this e-mail in error, please contact NALC Health Benefit Plan at 703-729-4677 and delete and destroy the original message and all copies.