RE: Import taking over 24 hours

Part of the reason is that is the only way to really get the raw data, since 
we don't have a layout of the file we are not positive of where each field 
begins or ends.....the ftp routine is only taking about 10% of the total 
time to complete, the rest seems to be in parsing down the data.


On Mon, 9 May 2005 13:18:51 -0700, Alan Campin wrote
> Why are you dumping the files to MS Access. You are running an 
> RS/6000. Why not just FTP them directly to the AS/400 or to the IFS?
> 
> Also, you did not break down the various times. How much time for 
> each phase of the project? Where are you spending your time?
> 
> -----Original Message-----
> From: dpalme@xxxxxxxxxxx [mailto:dpalme@xxxxxxxxxxx]
> Sent: Monday, May 09, 2005 6:44 AM
> To: RPG Group
> Subject: Import taking over 24 hours
> 
> We have been working on an import situation for a couple of months 
> now, and I would like to get some feedback from some of you on this. 
>  We have a RS/6000 which is running some older cobol programs, we 
> are not in a position to replace those programs at this time, what 
> we need to do is import that data (from standard unix cobol files 
> ACUCOBOL to be exact) into several physcial files on our 9406-170.
> 
> We are using MS Access (which I hate but it serves its purpose here) 
> to dump the data from the cobol files into tab delimited files which 
> are then ftp'd over to the 400 and then imported into the physical files.
> 
> Everything does work but it is taking around 28 hours for the import 
> process to complete.
> 
> The files can be broken into three groups, customer master data -
>  which takes about 20 minutes to complete....the flat file is 
> approximately 8.5mb in size.  We have several line item files which 
> are imported into one single physical file, the flat files for this 
> group are approximately 400mb.  The third and final file is a header 
> file which is approx 150mb in size.
> 
> We had an outside vendor actually write the code for us which as I 
> said before does work, but it is taking a very long time to process.
> 
> If anyone has any ideas or thoughts on how we can speed this process 
> up I would love to hear those ideas.  I need to check with the legal 
> department before posting the code since I am not sure what 
> contractual obligations we are under as far as the code goes.
> 
> I do know that he is splitting the fields up by checking for the tab 
> and then writing the data to the file. the header file has quite a 
> few fields in it, and if anyone wants to know the number of fields I 
> can verify that and re-post.
> 
> -- 
> This is the RPG programming on the AS400 / iSeries (RPG400-L)
>  mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To 
> subscribe, unsubscribe, or change list options, visit: 
> http://lists.midrange.com/mailman/listinfo/rpg400-l or email: RPG400-
> L-request@xxxxxxxxxxxx Before posting, please take a moment to 
> review the archives at http://archive.midrange.com/rpg400-l.


--
Open WebMail Project (http://openwebmail.org)