RE: Import taking over 24 hours

There's been some good ideas...

Here's a couple more...

1) stop journaling

2) Parallelism (particularly if you have more than 1 CPU), if using RPG submit 
X number of jobs that process 1/X of the rows, idea being you want CPU usage to 
be a close to 100% as possible.  If using CPYFRMIMPF, make sure that 
DEGREE(*NBRTASKS) is set with CHGQRYA and that the DB2 SMP license program is 
installed.


HTH,

Charles Wilt
iSeries Systems Administrator / Developer
Mitsubishi Electric Automotive America
ph: 513-573-4343
fax: 513-398-1121
 

> -----Original Message-----
> From: rpg400-l-bounces@xxxxxxxxxxxx
> [mailto:rpg400-l-bounces@xxxxxxxxxxxx]On Behalf Of dpalme@xxxxxxxxxxx
> Sent: Monday, May 09, 2005 9:44 AM
> To: RPG Group
> Subject: Import taking over 24 hours
> 
> 
> We have been working on an import situation for a couple of 
> months now, and 
> I would like to get some feedback from some of you on this.  
> We have a 
> RS/6000 which is running some older cobol programs, we are 
> not in a position 
> to replace those programs at this time, what we need to do is 
> import that 
> data (from standard unix cobol files ACUCOBOL to be exact) 
> into several 
> physcial files on our 9406-170.  
> 
> We are using MS Access (which I hate but it serves its 
> purpose here) to dump 
> the data from the cobol files into tab delimited files which 
> are then ftp'd 
> over to the 400 and then imported into the physical files.
> 
> Everything does work but it is taking around 28 hours for the 
> import process 
> to complete.
> 
> The files can be broken into three groups, customer master 
> data - which 
> takes about 20 minutes to complete....the flat file is 
> approximately 8.5mb 
> in size.  We have several line item files which are imported 
> into one single 
> physical file, the flat files for this group are 
> approximately 400mb.  The 
> third and final file is a header file which is approx 150mb in size.
> 
> We had an outside vendor actually write the code for us which 
> as I said 
> before does work, but it is taking a very long time to process.   
> 
> If anyone has any ideas or thoughts on how we can speed this 
> process up I 
> would love to hear those ideas.  I need to check with the 
> legal department 
> before posting the code since I am not sure what contractual 
> obligations we 
> are under as far as the code goes.
> 
> I do know that he is splitting the fields up by checking for 
> the tab and 
> then writing the data to the file. the header file has quite 
> a few fields in 
> it, and if anyone wants to know the number of fields I can 
> verify that and 
> re-post.
> -- 
> This is the RPG programming on the AS400 / iSeries (RPG400-L) 
> mailing list
> To post a message email: RPG400-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
> or email: RPG400-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/rpg400-l.
> 
>