RE: Speed in Reading

Kurt,
While there are undoubtedly horses for courses I have been tasked with data mining tasks over the last year where there are 10 million or more records driving the process each of which itself may generate scores of other reads. I've found that submitting up to simultaneous 10 jobs, each of which accepts parameters as to which portion of the input file drives it, has yielded exceptional performance. This does drive CPU right up but permits huge volumes to be processed overnight.

Peter

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Kurt Anderson
Sent: Saturday, 5 June 2010 6:13 a.m.
To: 'RPG programming on the IBM i / System i'
Subject: Speed in Reading

Hi all,

Lately I've been dealing with a new client that has a high volume of data for us to process, so I've been looking at various ways to get things moving a little faster.

I believe I've successfully implemented the use of SETOBJACC with the help of Chuck Pence & Mark Waterbury (thanks guys).

Now, this may be grasping at straws, but in a high data volume situation, should I see any benefit in restricting the fields in a LF to only the fields used by the program? (In essence, going from record length of 438 to 138.) My brain says, "yes," but the logic under-the-covers may not match the logic my brain uses to come to its conclusion.

I'm also curious about RPG blocking. This question mainly goes out to people using file encapsulation service programs, but is obviously open to anyone to answer. I have a file wrapped up in a service program. This service program pretty much handles all I/O operations for the file. Now, if all I want to do is loop through the file, it's not going to block read because it has operations such as READE in the service program. I wish block-reading would be determined at run-time (again, a wish without a full understanding of the compiler does its thing). I'm curious how people have circumvented this limitation?

Thanks,
Kurt Anderson
Sr. Programmer/Analyst
CustomCall Data Systems