RE: System slow-down - disk usage?

Kurt,

Hmm, can you get exclusive update access to the previous and
input files?

It sounds to me like a case for matching records <grin />.

In particular, how long does it take to do
CPYF PREVIOUS QTEMP/DISCARDME CRTFILE(*YES)
with full-sized PREVIOUS? That is a lot more work than reading
the whole file in key sequence, and for mere tens of millions of
records, I suspect it would be a lot faster than the nineteen
hours you mention in your first posting. Of course, you would
have to arrange to process the new records in the same sequence.

Of course, this gives a run time at least proportional to the
size of PREVIOUS, and at some size that will exceed the time for
a direct-access method. Do you know how big PREVIOUS might become?

This simple-minded scheme does not give a direct way to exploit
more processors or more memory, something which comes for free
if you use SQL to select the incoming transactions. Darn, I have
been trying so hard to find an excuse to use matching records.

HTH,
Terry.


On Tue, 2010-06-15 at 13:51 -0500, Kurt Anderson wrote:

A couple answers at once:

The previous file doesn't always start at 0 records. It contains
months worth of data. A previously processed record may be from the
same input file, or may not.

The previous file only contains the key fields to determine a
duplicate.

True, the majority of the time a duplicate is an exception to the
rule. However we do have clients that send us 40% duplicates (yes, I
know, sigh). If I were to do the Write and catch the Duplicate error
I'd have to also remove the job log entry (maybe not "have" to, but
100,000 job log messages about a duplicate write would be annoying).
This is where I wish Alison Butterill would recognize the need to
suppress a job log message when monitoring for an error. Sure, I can
do it with an API, I know (and have).

The Previous file, in this case, is gb's in size. We don't currently
have enough Main storage to accommodate, much less to allow multiple
jobs (using different files) to process at once. We are looking at
getting more memory.

Select Distinct may be a good idea for eliminating duplicates within a
single input file. I'll definitely consider that.

We are blocking the read on the input file. However, the size of the
Previous file is really the issue here. Empty the Previous file, and
the program flies. Fill up the Previous file, and the program grinds.

The Input & Previous file will never have deleted records in them.

Thanks for all of the responses,
Kurt