Re: File Reads -- MIDRANGE-L

Venu,

IMHO, the compile OPTIMIZE option is not where you would realize signifigant
gains.  It would be in how the problem is solved.
Using blocking factors may help, but without the details of your particular
situation, it's hard to comment beyond generalizations.

For our A/R A/P applications we are faced with having multiple records per 
invoice
and in this case expanding blocking for a indexed logical would hurt more than
help.

Blocking is great for sequential processing but not necessarily for indexed.  
Now
this depends on how organized the physical file is.  The more organized, the 
better
odds are that the next indexed record is within block( cache).  If you do not
RGZPFM often, the less the odds are that the next logical record is within the
block and you would be wasting memory and read cycles.  Not to mention that if 
you
block too high, you would face a "rotational delay".  I take it that you have
already taken that into account since you want the blocking factor to be a 
variable
based upon machine model and record length. (good move)

We managed to improve 1M+ record batch process time two ways:

1) If the main file is read to update/write to a subordinate file, only do the
update if some data changed.  This reduces the number of writes.  For example:  
our
customer receivable subsidiary ledger contains entries for the invoice issued 
and
the payment(s)/adjustment(s)  made, so there can be multiple records per 
invoice.
We have a subordinate "net" file (which only has one entry per invoice) that has
the gross amount, net amount, number of days (till due/past due/till paid in
full).  We know the difference by: if net amount = 0 Positive/negative days = 
days
in relation to due date that invoice was paid,  and if net amount <> 0
Positive/negative is either within terms (negative=number of days till due) or 
past
due (positive=number of days delinquent).  If we blast through the file, if the
invoice was paid in full last week/last month/etc. no value changed so we avoid 
the
update.  BTW, over 90% of the entries have been paid in full and do not have a 
data
value change so we avoid 90% of the UPDAT operations.

Now the above example assumes that the data base has been divided into
static/variable (time sensative) physical files.  If an invoice can have 
multiple
entries (the primary file) and you want to know the age of an invoice, the 
ageing
value should not be within the primary file.  The static information is within 
the
primary file since it is recorded as it occured and the variable information is 
the
invoices condition at a point in time and is subject to conditional (what day is
today) change.

2) If the file being blasted through needs to do a CHAIN to another file, 
writing a
prefetch program will get the CHAINed entry into cache.  Here's how that works: 
you
have your main program and a "prefetch" program started at the same time in
different memory pools. You use a data queue to sync the two programs.  The main
program starts reading FILEA with a level break that does a CHAIN operation.  
The
"prefetch" program also starts reading FILEA and tosses out the first level 
break
and does the chain on the next break.  What happens is that when the main 
program
gets it's second level break, the record it wants is already in cache and you 
have
synched the disk I/O.  The main program and "prefetch" program keep in time by 
the
data queue in that the main program sends a record at each level break and the
"prefetch" program goes into a pseudo wait state until it receives from the data
queue to process the next level break CHAIN.  You may be tempted to just pass 
the
key of the next CHAIN, but remember since the "prefetch" program is reading the
same file (FILEA) by the same path (one level break ahead of the main program) 
the
detail records are also cached. Provided that you have enough memory!  One last
condition:  the main program does enough calcs between level breaks so that the
"prefetch" program actually stays ahead of the main program for disk I/O.  IBM
wrote a "tech tip" on this a few years back and stated a substantial reduction 
in
processing time under the right conditions. If memory serves me: 6 hours 
reduced to
2.

Well that's my two cents worth.

HTH
James W. Kilgore
qappdsn@ibm.net



> Hi All,
>
> Recently we wrote a DateAging application. The application uses multi
> threading. Each data file in the user library is aged simultaneously as
> independeng aging jobs. We used this to enhancethe performance. Earlier 
>version
> was only single thread where the application used to age one datafile at a
> time. Although we gained a lot of improvement with the new design, we still 
>are
> facing some difficulties particularly with files of bigger size.
>
> We have this application written in ILE and we are having different modules to
> do different tasks, we have service pgms, binding directories...all that 
>stuff.
> We have compiles the main program as
>
> CRTBNDRPG with OPTIMIZE(*BASIC).
>
> We are still facing some performance problems. A data file of 1 million 
>records
> is taking more than 15 hrs to complete the job. Is there anything to do with
> this OPTIMIZE(*BASIC). I read in the ILE manuals that there will be 
>significant
> improvements if we change this to OPTIMIZE(*FULL). But I dont know what
> improvements it is referring. The majority of time is wasted in the I/O of the
> datafile. How is this OPTIMIZE parameter going to influence the data file 
>I/O??
> The manual says that there will be significant improvements with
> OPTIMIZE(*FULL) but there will be some problems also. What are these
> difficulties??
>
> We are also thinking of chaning the record blocking factor for the data file.
> That is, before calling our RPG program, we will do a OVRDBF of the datafile
> with NBRRCDS as say 10000. Will there be any improvements and what are the
> possible problems we will face if we change the NBRRCDS parameter?
>
> Any ideas or thoughts,. Thanx in advance for any help.
>
> Rgds
> Venu
>
> +---
> | This is the Midrange System Mailing List!
> | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> | To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
> | Questions should be directed to the list owner/operator: david@midrange.com
> +---

+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---