re:Sequential Processing

[Billy Rowe <billyrowe@xxxxxxx>]

Venkat Sreenivasan wrote:
> 
> How do I override the system block/buffer size in sequential processing.  I 
>know
> that I have to define the SEQONLY(*YES) and number of records in OVRDBF.  But
> I'm trying to find the calculation logic to determine the number of records to
> define in OVRDBF.
> 
> Example:  OVRDBF FILE(TESTPF)   SEQONLY(*YES 5000).
> 
> My TESTPF record length is 66 bytes.
> 
> Your help will be very much appreciated in this regard.  Kindly high light on
> the logic in calculation.


Ven,

I have attached an IBM document(text) from Software Knowledge base
entitled:

Document Number: 10064916
Blocking, Sequential Only, and the Effect on a Program.

There is also an article in the May 1998 Midrange Computing magazine
entitled:
Blocking Data for Better Performance that talks about different block
sizes
for the different OS/400 versions(i.e. 32KB prior to V3R1, or 64KB for
V3R1
and after).

The IBM document(as you will see) talks about 32K for CISC 400's and
128K
for RISC 400's.

Hope this helps,

Billy
Carpe Diem
 [Image] Document Feedback

---------------------------------------------------------------------------
IBM AS/400 Support Line Technical Document
__________________________________________________________________
[Image] Document Information

Document Title:
Blocking, Sequential Only, and the Effect on a Program

Document Description:

All high-level language programs (HLLs) use blocking at certain times and
use single record I/O at other times, based on program specifications.
Because blocking takes less system resource to perform a single I/O, a
program that blocks performs better and use less system resource. The
default for the HLL uses record blocking if opening a file for output only
(write) or input only (read). The programmer is expected to override the
file or use larger blocks if the default is not appropriate. The HLLs do
single I/O if files are opened for both input/output or update/delete.
Blocking works well only if one reads many records sequentially because the
block of records read are from some position in the file and the next N
records. If one is going to position randomly within the file (CHAIN or set
lower limit (SETLL) in RPG) and then read only two records, blocking may
actually be a waste of system resources as the read after the random
position reads a single record; however, the next read reads a block of
records and the program reads only one.

The parameter on the override command that expresses the intent to block is
sequential only (SEQONLY) to emphasize that the intent is to read some
number of records in a row (you choose the number). Sequential only does
not preclude the program from doing random positioning; however, remember
the system reads an entire block of records when the second READ is run
after the position (the first read accesses a single record only).

Blocking is controlled by the defaults that are established at compile time
and by overrides that are run at run time. The compiler places an
information message into the compiler listing if a file will be opened in
blocking (sequential only) mode. Blocking is established at full open for
the file that is being opened, and all reads (READ verbs, not random
positioning verbs such as CHAIN) get an entire block of records. Blocking
is performed when the first I/O is performed after the open. The system
reads a block of records into the system buffer (located in the Open Data
Path (ODP) for that file) when the first read in the user's program is run
and then moves the first of these records into the program buffer so the
program can process that data. When the program next performs a read, the
next record in the block is moved into the program buffer by the program
(no system calls are made). When the block no longer contains records that
the program has not processed, another call to the system is made, and
another block of records is placed into the system buffer, the first of
these is placed into the program buffer, and the cycle continues until the
file is closed or the end of file is reached. If the program positions in
the file and then does a read, the old block of records is discarded, and a
new block of records is obtained.

Writes are essentially the same. The program fills the system buffer with
some number of writes before a call to the system to write the block and
the final block being written at file close time (if the last write did not
happen to occur at a block boundary).

The most common problem reported with programs that block records is the
timeliness of the data in the block. For example, if a program reads a
block at 8 a.m. with 8 records in the buffer and the program reads one
record an hour, the data that the program sees at 4 p.m. is really the data
as it looked at 8 a.m. rather than the state of the data at 4 p.m. A
similar program that writes a record an hour, would hold all 8 records in
its buffer until the 5 p.m. write. Someone may complain that they know that
the record was written in the morning. While the timeframe in the example
is somewhat extreme, the blocks of data represent the state of a system at
block boundaries rather than at read/write boundaries (as most programmers
have been taught to believe).

Traditional batch processing, which usually has no long waits and is often
run when there are no others accessing the data, is ideal for blocking
while interactive real time database access often requires a real time view
of data and should closely control blocking if it is done at all.

Note: Blocking is not out of the question in this case. In an application
that presents an entire screen of data, the all at one time nature of the
blocked read may be exactly what is required.

The use of the blocking can greatly improve performance, and most
applications are tolerant to minor timing problems. Batch program defaults
that systems use are often too small as they are often a compromise between
what is best for batch and keeping blocks within reasonable size to
minimize timing problems.

If the compiler has defaulted the open for sequential only, the system
chooses a block size such that the number of records fits in a block of 4K.
This default relates to the System/38 implementation and has never been
changed. Making this blocking factor larger typically greatly improves
batch performance but it may make some interactive application worse;
therefore, it was not changed.

The override database file (OVRDBF) command is used to control both the
size of the block and to turn blocking on and off.

The sequential only (SEQONLY) parameter can override the program
specification of blocking to not block (SEQONLY *NO) or change the size of
the block to match the program intent:

Interactive - To match the size of an interactive screen (presumes the
application presents eight records on a screen) one set of data presented
to a user interactively (SEQONLY *YES 8). Most interactive applications'
writes are probably best coded as immediate writes (SEQONLY *NO). An
alternative commonly mentioned is to use the FEOD operation to force the
buffer (and while this does force the writes to the database, it also
causes the data to be forced to secondary storage (DASD) and makes it
likely that the page on which the data resides will be stolen). FEOD is
almost always less efficient from a system-wide perspective than using
single writes.

Batch - To change the number of records in the block to match the maximum
size of a single system I/O operation. The value that will usually result
in the best system performance, has proven to be the size that is closest
to the maximum size of a single system disk I/O. The size for CISC system
is 32,767 (32K), and for RISC systems is 131,072 (128 K). The SEQONLY
parameter is expressed in the number of records. So one must calculate
number of records from the ideal block size by knowing the physical record
length and dividing that into the I/O size. The calculation of physical
record length can be somewhat difficult. The length of the data, at least
an additional database status byte, and the record may have additional
information on variable length fields, or null value indicators. A
reasonable approximation is 32000(128000)/rcdlen+1 if no nulls or variable
length else 28000(110000)/reclen. The best length is one that is less than
the machine maximum.

There is a second parameter on the OVRDBF command called number of records
(NBRRCDS). This parameter does not control blocking and unblocking at the
program level. Instead, it is a database directive that instructs the
database to initiate an asynchronous read of the next block of storage in
the physical file (in physical file arrival order). In essence, this is a
directive to start double buffering of the physical block of records off of
DASD, and the I/O is performed while the application is processing the
current block (for example, asynchronously). If the data is being processed
in physical order, this parameter can improve the efficiency of the
processing of that data. If the file is opened with sequential only and
processing is arrival order, database open uses the same value for
(default) NBRRCDS as specified in the sequential-only parameter. If the
file being processed is keyed, the NBRRCDS parameter is not defaulted. This
parameter may be specified if the physical order matches the key order or
the file is opened in arrival order but the options do not allow sequential
only (for example, file is opened for update, but the file will be
processed via sequential reads). If the file is processed by a keyed order
file and the physical order of the data are not the same as the keyed
order, coding the NBRRCDS parameter will cause the system to bring
(physically buffer) data that has very little likelihood of being
referenced. In other words, causes worse performance than if the parameter
were not coded.

__________________________________________________________________
           PMR Number: [Image]
        Related APARs: [Image]
        Related Public [Im[Image]
            Documents:
                       [Image]
                       [Image]
                       [Image]
                       [Image]
  IBM disclaims all warranties, whether express or implied, including, but
not limited to, the implied warranties of merchantability and fitness for a
 particular purpose. By furnishing this document, IBM grants no licenses to
 any related patents or copyrights. Copyright (c) 1996,1997,1998, 1999 IBM
 Corporation. Any trademarks and product or brand names referenced in this
 document are the property of their respective owners. Consult your product
                manuals for complete trademark information.