Re: How to speed up record I/O?

On 03/01/2010, at 5:17 AM, Mike Amos wrote:

This post is related to a post that I will make to the MI group.

You asked different questions in each append so I've answered in both places.

I have need to speed up our record IO - esp. reads - and have tried
every reasonable combination of record and stream reads/writes using
_Rreadx, read() and fread() via native and IFS file systems both with
and without the IFSIO compiler options.

First, if you want speed don't bother with the standard C interfaces (fopen etc.). Use the _R family instead.

The best read time that I can
achieve, using _Rreadn with blocking, is 2x slower that the native FTP,
read and transfer, of the same 306MB file; therefore, the read must be
on the order of 4x slower than the read that IBM is doing ILE-C. IBM FTP
is using _Rreadn with blocking since I see the following on the stack:
_Rreadn, __reads, QDBGETM. At first glance, this appears to be the same
as the call sequence via _Rreadn using ILE-C; however, looking at the
stack my invoking module only calls __reads and QDBGETM. On examining
the ILE-C headers, I find that _Rreadn is actually implemented as a
macro that invokes __reads directly. Enabling the call path to _Rreadn,
__reads, QDBGETM by disabling the macro makes little or no difference.
The ILE-C header files define the _Rreadn function so the call is
allowed via a public interface.

Also, investigating the ILE-C header files leads me to conclude that the
__DFT and __NO_LOCK are not the only record locking bits provided in the
full IBM implementation AND I suspect that record locking is always
taking place with ILE-C, even when not needed due to higher order locks
applied to the file and member. I conclude this since the delays are the
same in my test code for any locking combination. The difference between
__DFT and __NO_LOCK relating to the holding of locks after the read when
the file is opened for update and does NOT appear to relate to record
read locking.

I doubt the difference has anything to do with record locking (unless you have a lot of record contention occurring). Record locking is different from the locks applied to the *FILE and *MEM.

You say your file is opened for update? There's the cause of the speed difference. When processing for update you can block records into main storage but you cannot block them into your program. Your program buffer has room for only one record. This is to flush the changed record out to main storage when it is updated. FTP does no updating just read/write and can block both operations with double-buffering in effect. Double-buffering refers to blocking records into main storage and blocking in your program.

If your program is opening the file for UPDATE then comparing I/O rates to a program that performs READS/WRITES is futile.

1. Does anyone know of a way to significantly speed up the read (and/ or
write) operation?

Read the file sequentially, use OVRDBF to set NBRRCDS(nnnn) and SEQONLY(*YES nnnn). Set nnnn to the maximum number of records that will fit in the maximum fetch size for your version of the OS. Used to be 32K but now I think is 128K.

Even if you MUST process the file in keyed sequence you can get significant speed increase by reorganising it in the desired keyed sequence and then process it sequentially. The speed improvement in the I/O can often outweigh the cost of the reorganise. You will need exclusive access to perform the reorganise.

NBRRCDS controls blocking into main storage. SEQONLY controls blocking in your program.

2. Does anyone know where I can find complete documentation for either
the __reads or _Rreadn call/macro interfaces?

No. As far as I know the documented interface is the _R family of functions.

3. Does anyone know where I can find additional record locking
information specific to __reads or _Rreadn?

As I said above, I doubt the problem is due to record locks.

4. Does anyone have an explanation for why the IBM blocked read via
QDBGETM is so much faster than the comparable ILE-C blocked read via
QDBGETM? I have not tried this with RPG as I have not touched RPG for
about 15 years and the RPG interface looked to run the same as the ILE-C
interface.

See above re update vs. read/write. It is always possible that IBM have faster internal access methods but I think in this case the more likely explanation is entirely due to the type of file processing performed by your program.


Regards,
Simon Coulter.
--------------------------------------------------------------------
FlyByNight Software OS/400, i5/OS Technical Specialists

http://www.flybynight.com.au/
Phone: +61 2 6657 8251 Mobile: +61 0411 091 400 /"\
Fax: +61 2 6657 8251 \ /
X
ASCII Ribbon campaign against HTML E-Mail / \
--------------------------------------------------------------------