Re: Need ideas to reduce open/close in RPG program

It would indeed require defining** all of them and then keeping them all open. But other than by using SQL how else could you possibly avoid the open/close overhead? Regardless of which approach you use, some program(s) or other has to be keeping all the files open.

Normally (using either of the approaches I suggested) you might also incorporate the ability to selectively close/reopen one of the files if this was essential to allow other processing to proceed. But then you must already be dealing with that problem - i.e. when a message arrives on a queue that applies to a file that is not “open for business” currently.

** If it were an absolute requirement to have an infinite number of files available and it was not practical to recreate the “holding” program every time a new one came on stream then you could have the “holding” program written in C which would allow any number of files to be opened based on some table or other. For that matter you could use the C open APIs from an RPG program which would avoid the need for C programming skills - you just need to copy an example of using the C function from RPG and there are a few of those around.

On Dec 12, 2015, at 12:11 PM, Vinay Gavankar <vinaygav@xxxxxxxxx> wrote:

Thanks everyone.
Buck: They do use performance gathering tools (iDoctor etc), and I think it
showed that the process was doing open/close at the rate of about 1000 per
second (300K in a 5 minute period). Hence the panic.

They are not very keen on on using SQL here, don't know why.

Glenn: To open all the files at once, wouldn't I have to 'define' all the
files (with actual names or dummy names) in the program?

Ken: Every update is done to every file defined in the control file. So
keeping a file open is not really an option.

Jon: Keeping 'all' the files open would imply defining all the files in the
program, right?

Mark: It is a very 'elegant' solution, and I am going to shoot for that. I
am not sure that they will accept it as a 'quick fix', but definitely as a
long term solution.

On Sat, Dec 12, 2015 at 8:54 AM, Buck Calabro <kc2hiz@xxxxxxxxx> wrote:

I seem to recall that we achieved a “soft” file open/close in RPG
by opening all the files in a CL at the root level of the job and
sharing the ODP. The programs would then issue and OVR
to the version they needed and do their own opens to the
shared ODP.

Having a shared ODP meant quite literally that the file cursor was the
same across multiple programs in the job. Imagine doing a SETLL in
PGMA and then CALL PMB. PGMB does a READ and it gets the first record
pointed to by PGMA.

We definitely did this for performance reasons, but there was a fair
amount of programmer mental overhead. Adding a new file to this
system meant tracing back to where the OPEN happened and adding it
there, too - often in a CLP, or a bespoke RPG program that did nothing
but OPEN and RETURN.

Which reminds me of the other half of those ancient times. RETURN
with LR OFF. RPG does an initialisation of all sorts of things when
it starts up. We all know and love this. Well, if we CALL PGMB which
does a RETURN, and then CALL PGMB again, the RPG runtime will detect
that LR was not on and will skip most of the initialisation. The
point of that is that this is code which does not run thousands and
thousands of times. The Dark Side is that you, the RPG programmer,
need to be very, very careful about /assuming/ the program variables
are reset between CALLs.

If you're thinking about this (and I regret that I am even mentioning
it) then make sure you are running in a single named activation group.
*NEW is not a performance enhancer. There's nothing wrong with *NEW
don't take this as a blanket recommendation to eliminate it - but if
you want to wring every millisecond out of a job, don't use *NEW.

There is one other ancient technique that we used to use back when
terminals were actual 5250 devices the size of a small refrigerator:
FMTSLR. The idea was to create a logical file that had multiple
record formats, typically an order header, detail and total. Just
like program described files <shudder>. The format selector was an
additional program one would write which would tell data management
which of those record formats to write to.

If it were my system and I was the lord of all I survey, I'd use
Mark's technique of a 'router' and offload the actual I/O to one or
more asynchronous processes. But I've never been in that position,
and you may not be either.

Which brings me to the most important idea of all. MEASURE FIRST.
Blindly poking at performance problems often results in more
performance problems. Measure the process, analyse the stats, and
take action based on them. Someone's gut feeling that it's an
OPEN/CLOSE problem may be in the same league as someone's idea to
simply duplicate the files instead of adding another key. You are FAR
more likely to fix the actual performance problem if you identify it.
It could be something as simple as allocating more memory to a
subsystem now that the volume and number of files has grown...

--buck
--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.

--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.

Jon Paris

www.partner400.com
www.SystemiDeveloper.com