Re: Need to run multiple threads in one RPG program

On 1/19/2024 12:55 PM, smith5646 wrote:

When we refresh our test environments, there are some HUGE files that we
strip down after they are restored. Because we can't just strip down all
files, these specific files are kept in a driver file.

There is an RPGLE program that reads through the driver file. If the file
exists (already restored) and it is not locked (still being restored), the
program checks to see if the file exceeds the number of records to be kept.
If so, it strips it down. If the process does not find the file (not yet
restored) or it is locked (currently being restored), it bypasses it and
trips a flag so that the program knows that it needs to loop through the
driver file again to catch what was missed. When all of the files exist,
none of them are locked, and they are all below the record count threshold,
it ends. Due to the way it was written, this is single threaded processing.
I need to be able to multi thread the process and I really don't want to
have to rewrite the program to submit individual jobs.

Is there a "threading" process in RPG that would allow the RPG program to
process multiple files at the same time? I've tried to google but I'm
apparently not searching for the correct keywords.

Some refactoring, a dataqueue, and you should be good to go.
Your driver program only looks for eligible files. When it finds one, it puts the file/lib/member/whatever on a data queue, and loops around looking for the next one.

You write a second program that listens to the dataqueue, and processes/strips down the file. You can submit that one to batch and it'll read the data queue entries one at a time, processing files sequentially just as before.

Sounds like treading water until you submit a second and third and fourth processing job, each one listening to the same dataqueue. You can scale that up right until the disk arms cry for mercy.

You can implement the dataqueue code with the APIs QSNDDTAQ and QRCVDTAQ, or with SQL's DATA_QUEUE_ENTRIES table function