Depending on HOW LARGE the memory chunk will be you would want to do some heavy testing before removing the tables from the mix.
If the process cracks during processing what is the fallout/recovery ?
Maybe you've already though that out.
If you're writing to tables it shouldn't really be inefficient.
Hard to know without more specifics.
If it makes sense you could always move your temp tables to a library called TEMP or TMP and then the tables can be shared and potentially permanent.
Again not knowing the potential for process collisions could make things interesting.
Another thought could be to re-create that process in something like Python and try using something like MariaDB or even SQLite if only temporary storage is needed. SQLite is a smoking fast local db.
These are just ideas to think about.
Regards,
Richard Schoen
Web:
http://www.richardschoen.net
Email: richard@xxxxxxxxxxxxxxxxx
----------------------------------------------------------------------
message: 1
date: Tue, 27 Feb 2024 09:02:28 -0500
from: Brian Garland via MIDRANGE-L <midrange-l@xxxxxxxxxxxxxxxxxx>
subject: Passing large amount of data between programs
This is sort of a best practice question.
We have an RPGLE based web service that prices an order. Since we already had pricing routines that worked off database tables containing order information we take what comes into the web service and create a QTEMP version of the tables with the order and run our existing pricing routines.
This has proven to be fairly slow so we started analysing where. Currently the largest chunk of time is spent clearing and writing to these tables.
We do build them once per job which makes a difference. We also include some indexing. Our initial thought is that if we could keep the information in memory and modify our pricing routine to work off that memory we could improve the performance. The fallout from this would be updating our routine that runs off tables to copy the info needed into memory before pricing and the update the tables afterward. That is to avoid having two routines that do the same thing. This bit of extra time on that process is okay.
So, the question is what is the best way to pass that chuck of memory between programs so that several programs can each update their part?
Brian
midrange.com
