Re: Passing large amount of data between programs

If the desire is to have the tables in question at a single point in time,
maybe temporal tables might help?

I'm still trying to digest the use case for why a copy of the data is
needed. That seemingly is the problem statement, the time needed to make
the copy, so why make it in the first place?

--
Jim Oberholtzer
Chief Technical Architect
Agile Technology Architects


On Tue, Feb 27, 2024 at 8:30 AM Richard Schoen <richard@xxxxxxxxxxxxxxxxx>
wrote:

Depending on HOW LARGE the memory chunk will be you would want to do some
heavy testing before removing the tables from the mix.

If the process cracks during processing what is the fallout/recovery ?

Maybe you've already though that out.

If you're writing to tables it shouldn't really be inefficient.

Hard to know without more specifics.

If it makes sense you could always move your temp tables to a library
called TEMP or TMP and then the tables can be shared and potentially
permanent.

Again not knowing the potential for process collisions could make things
interesting.

Another thought could be to re-create that process in something like
Python and try using something like MariaDB or even SQLite if only
temporary storage is needed. SQLite is a smoking fast local db.

These are just ideas to think about.

Regards,
Richard Schoen
Web: http://www.richardschoen.net
Email: richard@xxxxxxxxxxxxxxxxx


----------------------------------------------------------------------

message: 1
date: Tue, 27 Feb 2024 09:02:28 -0500
from: Brian Garland via MIDRANGE-L <midrange-l@xxxxxxxxxxxxxxxxxx>
subject: Passing large amount of data between programs

This is sort of a best practice question.

We have an RPGLE based web service that prices an order. Since we already
had pricing routines that worked off database tables containing order
information we take what comes into the web service and create a QTEMP
version of the tables with the order and run our existing pricing routines.

This has proven to be fairly slow so we started analysing where.
Currently the largest chunk of time is spent clearing and writing to these
tables.
We do build them once per job which makes a difference. We also include
some indexing. Our initial thought is that if we could keep the
information in memory and modify our pricing routine to work off that
memory we could improve the performance. The fallout from this would be
updating our routine that runs off tables to copy the info needed into
memory before pricing and the update the tables afterward. That is to
avoid having two routines that do the same thing. This bit of extra time
on that process is okay.

So, the question is what is the best way to pass that chuck of memory
between programs so that several programs can each update their part?

Brian


--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related
questions.