Re: Avoiding disk arms with CLOB

Here would be a simplistic description of the process:

1. Receive the load information from the client (Shipper, Receiver, etc.)
and that includes all lines, each line being a stop.

2. Parse and process each line, writing it to a table at the end of the
process including our charges (if any) and line status (accepted /
rejected).

3. return an xml back to the client with basically the same data they sent
us but with a status update for each line (charges and status).

If a shipper sends me more than say 4 lines with every xml tag completed
I'm over the 32k limit...

On 9/19/2013 9:21 AM, RPGLIST wrote:

I'm facing the need to process a very large amount of data, which is
more
than likely going to come in as a SOAP or XML stream but I need to
process
this quickly and without losing time with the disk arms if at all
possible.

If you're going to store this information, you're going to have to write
it to disk at some point.

I know that a clob will allow me to store a larger set of data but from
what I'm reading, you can't just access and read it like a normal
variable.

Correct.
http://www.ibmsystemsmag.com/ibmi/developer/general/BLOBs,-CLOBs-and-RPG/

I assume, and if I'm wrong feel free to slap with a lead pipe, that like
with regular PF's writes are going to cost me time. I know some
individuals don't care about time or say its not important but I need to
return a response in a matter of seconds like 5-8 or lower if possible
and
I'm not going to argue that point. It is what it is.

What's the application like? Is it a simple matter of receive, store,
acknowledge and process later? Or is it a case of receive a line,
process a line, store the data, acknowledge the transaction? The advise
you get will depend a lot on the way you need to handle things.

The data coming in if you put it in a PF would have a record length of
somewhere in the 12k range, so you can imagine it'll be larger with XML
tags and that's a single record, whereas we could receive up to 99 lines
with each line in the 12k+ range.

Offhand, 120k sounds like no big deal in terms of disk I/O. Have you
mocked something up and found a performance issue somewhere, or is the
question a matter of being proactive? After 30+ years of tuning
applications, I feel confident stating that performance tuning without
measurement is the road to madness. Ask my therapist :-)

Now, if someone has a suggestion I'm all ears :)

If you suspect disk I/O will be the bottleneck (and you know your
machine better than I do!) then consider creating an asynchronous
process to do the writes. One job will receive the data from your
trading partner and then say, dump it into a data queue and keep
reading. Another job will read entries off the queue and write them to
disk.
--buck
--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.