Re: Avoiding disk arms with CLOB

I'm not even trying to touch performance tuning for that same reason :)
and its outside my area :)

That said, I'm looking at a single load line having potentially a length
of over 12k and with a limit of 64512 on a queue I'm limited in the number
of lines I can handle. They will all come through as one stream if you
will.

On 9/19/2013 9:21 AM, RPGLIST wrote:

I'm facing the need to process a very large amount of data, which is
more
than likely going to come in as a SOAP or XML stream but I need to
process
this quickly and without losing time with the disk arms if at all
possible.

If you're going to store this information, you're going to have to write
it to disk at some point.

I know that a clob will allow me to store a larger set of data but from
what I'm reading, you can't just access and read it like a normal
variable.

Correct.
http://www.ibmsystemsmag.com/ibmi/developer/general/BLOBs,-CLOBs-and-RPG/

I assume, and if I'm wrong feel free to slap with a lead pipe, that like
with regular PF's writes are going to cost me time. I know some
individuals don't care about time or say its not important but I need to
return a response in a matter of seconds like 5-8 or lower if possible
and
I'm not going to argue that point. It is what it is.

What's the application like? Is it a simple matter of receive, store,
acknowledge and process later? Or is it a case of receive a line,
process a line, store the data, acknowledge the transaction? The advise
you get will depend a lot on the way you need to handle things.

The data coming in if you put it in a PF would have a record length of
somewhere in the 12k range, so you can imagine it'll be larger with XML
tags and that's a single record, whereas we could receive up to 99 lines
with each line in the 12k+ range.

Offhand, 120k sounds like no big deal in terms of disk I/O. Have you
mocked something up and found a performance issue somewhere, or is the
question a matter of being proactive? After 30+ years of tuning
applications, I feel confident stating that performance tuning without
measurement is the road to madness. Ask my therapist :-)

Now, if someone has a suggestion I'm all ears :)

If you suspect disk I/O will be the bottleneck (and you know your
machine better than I do!) then consider creating an asynchronous
process to do the writes. One job will receive the data from your
trading partner and then say, dump it into a data queue and keep
reading. Another job will read entries off the queue and write them to
disk.
--buck
--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L)
mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.