Re: performance in parsing xml with Expat

Thanks for all the input.
To answer a few - yes it's a one time run (12 xml files, each a years worth of orders) and each order
is approx 10 tables, and many hundreds of records per order. Each file could be 500 meg to a gig.
In a prev post weeks ago I explained the basic layout, & Barbara Morris suggested the native RPG opcodes not a good fit. The code around the Expat all working well, and there is some necessary logic
to get the output of the 10 tables "in sync". but not too much.
I can test the parse versus the other logic for performance, but think the buffer and the optimize may go long way. v6r1 btw.
Jim Franz

----- Original Message ----- From: "Scott Klement" <rpg400-l@xxxxxxxxxxxxxxxx>
To: "RPG programming on the IBM i / System i" <rpg400-l@xxxxxxxxxxxx>
Sent: Monday, August 27, 2012 6:32 PM
Subject: Re: performance in parsing xml with Expat

Hi Jim,

On 8/27/2012 2:49 PM, James Franz wrote:

Looking for suggestions to speed up parsing xml files with millions of records.
Using Scott's Expat RPGLE and works great, but taking couple hours in batch per file.

First of all, I don't deserve any credit for Expat. I had no part in
writing it at all. All I wrote was a few simple prototypes to call it
from RPG.

It's really hard to address a performance problem without first
determining where your program is spending it's time. Logic dictates
that the time would be spent interpreting/parsing the XML, but it could
also be spending time reading from disk, or it might be memory strapped,
causing it to slow down significantly due to that. Or, of course, it
could be in your back-end code (the code that runs in your handler routines)

Just some general suggestions:

1) Try recompiling Expat with OPTIMIZE(40) and DBGVIEW(*NONE) this can
make a significant difference in C code, and you're unlikely to need to
debug Expat, anyway.

2) Make sure the code that reads the XML data (you didn't say where
you're getting it from... should I assume a stream file?) is reading
optimially. If it's a stream file, make sure the buffer size is a
multiple of the disk block size (available from the statvfs() API). I
would suggest about 20-30 times the disk block size would be a good
starting value.

3) Try using the PEX APIs to insert milestone checkpoints that you can
use to pare down where the performance issues are occurring.

-SK
--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.