Re: Large XML file processing via XMLTABLE

I would imagine that Python, PHP or Java on IBMi would handle this OK as well.

But I agree the PC as a pre-processor is always an option.

Regards,
Richard Schoen
Director of Document Management
e. richard.schoen@xxxxxxxxxxxxxxx
p. 952.486.6802
w. helpsystems.com

------------------------------

message: 4
date: Fri, 22 Feb 2019 16:13:11 -0500
from: John Yeung <gallium.arsenide@xxxxxxxxx>
subject: Re: Large XML file processing via XMLTABLE

I think the midrange list (which this was originally posted to) would have been a more appropriate venue for this, but since the RPG list is where people seem to have responded, I guess it's stuck here.

My question is: Is this a one-time thing, or will it be recurring?

The reason I ask is that if it's a one-time thing, then in your shoes, I would preprocess the huge XML files on my PC. In my experience, for many tasks, my PC is quite a bit faster than our i. This is not a knock on the i at all, because the i handles multiple users and heavy disk I/O loads quite well, whereas my PC only has to handle one user.

So in this case, I would use tools on my PC (such as Python, but also conceivably other software) to transform the XML into either smaller, more manageable pieces, or a more manageable transport format, or (even more likely) just insert the final, parsed data directly into the database.

Regardless of whether you're sticking to RPG or open to alternatives, one thing to keep in mind is that SAX-style parsing is generally better suited to very large files, because it only needs to work with a small chunk of the file at a time; whereas DOM-style parsing (as exemplified by XML-INTO) needs to work with the whole file basically as a unit. When DOM parsing works, it is generally faster than SAX.
But maybe you've reached the threshold where DOM is impractical. At a minimum, SAX will more easily allow you to monitor your progress. If you try to load the whole thing at once with DOM, and you quit after 3 hours, you have no idea whether just waiting 10 minutes more would have allowed it to complete, or whether it still wouldn't have finished after another 3 hours.

John Y.