×
The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.
My company exchanges XML files with other tolling agencies for national interoperability. We retrieve files from an interop hub, unzip them in the IFS and process them using RPG and XML-INTO, which I've found to perform really well. The business rules require that we ack each file within an hour, which is normally not a problem, but I struggled initially with one of the files, a weekly bulk electronic transponder file, because it is north of 4GB unzipped. The RPG XML operations have an approx. 2GB file size limitation, so I ended up using Java and experimented with different approaches to be able to process and ack the file in under an hour. Currently, we retrieve, unzip and parse these large XML files and load ~33-34 million records into two DB2 tables in around 10 mins. My RPG transponder file processing program employs some Java classes, which implement the Java STaX parser functionality to parse the XML. To load the data into DB2, I found the best performance using the Java Toolbox type 4 JDBC driver to insert records in 500-1000 record batches. I experimented with various batch sizes, but 500-1000 seems to perform best. The batch insert approach really speeds things up. In RPG, a similar approach can be accomplished with data structure arrays. I found the type 4 driver performs faster than the type 2 driver, and the JDBC batch insert approach faster than the Toolbox RLA classes. I also tested parsing the large XML files using Expat because of the lack of a file size limitation, but found Java to be a lot faster in my experience.
I would be glad to answer any questions, Paul, if you are interested in potentially utilizing any of these approaches.
Regards,
Blake Butterworth
Application Development Manager
Kansas Turnpike Authority
As an Amazon Associate we earn from qualifying purchases.
This thread ...
Re: Large XML file processing via XMLTABLE, (continued)
This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact
copyright@midrange.com.
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.