Re: Large XML file processing via XMLTABLE

Yeah, well ... testing today has not given me the results I was getting yesterday.
I think I was mistaken in thinking that processing the XML document from an XML column in an SQL table would be faster.
Ran a test today over a 35Meg file. I had to cancel the job after just under 3 hours. The xmltable had still not finished.

-----Original Message-----
From: RPG400-L [mailto:rpg400-l-bounces@xxxxxxxxxxxxxxxxxx] On Behalf Of Vernon Hamberg
Sent: Friday, February 22, 2019 1:21 PM
To: RPG programming on the IBM i (AS/400 and iSeries) <rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: Large XML file processing via XMLTABLE

Hi Paul

Interesting about copying to an SML column in a table - I didn't do that, never tested it, - Our XML is coming in from the field through a web service, and we put it into the IFS now - copying it would also take time,

But it'd be interesting to consider directly writing it to an XML column, maybe someone knows offhand whether that is possible.

Regards
Vern

On 2/21/2019 7:17 PM, Paul Therrien wrote:

Thanks Vern,
One thing that was slowing me down was I was working with an .xml file
in the IFS and passing that into XMLTABLE. - Very Slow.
I then copied the XML file to an SQL table to an XML column. And ran
XMLTABLE over that and it sped things up significantly.
It is still slow, but at least it finished processing the data in
under a half-hour.

Paul

-----Original Message-----
From: RPG400-L <rpg400-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of
Vernon Hamberg
Sent: Thursday, February 21, 2019 4:59 PM
To: RPG programming on the IBM i (AS/400 and iSeries)
<rpg400-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: Large XML file processing via XMLTABLE

Hi Paul

I'll take a stab at this - I'm going to say that XMLTABLE is probably
the fastest - at least no slower, and, for me, pretty easy to set up -
again, for me, easier than XML-INTO.

Here's why -we receive from our field associates several XML files,
one of which is for the various components of the service they do.

It was being processed with XML-SAX, and at 7.1 we had trouble with
conversion of "interesting" characters into CCSID(37).

I learned about XMLTABLE, and we use that now - it's a dream, and it
did character substitution better than RPG did - and I have around
half a dozen evocations of XMLTABLE, despite which, the time to
process things is no worse than it was with XML-SAX - the latter would
have a single pass through the XML, multiple uses of XMLTABLE would
have a pass for each use - yet, the overall time was not longer with those multiple passes.

I know that Jon P. doesn't grok XMLTABLE, and I stumble over complex
stuff with XML-INTO, and all the code I have to use for XML-SAX, well,
the SQL solution is just cleaner all around for me.

So that's as much as I know and have seen here - HTH!

Vern

On 2/21/2019 1:37 PM, Therrien, Paul via RPG400-L wrote:

(Cross Posting from Midrange-L)

We are on IBM I V7R3.

The immediate question: Is XMLTABLE the best way to process a large
XML

file?

We have XML files that contain over 40,000 individual data records
with

file sizes over 7 megs and one even over 70 megs in one instance.

I am parsing the data from XML into an SQL table using the following

statement:

exec sql
insert into pjtlib/zmpsnxfl
select x.*
from pjtlib/xmltext,
XMLTABLE( '$doc/IMB_Personalization/MailingData/Personalization'
passing info as "doc"
columns
mailingNumber varchar(5) default ' ' path

'../../MailingNumber'

, imbNumber varchar(21) default ' ' path '../IMB_Number'
, psnValue varchar(100) default ' ' path 'PsnAlpha'
, psnCode varchar(3) default ' ' path 'PsnCde'
, MFentity varchar(3) default ' ' path 'MFentity'
, MFelement varchar(3) default ' ' path 'MFelement'
) X

This works great for a file with a few hundred transactions in it.
But when we have 40,000 transactions it takes a lot of time. (I have
a file that is 35meg in size, with over 50,000 records and it took 20
minutes to complete.)

Is there a way to speed this up using XMLTABLE() or is there a more

efficient way other than XMLTABLE?

Thanks,

Paul

+++++ This email and related attachments may contain confidential

information intended exclusively for the addressee. Unauthorized use,
disclosure or distribution of this material is prohibited. If you
received this message in error, please advise the sender and delete all copies of it.
Content is provided by the individual sender and does not necessarily
reflect the views of the Company. Though sender believes this
transmission to be virus-free, it is the recipient's responsibility to ensure that it is.

--
This is the RPG programming on the IBM i (AS/400 and iSeries)
(RPG400-L) mailing list To post a message email:
RPG400-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list
options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link:
https://amazon.midrange.com

--
This is the RPG programming on the IBM i (AS/400 and iSeries) (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com