Re: Huge XML file processing from IFS (IBMi)

Glad my contribution helped and that got ot running.

If you decide to try XML-SAX later, I've written a couple of introductory pieces on the topic, but only one still seems to be available on the web. You can find it here: https://www.itjungle.com/2016/06/14/fhg061416-story01/ <https://www.itjungle.com/2016/06/14/fhg061416-story01/>

XML-SAX is a lot more flexible, and can handle situations that are all but impossible with XML-INTO. But it is a lot more work. The article referenced above talks about some of the areas where XML-SAX scores.


Jon P.

On Jul 31, 2023, at 1:54 PM, Prakash Shanmugam <pshanmug@xxxxxxxxx> wrote:

Looks like it was an issue with either DS declaration or XML-INTO options.
After doing minor adjustments, found it to be working fine.
Thank you all, for your inputs.
PS: Due to array size limitation, using %HANDLER to retrieve XML data in
multiple sets.

Thank you Jon Paris. Added that option to XML-INTO.

Have not used SAX before. Will probably give it a try. Thanks Alan!

It is going to be a daily scheduler job Brad. Agree about XML vs JSON.
Thanks!

You are absolutely correct Charles. Thanks!

Got it Vernon. Thanks for your suggestion.

On Thu, Jul 27, 2023 at 5:04 PM VERNON HAMBERG Owner via RPG400-L <
rpg400-l@xxxxxxxxxxxxxxxxxx> wrote:

Yeah, I see limits for SQL in the 2 147 483 647 area - your 20MB is not
even close.


I tend to like SQL's XML_TABLE myself - to me it seems very similar to a
CREATE TABLE in the syntax. But you pays your money and you takes your
choice!


Cheers
Vern


On Thu, 27 Jul, 2023 at 4:26 PM, Charles Wilt <charles.wilt@xxxxxxxxx>
wrote:


To: rpg programming on ibm i

I would expect the XML-INTO parser to handle CDATA just fine...

Are you sure that's where your problem lies?

20MB isn't that big...I've hit the 1GB* limit in XML-INTO for XML docs
before ;)

*1GB for the default UCS processing...change it to SBCS and double your
limit.

Charles

On Thu, Jul 27, 2023 at 2:45 PM Prakash Shanmugam <pshanmug@xxxxxxxxx
<mailto:pshanmug@xxxxxxxxx>>
wrote:

Hi All,
Would like to get your thoughts on the below situation-

Need to process a huge XML (around 20MB size currently with 25K rows and
could grow) from IFS folder using ILE RPG.


First trying with XML-INTO, but got below error, due to XML data

containing

CDATA.


XML data element: <PriceListID><![CDATA[1]]></PriceListID>
Data structure subfield declaration: PriceListID varchar(25);

Error Message: The XML document does not match the RPG variable; reason
code 5.
Reason code 5. The XML document contains extra XML attributes or elements
that do not match subfields.

Kindly suggest how to parse <![CDATA[...]], to get the value.


Understand another option is to use the SQL capabilities. Came to know

that

it also has some limits. Not sure what are the limitations and how to get
around these? Any examples of using SQL to read XML from IFS?

Thanks!
--
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx<mailto:

RPG400-L@xxxxxxxxxxxxxxxxxx>

To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx<mailto:

RPG400-L-request@xxxxxxxxxxxxxxxxxx>

Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx<mailto:

support@xxxxxxxxxxxxxxxxxxxx> for any subscription related

questions.

--
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx<mailto:
RPG400-L@xxxxxxxxxxxxxxxxxx>
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx<mailto:
RPG400-L-request@xxxxxxxxxxxxxxxxxx>
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx<mailto:
support@xxxxxxxxxxxxxxxxxxxx> for any subscription related questions.
--
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related
questions.

--
This is the RPG programming on IBM i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/rpg400-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.