RE: YAJL and BIG files

SQL may also be an option but I suspect that size limits would come into play there too.

SQL can handle JSON data up to 2 GB.
When working in embedded SQL with RPG-Variables the maximum size is restricted to the RPG Limit (16MB).
When using CLOB-Locators or reading/writing IFS Files embedded SQL RPG can also handle up 2GB.

Mit freundlichen Grüßen / Best regards

Birgitta Hauser
Modernization – Education – Consulting on IBM i
Database and Software Architect
IBM Champion since 2020

"Shoot for the moon, even if you miss, you'll land among the stars." (Les Brown)
"If you think education is expensive, try ignorance." (Derek Bok)
"What is worse than training your staff and losing them? Not training them and keeping them!"
"Train people well enough so they can leave, treat them well enough so they don't want to. " (Richard Branson)
"Learning is experience … everything else is only information!" (Albert Einstein)

-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of Jon Paris
Sent: Thursday, 23 January 2025 21:55
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: YAJL and BIG files

If the created json is too big Jim then xml would likely be worse. xml is typically 20% plus larger and probably has the same buffer limitations.

I wonder if data-gen would work for the OP as it allows the data to be built in chunks if needed. Similarly, you could always run the current solution for a defined number of items and then switch to a new file and merge the files when all is done.

SQL may also be an option but I suspect that size limits would come into play there too.


Jon Paris
Jon.Paris@xxxxxxxxxxxxxx

On Jan 23, 2025, at 3:23 PM, Jim Franz <franz9000@xxxxxxxxx> wrote:

is json preferred or required? If xml can be an option, there are
tools (including Scott's xpat port (I've used this when other tools
had size limits)... Sounds like you need something to read/build in chunks.
Define "big".

Jim Franz

On Thu, Jan 23, 2025 at 2:42 PM <smith5646midrange@xxxxxxxxx> wrote:

Nevermind. I got the answer in a different form. The program abended.
Apparently, it exceeded the max space for the YAJL buffer.

-----Original Message-----
From: smith5646midrange@xxxxxxxxx <smith5646midrange@xxxxxxxxx>
Sent: Thursday, January 23, 2025 2:27 PM
To: 'Midrange Systems Technical Discussion'
<midrange-l@xxxxxxxxxxxxxxxxxx

Subject: YAJL and BIG files

I have a program that uses SQL to read a table and write the JSON
reformatted data to the IFS using YAJL. This is going to be a BIG file.
Since YAJL builds the file in memory, this is going to consume a LOT
of memory.

In the Windows world, when a program needs memory and none is
available, Windows shuffles some memory to disk and frees up that
memory. Does the IBM i basically work the same way or am I going to
eat all of the memory and crash the machine? ☹

I'm thinking that I might need to rewrite the program to get away
from YAJL and just manually write each record to the IFS as it is processed.

Thoughts?



--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription
related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.