RE: YAJL and BIG files

The company has decided to switch directions from my program which has a 16M limitation and they want to use Birgitta's SELECT2JSON and WRT2IFS (not sure that last one is spelled correctly). Since these are written in SQL language, I am lost but I believe those still have the 16M limitation.

Someone in the company mentioned a CLOB_LOCATOR (not sure of the spelling) but again, not knowing SQL language, I am sitting with the "deer in the headlights" look on my face.

This might be a question for Birgitta since she owns the tools but I will take anyone's answer. Is there a way for SELECT2JSON to bypass the 16M limitation?

To answer Jim's questions, one of the files is about 10M as it currently sits in the DB2 PF with packed fields and no field names...and they expect the data to grow well beyond that. The other answer is that JSON is a requirement. I don't know why, it just is.



-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of Jon Paris
Sent: Thursday, January 23, 2025 3:55 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: YAJL and BIG files

If the created json is too big Jim then xml would likely be worse. xml is typically 20% plus larger and probably has the same buffer limitations.

I wonder if data-gen would work for the OP as it allows the data to be built in chunks if needed. Similarly, you could always run the current solution for a defined number of items and then switch to a new file and merge the files when all is done.

SQL may also be an option but I suspect that size limits would come into play there too.


Jon Paris
Jon.Paris@xxxxxxxxxxxxxx

On Jan 23, 2025, at 3:23 PM, Jim Franz <franz9000@xxxxxxxxx> wrote:

is json preferred or required? If xml can be an option, there are
tools (including Scott's xpat port (I've used this when other tools
had size limits)... Sounds like you need something to read/build in chunks.
Define "big".

Jim Franz

On Thu, Jan 23, 2025 at 2:42 PM <smith5646midrange@xxxxxxxxx> wrote:

Nevermind. I got the answer in a different form. The program abended.
Apparently, it exceeded the max space for the YAJL buffer.

-----Original Message-----
From: smith5646midrange@xxxxxxxxx <smith5646midrange@xxxxxxxxx>
Sent: Thursday, January 23, 2025 2:27 PM
To: 'Midrange Systems Technical Discussion'
<midrange-l@xxxxxxxxxxxxxxxxxx

Subject: YAJL and BIG files

I have a program that uses SQL to read a table and write the JSON
reformatted data to the IFS using YAJL. This is going to be a BIG file.
Since YAJL builds the file in memory, this is going to consume a LOT
of memory.

In the Windows world, when a program needs memory and none is
available, Windows shuffles some memory to disk and frees up that
memory. Does the IBM i basically work the same way or am I going to
eat all of the memory and crash the machine? ☹

I'm thinking that I might need to rewrite the program to get away
from YAJL and just manually write each record to the IFS as it is processed.

Thoughts?



--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription
related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related questions.