Re: YAJL and BIG files

Once the company saw the source to Brigitta's functions, they changed
their mind. No offense intended Birgitta. They just don't have anyone
that can support SQL language procedures and that concerned them.

I have been tasked with building the program that manually builds the JSON
data (braces, brackets, etc.) and writes it to the IFS. My plan is to
build a buffer and periodically dump it to the IFS which I have that
working.

Now my question is when to dump the buffer to the IFS. Does anyone know
if there an optimum buffer size to write to the IFS? In other words where
is the best performance between using a ton of small writes using a 1K
buffer vs one write using a 10M buffer and tying up that 10M of memory in
an RPGLE program.


-----Original Message-----
From: smith5646midrange@xxxxxxxxx <smith5646midrange@xxxxxxxxx>
Sent: Thursday, January 23, 2025 4:22 PM
To: 'Midrange Systems Technical Discussion' <midrange-l@xxxxxxxxxxxxxxxxxx

Subject: RE: YAJL and BIG files

The company has decided to switch directions from my program which has a
16M limitation and they want to use Birgitta's SELECT2JSON and WRT2IFS (not
sure that last one is spelled correctly). Since these are written in SQL
language, I am lost but I believe those still have the 16M limitation.

Someone in the company mentioned a CLOB_LOCATOR (not sure of the spelling)
but again, not knowing SQL language, I am sitting with the "deer in the
headlights" look on my face.

This might be a question for Birgitta since she owns the tools but I will
take anyone's answer. Is there a way for SELECT2JSON to bypass the 16M
limitation?

To answer Jim's questions, one of the files is about 10M as it currently
sits in the DB2 PF with packed fields and no field names...and they expect
the data to grow well beyond that. The other answer is that JSON is a
requirement. I don't know why, it just is.



-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of Jon
Paris
Sent: Thursday, January 23, 2025 3:55 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: YAJL and BIG files

If the created json is too big Jim then xml would likely be worse. xml is
typically 20% plus larger and probably has the same buffer limitations.

I wonder if data-gen would work for the OP as it allows the data to be
built in chunks if needed. Similarly, you could always run the current
solution for a defined number of items and then switch to a new file and
merge the files when all is done.

SQL may also be an option but I suspect that size limits would come into
play there too.


Jon Paris
Jon.Paris@xxxxxxxxxxxxxx

On Jan 23, 2025, at 3:23 PM, Jim Franz <franz9000@xxxxxxxxx> wrote:

is json preferred or required? If xml can be an option, there are
tools (including Scott's xpat port (I've used this when other tools
had size limits)... Sounds like you need something to read/build in

chunks.

Define "big".

Jim Franz

On Thu, Jan 23, 2025 at 2:42 PM <smith5646midrange@xxxxxxxxx> wrote:

Nevermind. I got the answer in a different form. The program abended.
Apparently, it exceeded the max space for the YAJL buffer.

-----Original Message-----
From: smith5646midrange@xxxxxxxxx <smith5646midrange@xxxxxxxxx>
Sent: Thursday, January 23, 2025 2:27 PM
To: 'Midrange Systems Technical Discussion'
<midrange-l@xxxxxxxxxxxxxxxxxx

Subject: YAJL and BIG files

I have a program that uses SQL to read a table and write the JSON
reformatted data to the IFS using YAJL. This is going to be a BIG file.
Since YAJL builds the file in memory, this is going to consume a LOT
of memory.

In the Windows world, when a program needs memory and none is
available, Windows shuffles some memory to disk and frees up that
memory. Does the IBM i basically work the same way or am I going to
eat all of the memory and crash the machine? ☹

I'm thinking that I might need to rewrite the program to get away
from YAJL and just manually write each record to the IFS as it is

processed.

Thoughts?



--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription
related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription

related questions.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related
questions.



--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxxxxxxxxxx for any subscription related
questions.