Re: Loading Variable Array From SQL Caveat

[VERNON HAMBERG Owner <vhamberg@xxxxxxxxxxxxxxx>]

Hi Sam


That confirms my thoughts on the limits since 7.4.


As to whether you get all the records, didn't someone mention the SQLERRD structure? Oh yeah. SQLERRD(3) has the number of rows fetched. That looks useful to check if everything is done.

Regards
Vern


On Fri, 16 Feb, 2024 at 11:00 AM, Sam_L <lennon_s_j@xxxxxxxxxxx> wrote:
 

To: rpg400-l@xxxxxxxxxxxxxxxxxx

Replying to my own post...
I have resolved, I think, my issue.  It depends on how much data you are 
requesting in the fetch. In my initial testing I was loading 4 fields, 
total 36 bytes and this loaded only 32767rows.
If I load only an int(10) field, 4 bytes, then all my 42654 elements load.

I tweaked Marco's program to use my 4 field data and it also loads only 
32767 rows.
Significant changes were:
Limits(1) = 10000;
Limits(2) = 20000;
Limits(3) = 41000;
Where rrn(a)  <= :wD;

Takeaways:
1) If you fetch multiple rows into an array, each fetch loads starting 
at element 1.
2) Depending on the amount of data you are loading, you may not get all 
the rows you asked for. SQL does not show end of file, but if you do 
another fetch for the rest of the data, 1) applies.

Granted, it is most likely an extreme edge case to load this many 
records. I was creating a program to create large amounts of test data 
for a performance test, and wanted realistic city/state/zip data. It was 
a chance to experiment with random numbers and variable length arrays 
and I wanted to see the impact of re-allocating memory as the array 
increased vs trying to allocate enough memory once. My take is that 
unless you are loading the array many times it makes little observable 
difference.

On 2/15/2024 4:13 PM, Sam_L wrote:
> What am I doing wrong in the code below? Each subsequent fetch from the 
> cursor seems to load into element 1 of the array.
>
> (There are 42,654 rows to be loaded. Granted, this is an unusually large 
> number to load, but I'm accessing it by index to build random test data 
> and loading rows one at a time works great, so I'm not stuck. But I am 
> curious about this and why I can't get this to work.)
>
> Incidentally, snd-msg is a great replacement for dsply in these 
> situations--thanks Marco.
>
> After a run of the code below, these are the results:
>
> Allocated default: 100
> Allocated after *alloc: 42755
> "fetch for 'n' rows" : 42654
> Elements before fetch: 0
> After First Fetch. Elements: 42654. SQLSTATE: 00000. Allocated: 42755
> After Loop Fetch.  Elements: 0. SQLSTATE: 02000. Allocated: 42755
> Final Elements: 0. Allocated: 42755
>
> If I set rows to fetch to 25,000 these are the results:
>
> Allocated default: 100
> Allocated after *alloc: 42755
> "fetch for 'n' rows" : 25000
> Elements before fetch: 0
> After First Fetch. Elements: 25000. SQLSTATE: 00000. Allocated: 42755
> After Loop Fetch.  Elements: 17654. SQLSTATE: 00000. Allocated: 42755
> After Loop Fetch.  Elements: 0. SQLSTATE: 02000. Allocated: 42755
> Final Elements: 0. Allocated: 42755
>
> This is the demo code:
> **free
> ctl-opt Option(*nounref: *nodebugio: *srcstmt);
> dcl-c SQLSUCCESS '00000';
> dcl-c SQLNOMOREDATA '02000';
> dcl-ds csz_a dim(*auto : 50000) qualified;
>      zip int(10);
> end-ds;
> dcl-s cszCount int(10);
> exec sql set option closqlcsr=*endmod;
> exec sql declare csz_cur cursor for
>    select zip
>    from lennons1.csz where length(trim(city)) <= 20;
>
> // Allocate enough memory/elements
> exec sql select count(*) into :cszCount
>    from lennons1.csz  where length(trim(city)) <= 20;
> snd-msg ('Allocated default: ' + %char(%elem(csz_a : *alloc)));
> %elem(csz_a : *alloc) = cszCount + 1;
> snd-msg ('Allocated after *alloc: ' + %char(%elem(csz_a : *alloc)));
> // cszCount = 25000; // pick a number, e.g. 25,000
> snd-msg ('"fetch for ''n'' rows" : ' + %char(cszCount));
> snd-msg ('Elements before fetch: ' + %char(%elem(csz_a)));
> exec sql open csz_cur;
> exec sql fetch from csz_cur for :cszCount rows into :csz_a;
> snd-msg ('After First Fetch. Elements: ' + %char(%elem(csz_a))
>      + '. SQLSTATE: ' +SQLSTATE+ '. Allocated: '
>      +%char(%elem(csz_a : *alloc)));
> dow sqlstate <> SQLNOMOREDATA;
>      exec sql fetch next from csz_cur for :cszCount rows into :csz_a;
>      Snd-msg  ('After Loop Fetch.  Elements: ' + %char(%elem(csz_a))
>      + '. SQLSTATE: ' +SQLSTATE
>      +  '. Allocated: ' + %char(%elem(csz_a : *alloc)));
> enddo;
> snd-msg ('Final Elements: ' + %char(%elem(csz_a)) + '. Allocated: '
>      +%char(%elem(csz_a : *alloc)));
> return;