Re: %lookup on a Data Structure Array

["Mark S. Waterbury" <mark.s.waterbury@xxxxxxxxxxxxx>]

Hi, Kurt:

Loading the entire file into one or more arrays will incur the overhead 
of loading the entire file for each job that needs to search the array, 
and this increases the virtual storage used by each and every job that 
needs to access this file.

So, before you decide to load all records of this file into an array and 
search using a binary search, consider the following alternatives:

#1. issue SETOBJACC  to load the entire *FILE into a dedicated memory 
pool; sized based on the size of the file; this will speed up access for 
all jobs using this file, without any coding changes to the programs 
using the *FILE.

#2. create a "server" job that loads the array, and communicates with 
requester jobs via data queues.  Instead of loading every record into 
the array, start out with an empty array, or load just one record. When 
a client program sends a message to the request data queue, requesting a 
"look-up", the server job first looks in the local "cache" (the array), 
and if not found, it then chains out to the file, and adds that record 
to the "cache" array, so any subsequent look-ups for that record will be 
faster, before returning the results via a results data queue (one per 
client).  Typically each client sends the name of its results data queue 
as part of the request message. The server job will need to maintain a 
counter of how many records are in the array, for use on the %lookup BIF.

#3. create a user index, and load the data into the user index, and use 
the MI builtin function _FNDINXEN to do the look-up. This should be 
faster than database I/O, but you will have to incur the overhead to 
initially load the user index.  Note that you can do this just once, and 
use an insert, update and delete trigger attached to the file to 
maintain the *USRIDX whenever the corresponding file is changed.  All 
users or jobs can then share this one *USRIDX object.

_Performance trade-offs_
On average, how many records or rows of this *FILE would be accessed by 
any given job or interactive user session, in a given day? What 
percentage of the total file?

Hope that helps,

Mark S. Waterbury

> On 11/10/2011 12:09 PM, Kurt Anderson wrote:
> I'm trying to reduce some program I/O for program processing millions of records.  I've identified a file that I feel could be loaded into an array instead of having the program chain out for every record.  Usually this file is between 100-1000 records.  However one of our clients has 20,000 records in the file.  So before making the change, I decided to do a little reading on %lookup, to make sure I'd be using it correctly.
> http://www.ibmsystemsmag.com/ibmi/developer/rpg/iSeries-EXTRA--Look-Before-You--Lookup/
>
> It turns out that I had misunderstood how to trigger the binary search with a %lookup.  I thought sorting the array was enough, however to get the binary search one needs to explicitly specify Ascend or Descend on the array.
>
> So I went to add Ascend to a data structure array, and it won't let me:
>       D ds_Cust      DS                  Qualified Inz Dim( 50000 ) Ascend
>       D  Cust                         3p 0
>       D  Cycle                       1a
> Error:  RNF3501E Keyword is not allowed for a data-structure definition; keyword is ignored.
>
> I tried adding the Ascend keyword to the subfield that I use in the lookup, but it doesn't like that either (I wasn't expecting this to work, but at this point I'm grasping at straws).
>
> I suppose I could make these two fields their own arrays.  Though I'm not really a fan of that.  I love using data structures to group like-data together.
>
> Is there any way to get the new data structure array lookups to use a binary search?
>
> Kurt Anderson
> Sr. Programmer/Analyst
> CustomCall Data Systems