× The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.



Hi, Kurt:

Loading the entire file into one or more arrays will incur the overhead of loading the entire file for each job that needs to search the array, and this increases the virtual storage used by each and every job that needs to access this file.

So, before you decide to load all records of this file into an array and search using a binary search, consider the following alternatives:

#1. issue SETOBJACC to load the entire *FILE into a dedicated memory pool; sized based on the size of the file; this will speed up access for all jobs using this file, without any coding changes to the programs using the *FILE.

#2. create a "server" job that loads the array, and communicates with requester jobs via data queues. Instead of loading every record into the array, start out with an empty array, or load just one record. When a client program sends a message to the request data queue, requesting a "look-up", the server job first looks in the local "cache" (the array), and if not found, it then chains out to the file, and adds that record to the "cache" array, so any subsequent look-ups for that record will be faster, before returning the results via a results data queue (one per client). Typically each client sends the name of its results data queue as part of the request message. The server job will need to maintain a counter of how many records are in the array, for use on the %lookup BIF.

#3. create a user index, and load the data into the user index, and use the MI builtin function _FNDINXEN to do the look-up. This should be faster than database I/O, but you will have to incur the overhead to initially load the user index. Note that you can do this just once, and use an insert, update and delete trigger attached to the file to maintain the *USRIDX whenever the corresponding file is changed. All users or jobs can then share this one *USRIDX object.

_Performance trade-offs_
On average, how many records or rows of this *FILE would be accessed by any given job or interactive user session, in a given day? What percentage of the total file?

Hope that helps,

Mark S. Waterbury

> On 11/10/2011 12:09 PM, Kurt Anderson wrote:
I'm trying to reduce some program I/O for program processing millions of records. I've identified a file that I feel could be loaded into an array instead of having the program chain out for every record. Usually this file is between 100-1000 records. However one of our clients has 20,000 records in the file. So before making the change, I decided to do a little reading on %lookup, to make sure I'd be using it correctly.
http://www.ibmsystemsmag.com/ibmi/developer/rpg/iSeries-EXTRA--Look-Before-You--Lookup/

It turns out that I had misunderstood how to trigger the binary search with a %lookup. I thought sorting the array was enough, however to get the binary search one needs to explicitly specify Ascend or Descend on the array.

So I went to add Ascend to a data structure array, and it won't let me:
D ds_Cust DS Qualified Inz Dim( 50000 ) Ascend
D Cust 3p 0
D Cycle 1a
Error: RNF3501E Keyword is not allowed for a data-structure definition; keyword is ignored.

I tried adding the Ascend keyword to the subfield that I use in the lookup, but it doesn't like that either (I wasn't expecting this to work, but at this point I'm grasping at straws).

I suppose I could make these two fields their own arrays. Though I'm not really a fan of that. I love using data structures to group like-data together.

Is there any way to get the new data structure array lookups to use a binary search?

Kurt Anderson
Sr. Programmer/Analyst
CustomCall Data Systems


As an Amazon Associate we earn from qualifying purchases.

This thread ...

Follow-Ups:
Replies:

Follow On AppleNews
Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.