RE: Array efficiency was Dynamic Arrays

On Wed, 16 Oct 2002, Buck Calabro wrote:

> >With an array everything is in memory, from
> >what I understand and correct me if I am wrong,
>
> Unless your memory pool is outlandishly large and you have no competition
> for memory at all, you cannot guarantee that all elements of an array are in
> memory.  Some may be paged out, especially in a large array.  Contrariwise,
> if you have such an outlandish amount of memory, the entire disk file may
> well be paged in and you don't have to "go out to disk" to get the records.

[snip some good stuff]

> reference one of those elements, you reset the memory page life timer.  The
> system knows that this memory page is in use and will try hard not to swap
> it out.  If elements 5000-32000 are never referred to, the pages they are in
> will be given away, even though element 1 is referenced every cycle.  Do

[snippety snip]

> Turn that around and imagine reading a file repeatedly by key.  If you read
> the same 5 records over and over, the memory pages they live in will stay in
> memory because they are referred to often.  Despite the fact that I am doing

[snip again]

> That's why I tell people not to think of OS/400 as a PC with an accent.  A

But you have given a beautiful explanation of PC memory management!  Note
what Hans said, "Forgive me for nitpicking, but what you describe is
applicable to any modern operating system, not just one with single-level
store."  The AS/400 isn't different in this regard than any other modern
OS.  So arguing that accessing files should be preferred to arrays because
of the underlying OS is wrong.

I believe (note I said "I believe" so everything that follows really is a
matter of opinion here, this is how I like to do things) that arrays
should be used instead of temp files when the data being stored is local
to the program's work.  For example, suppose I'm writing a CGI program
that receives several variables as environment variables.  If I wanted to
store these variables in an easy to access way an array (or MODS or
struct) is the perfect solution.  A temporary file would be a poor choice
since it would require authority to create a file and update it.  My CGI
program may not have that authority.  Besides, why create something
external to the program (a temp file) for something that only the program
uses (and does not need to persist between invocations of the program)?

How is a more complex example.  Suppose I want to write a program that
finds all the files in a directory on the IFS.  I don't know how many
files there are going to be, and there could be alot.  I certainly don't
want this information to persist between invocations of the program since
the files are very likely to change in the meantime, probably even during
execution of my program.  Again, an array is the solution.  And not a
predefined array, either.  I want one that can grow and shrink as needed.
There is no need to use a temp file to do this.  A temp file just adds
another thing that can go wrong (most likely authority, but could be other
things, maybe level check, maybe it doesn't exist for some reason, etc.).

Both of these are cases where the data being stored is only needed for the
current execution of a single program which does not need to persist
between calls.  In these cases I feel that a work file is the wrong
solution.  Dynamic arrays (or MODS or structs) of arbitrary size are
needed.

James Rich