RE: Accessing a file in memory

["Richard Jackson" <richardjackson@xxxxxxxxxxxxxxxxxx>]

We can disagree but I am fairly happy with my earlier statements :)

The object is pinned.  "Pinned" doesn't mean that none of it can page.  If I
recall correctly, set access state is a request or a suggestion, it is not a
suicide pact.  Pages brought into memory via set access state are not E=R or
non-pageable virtual memory, they are backed by disk - they can page.  Doug
Anderson once accused me in open forum of writing code that used set access
state to pin something into the machine pool.  Since it wasn't true, I
denied it but that was sufficient cause for me to spend some effort to
understand the instruction in some detail.  At that time, I did use set
access state to rip through the WCBT in large pieces but I double-buffered
the requests and I paged in the users pool.  After some hard lessons, I was
extremely carefully about that.  I had a WRKACTJOB command running on the
system 38 that, after the first execution, finished scanning and sorting the
jobs and painted the screen before you could get your finger off the refresh
key.

I don't know about the new extents.  I never thought to ask.

I was assured that objects brought in using SETOBJACC were paged into the
same pool.  It may be possible for me to find the person who told me this.
I will have to think about that.

Objects faulted into a public pool are subject to rogue jobs with large
memory demands.  As they keep hammering on SMPO for new free page frames,
pieces of my favorite object are lost.  That is one reason for using
SETOBJACC.

For a demonstration of this effect, I suggest that you execute a SQL
statement in the form "select ... from x where ... union select ... from Y
where ... order by ..." When X and Y are large and the where clauses cannot
use an index - a table scan is required.  This occurs (at least) when the
where clause is in the form fld1 like "%aspirin%".  The best demonstration
will occur when files X and Y have a total of more than 5 million rows and
the result set is a majority or all of the rows - under these circumstances,
the demonstration is very satisfactory - it will probably flush a 2 GB
memory pool.  I observed several concurrent statements wreaking havoc on a
busy 10 GB memory pool.  If three or more of these statements run at once on
a large system, you may see the non-database faults per second exceed 1,200
in that pool.  I believe that, in this case, this value was
disk-subsystem-limited.

Tuning decisions (or the absence thereof) may place jobs with different
memory usage patterns in the same pool.  Since one key use for SETOBJACC is
to accelerate batch jobs, normally-paged-in data may fall victim to
voracious interactive jobs flowing through the same pool.  Dynamic priority
adjustment addresses how the object is dispatched, I do not think that it
effects SM page fault decisions.

I agree that SETOBJACC is best for objects that are not changing.  I haven't
thought through the write fault effects.  I like this command for
very-frequently-referenced read-mostly data used by many jobs.  As a
performance consultant, I would suggest that a customer purchase the memory
to run it and justify that suggestion with numbers.

Comments?  War Stories?

Richard Jackson
mailto:richardjackson@richardjackson.net
www.richardjacksonltd.com
Voice: 1 (303) 808-8058
Fax:   1 (303) 663-4325

-----Original Message-----
From: owner-midrange-l@midrange.com
[mailto:owner-midrange-l@midrange.com]On Behalf Of Simon Coulter
Sent: Tuesday, July 18, 2000 6:59 PM
To: MIDRANGE-L@midrange.com
Subject: Re: Accessing a file in memory


"
Hello Richard,

I don't think there is any point in using SETOBJACC if you can't give it
enough memory
to contain the object.

Your comments on "hot" and "cold" paging are correct as far as they go but
isn't that
what happens to an object anyway under normal paging conditions?  SETOBJACC
doesn't
really pin the object in main storage -- it just loads it into specific
storage.  The
difference is that pinned pages don't get paged.

When a job requests data that isn't currently in main storage that data is
loaded into
storage allocated to the subsystem in which the job is running.  If the data
is already
in main storage, even if it is in a different pool, it is simply USED from
there.

So if you load an object in to a sufficently large private pool it will stay
resident
and be accessible from all jobs.  If, as you suggest, the pool isn't large
enough then
SETOBJACC will bring the object in to the pool and replace earlier loaded
pages with
later pages.  You end up with a partially loaded object.

I'm not sure, but I suspect that activity on the loaded object that causes
more pages to
be brought into main storage will go to the job's pool not the SETOBJACC
pool.  Thus
defeating the exercise.

A similar problem occurs if the object has new space allocations, e.g.,
adding new
records.  The new data is paged into the job's pool, not the pool specified
on the
SETOBJACC command.

SETOBJACC is really only useful for frequently accessed data that is fairly
static and
that can be loaded entirely into main storage.

Regards,
Simon Coulter.

«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»
«» FlyByNight Software         AS/400 Technical Specialists       «»
«» Eclipse the competition - run your business on an IBM AS/400.  «»
«»                                                                «»
«» Phone: +61 3 9419 0175      Mobile: +61 0411 091 400           «»
«» Fax:   +61 3 9419 0175      mailto: shc@flybynight.com.au      «»
«»                                                                «»
«» Windoze should not be open at Warp speed.                      «»
«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»«»
//--- forwarded
letter -------------------------------------------------------
> X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0)
> Date: Tue, 18 Jul 2000 07:53:11 -0600
> From: "Richard Jackson" <richardjackson@richardjackson.net>
> To: RPG400-L@midrange.com
> Reply-To: RPG400-L@midrange.com
> Subject: Accessing a file in memory
> Importance: Normal

>
> Simon:
>
> Agreed except ...
>
> If the pool isn't big enough, the object will page normally in and out of
> the private pool.  Since the object paging isn't in response to
competition
> with other objects, the "hot" parts of the object will stay in memory and
> the "cold" parts will leave.  If access has distinct hot and cold
patterns,
> this will provide the best balance between access time and memory space.
If
> access is random - has no hot and cold pattern - a higher faulting rate
will
> occur but if it is the only object in the pool, you can't do any better.
>
> If multiple objects are pinned into the same private pool, this advantage
> goes away at a rate proportional to the _fault_ rate in the pool.
Remember,
> database doesn't normally fault to bring in pages, it uses bring, so
> database faults are bad.
>
> Experiment with pinning the mort important logical files into memory.  You
> could avoid all index page faults.  Indexes are usually smaller than the
> physical and access patterns for index pages are usually non-random.
>
> Richard Jackson

+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator:
david@midrange.com
+---

+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---