Re: Memory pool , object question -- MIDRANGE-L

Hi Paul

Thanks for the details - helps to understand better what's going on!

As I recall from my time testing at IBM, it is a real challenge to repeat conditions for the tests. Your matching tests do a good job of isolating the effect of SETOBJACC on using matching records, I think.

Just for fun it would be interesting to look at WRKSYSSTS and see page faults - I hope I remember right here, but I think that, with the full file in memory, those should be really low. Without it in memory, I'd expect a higher rate, if the access is truly random, since the pages requested would require true disk I/O, not just logical I/O - and that reminds me that DSPFD could have interesting values along those lines. Also WRKDSKSTS, again, IIRC, as to page size and number of pages.

Not like you don't have enough to do, right!!!

Cheers
Vern

On 4/9/2018 8:42 PM, Steinmetz, Paul wrote:

Vern,

SETOBJACC was used for both tests.
The SETOBJACC was on SHRPOOL1, (my batch pool) no other batch jobs were running during this period.

I re-ran Input primary master file, 1,000,000 records, input secondary transaction file, 400,000,000 records, using matching records.
This time WITHOUT SETOBJACC, 20 minutes, compared to 15 minutes with SETOBJACC.

My conclusions thus far.
SETOBJACC adds slight improvement, not what was expected.
If file blocking is used, adds a greater performance improvement, if blocking is possible.
Blocking is not always possible.
https://www.ibm.com/support/knowledgecenter/en/SS4QVT_8.5.1/com.ibm.etools.iseries.langref.doc/evfrilsh450.htm

I tried adding blocking to my other test, but not possible due to the reade.
>From the knowledge base doc - (If any READE or READPE operations are used, no record blocking will occur for the input file)
Input primary master file, 1,00,000 records, chain, reade to transaction file, 400,000,000 records.
FCUMST IP E K disk
FARNRSQ03 IF E K DISK BLOCK(*YES)
>From the compiler.
*RNF7092 10 3 000400 BLOCK(*YES) is specified for file ARNRSQ03 but record blocking will not be done.

Further conclusion.
A new P9 with 2 or 4 times the current 256gb memory will have little impact with improving performance for longer running batch jobs from a memory/IO viewpoint.
No current memory issues, but I was hoping to keep some files in memory to improve performance.
More analysis and testing may be needed.

IF a batch application with NO blocking can be changed/modified to include blocking, this will result in the larger impact on performance.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Vernon Hamberg
Sent: Monday, April 09, 2018 6:23 PM
To: Midrange Systems Technical Discussion
Subject: Re: Memory pool , object question

Did you use SETOBJACC for both tests? If so, this isn't a test of SETOBJACC, seems to me.

If you ran the first one without SETOBJACC, then the 2nd one with SETOBJACC, that's not a fair test, either.

Did you put the transaction file into a fixed size pool with no jobs in it? Or shared pool with size settings that prevent it changing and no jobs in it? Those are kinda basic principles of using SETOBJACC, according to the knowledge base I think I'd posted here - or someone else did.

When I was contracting at IBM and testing the new query enging (what became known as SQE), we used CLRPOOL and SETOBJACC quite a bit - CLRPOOL was important to establish a clean environment. We were also usually running in our own partition out of 3 on a 12-processor system, as I recall - this was in 2001.

Now for RLA it might not have been necessary, but for SQL we ran the test at least 3 times, to establish the statement cache or some such.

Anyhow, I hope you are following a good rule when testing - change exactly 1 thing at a time a test the living daylights out of it.

Regards
Vern

On 4/8/2018 5:31 PM, Steinmetz, Paul wrote:

Jim,

The intention is to reduce run time by reducing I/O by using memory.

I did two more tests with SETOBJACC.

1) Input primary master file, 1,00,000 records, chain, reade to transaction file, 400,000,000 records.
65 minutes.

2) Input primary master file, 1,000,000 records, input secondary transaction file, 400,000,000 records, using matching records.
15 minutes.

Using the RPG cycle with matching records, and SETOBJACC has achieved the best results.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of
Jim Oberholtzer
Sent: Sunday, April 08, 2018 9:23 AM
To: Midrange Systems Technical Discussion
Subject: Re: Memory pool , object question

Paul,

Once the program is called it's in memory so the test is not valid.
All you are testing is I/O, not memory usage. Pinning the program in
memory is almost completely worthless in theses tests

Jim Oberholtzer
Agile Technology Architects

On Apr 7, 2018, at 3:01 PM, Steinmetz, Paul <PSteinmetz@xxxxxxxxxx> wrote:

Completed the next test.
Chain using record number, starting with last record, 400,000,000, ending with the first record, 1.

90 minutes.

With SETOBJACC
SETOBJACC OBJ(PAULS/CCEKCPP) OBJTYPE(*FILE) POOL(*SHRPOOL1) 46877104K
of CCEKCPP brought to pool with 10732K unused.
72 minutes, savings of 18 minutes.

I expected a larger savings.
This was on a Production LPAR, somewhat idle.
Any thoughts from the group?

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Charles Wilt
Sent: Friday, April 06, 2018 5:57 PM
To: Midrange Systems Technical Discussion
Subject: Re: Memory pool , object question

An even that's probably not really a valid test...

Likely that many of the blocks will remain in memory...unless you're testing on a box with a production workload so that data is likly to be paged out...

Charles

On Fri, Apr 6, 2018 at 1:53 PM, Steinmetz, Paul
<PSteinmetz@xxxxxxxxxx>
wrote:

Charles,

That is my next test, work in progress.
RPGLE.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Charles Wilt
Sent: Friday, April 06, 2018 3:50 PM
To: Midrange Systems Technical Discussion
Subject: Re: Memory pool , object question

Not really a valid test...

Assuming RPG RLA with an input file...RPG will buffer data..

If using SQL, the Db will asynchronously bring in the data...

Change your program to do 400M random reads.

Charles

On Fri, Apr 6, 2018 at 1:32 PM, Steinmetz, Paul
<PSteinmetz@xxxxxxxxxx>
wrote:

Jim,

Do you have any time savings of your use of SETOBJACC?

I created a 400,000,000 record test file.
Created a simple test program, read the file, beginning to end.
14 minutes to read all 400,000,000 million records.

SETOBJACC OBJ(PAULS/CCEKCPP) OBJTYPE(*FILE) POOL(*SHRPOOL1)
46877104K of CCEKCPP brought to pool with 25441484K unused.

Reran the same test program.
11 minutes to read 400,000,000 million records.

SETOBJACC OBJ(PAULS/CCEKCPP) OBJTYPE(*FILE) POOL(*PURGE) Member
CCEKCPP purged from main storage.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Jim Oberholtzer
Sent: Friday, April 06, 2018 1:59 PM
To: 'Midrange Systems Technical Discussion'
Subject: RE: Memory pool , object question

SETOBJACC has some limitations as well.

IF the object is truly read only such as a program or data area it
will stay
in memory. But if the object is updated, such as a file or data queue,
then the system will force it to DASD at some point and it will
fall out of that pinned memory.

I will always use a shared pool like *SHRPOOL60 and start working
back from there since those are really rarely used. No work in
those pools except the objects that are pinned in there.

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf
Of Vernon Hamberg
Sent: Friday, April 06, 2018 12:51 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
Subject: Re: Memory pool , object question

Paul

I would not put anything into *BASE using SETOBJACC - the article I
cited says these should go into a pool where there is no work, a
fixed pool, that will not be affected by any optimization - you want it static.

It is not just data that is paged in and out of a pool - it is also
*PGM objects - and potentially almost anything else jobs work with.
*BASE would be the place where almost everything will get paged out
of main memory eventually and maybe not before too long a time.

Also be sure not to over-manage this stuff - see that article,
again, for what makes good candidates for this technique.

Cheers
Vern

On 4/6/2018 10:18 AM, Steinmetz, Paul wrote:
-snip-
I feel I can improve the performance of some long running batch jobs.
Some

of these run multiple times per day.

What's the solution?
SETOBJACC, Keep in memory (KEEPINMEM), combination of both.

I'm considering the use of SETOBJACC, especially once the P9
arrives with

ample memory.

SETOBJACC needs to point to a storage pool.
The objects being considered for SETOBJACC might be used by both
interactive (*INTERACTIVE) and batch (*SHRPOOL1) My thoughts were
run

everything out of *BASE, SETOBJACC would point to *BASE.
-snip-
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
https://archive.midrange.com/

midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link:
http://amzn.to/2dEadiD

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
https://archive.midrange.com/

midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our
affiliate
link: http://amzn.to/2dEadiD

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: http://amzn.to/2dEadiD

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: http://amzn.to/2dEadiD