RE: Ready to scrap an AS/400

Rob:

We used AS/400 performance tools for both benchmarks.  We didn't need
anything more sophisticated than that.

These were real tests backed up by field experience.  I designed and managed
the RPG tests.  The same concept was and is used for the client server tests
but the driver tools are different and the tests and test techniques have
matured.

In both cases, we asked the field support and field consulting people to
define application scripts and realistic database content that resembled a
"typical customer" for financials (AR, AP, and GL), manufacturing (BOM, shop
floor order processing, forecasting, and requirements planning),
distribution (inventory, sales, and purchasing).  The RPG tests were
executed using PETE.  The client-server testing was done with Mercury
WinRunner to run the scripts and a custom tool to capture the SQL.  Then
Mercury LoadRunner and custom tools to hurl the collected SQL at the server
in controlled ways.  A test was Financials or Manufacturing or Distribution
or Everything.  Testing was intended to represent several users entering
valid transactions where their typing came from random but valid sets of
data.  The tests are executed with varying numbers of users.  The latest
client server tests run with hundreds or thousands of simulated users.

AS/400 performance tools are run during the tests.  As the number of
simulated users increases, system tuning is observed and modified to get the
best from the system under test.  Systems that run out of resources like
memory are upgraded until they can take the strain and run at 80 percent or
more CPU utilizatoin.  A graph is drawn from test results showing the CPU
utilization and the number of users.  Suppose that a given system has 5,000
CPW and, in our test, 1,000 users caused the system to run at 60 percent CPU
utilization.  We would conclude that each user required 3 CPW computed as
follows: 5,000 CPW times 0.6 CPU utilization equals 3,000 CPW used.  3,000
CPW divided by 1,000 users equals 3 CPW per user.  If the same CPU result
were achieved with 75 users, we would conclude that the application required
40 CPW per user.

If we had results for 100, 250, 500 and 750 users giving about the same CPW
per user answer, we would be pretty confident.  We did and we were.

We also had to go out into the field and work with customers who were having
difficulties - rather like Mr. Wesloskie.  We discovered that our tests were
usually on the low end of realistic.  So, when the test results were used
for CPU sizing, we adjusted them up.

Item 2 comes from reading a lot of code and a lot of books.

Item 3 comes from trying to make systems perform.  AS/400 performance issue
handling is one of my professional activities and I have quite a bit of
practice in this area.

Item 4 comes from reading and talking to people about the major ERP
applications out there.

Item 5 is speculation but based on running side by side tests with the
scripts as described above, comparing the results, then trying to understand
why the 400 is slower.  Also, from long discussions with other AS/400
techies as we speculated about reasons.  We don't know if this is the reason
and there are damn few tools to test this one but I cannot come up with
another reason that accounts for the difference.  BTW, this is not my idea,
the credit for bringing it up the first time goes to a guy named Randy West.

Item 6 results from the experience of trying to trouble-shoot SQL via ODBC
on the AS/400 and the general problems of trying to figure stuff out.  I
spent 3 weeks in Helsinki last winter working on a problem.  During one
week, one of the Rochester microcode-level performance guys was with me.  I
learned a lot from him and I am grateful for what I could pick up but there
is a lot more to know, IBM doesn't have enough of their own guys, and IBM
has problems training people who aren't IBM to do this kind of work.

BTW, even using display alter dump, a brilliant mind, and the right space
offsets, we still couldn't determine which person was sending in the really
ugly SQL statements.  Once we had a pattern of statements, we sent out a
system-wide message and asked if anyone recognized the query.  A person from
a small town on the Gulf of Bothnia called in and that was the beginning of
the solution.

Richard Jackson
mailto:richardjackson@richardjackson.net
http://www.richardjacksonltd.com
Voice: 1 303 808 8058



> -----Original Message-----
> From: owner-midrange-l@midrange.com
> [mailto:owner-midrange-l@midrange.com]On Behalf Of Rob Berendt
> Sent: Monday, July 24, 2000 11:56 AM
> To: MIDRANGE-L@midrange.com
> Subject: RE: Ready to scrap an AS/400
>
>
> My reply was intended to be sarcastic.  That's just my way.
> I thought the 'find a new job' was the clue.  Interesting points
> you bring up.  What tools are you using for performance measuring?
> Not just WRKACTJOB I hope.
>
>
>
>
>
> richardjackson@richardjackson.net on 07/24/2000 12:14:13 PM
> Please respond to MIDRANGE-L@midrange.com@Internet
> To:   MIDRANGE-L@midrange.com@Internet
> cc:
> Fax to:
> Subject:      RE: Ready to scrap an AS/400
>
> Rob, I agree with some of what you are saying.
>
> About five years ago, I benchmarked a mainstream AS/400 software product.
> At that time, that product required an average of 0.6 CPW per user.
>
> Over the last several years, that vendor re-wrote their entire suite.  The
> current release of the new product uses between 3.0 and 40 CPW per user.
>
> In both cases, the numbers are for interactive work, no batch is included.
> As far as I can tell, the major differences are:
>
> 1. SQL database IO requires about 10 times more memory than an RPG program
> and between 5 and 100 times more CPU cycles.
>
> 2. Client Server applications are written in object-oriented style.  OO is
> better for highly complex applications for all the reasons that
> you already
> know.  However, one of those reasons is: less shared memory access.
> Translated into RPG-speak, that means less use of programmer-written cache
> routines.  And that means, if I need access to an object, I just
> go and get
> it.  No matter that the last guy went and got it, I will get it again.  CS
> apps tend to do more IO than RPG apps for the same amount of
> end-user work.
>
> 3. Client Server applications usually use commitment control.  Most AS/400
> shops don't know how to buy disk and controllers to serve
> commitment control
> and they don't know how to tune for that level of journaling and the extra
> commit-time CPU hit is usually ignored during planning.
>
> 4. Instead of custom-written, close-to-the-metal report and file-update
> programs, most CS applications use general purpose reporting tools.  Where
> the close-to-the-metal programs performed a lot of print and file update
> work very efficiently, general purpose reporting tools are relatively
> inefficient.  It requires a new mindset to figure out how to do some mass
> operations that used to be done in batch programs.
>
> 5. Most client server applications are written in C.  C programmers use a
> lot of pointers.  On Unix and NT machines, C pointers are very efficient -
> modifying a pointer is a very light-weight operation.  On the AS/400,
> pointer manipulation requires considerably more CPU cycles.  The obvious
> throughput difference makes the AS/400 look bad compared to Unix and NT.
>
> 6. Performance monitoring has not kept up with client server applications.
> For example, have any of you ever used SQL over ODBC (Client
> Access Express)
> and tried to figure out which QZDASOINIT job belonged to you?  The
> higher-level tools are still focused on green-screen and RPG/CL batch jobs
> and there is no training or support for lower-level tools.  There are no
> tools (that I know of) to measure the time required to perform the pointer
> manipulation described in the paragraph 5.  Well, there is a tool called
> Flasher that might help but I can't find any documentation for it.
>
> Questions? Comments?  War stories?
>
> Richard Jackson
> mailto:richardjackson@richardjackson.net
> http://www.richardjacksonltd.com
> Voice: (303) 808 8058
>
> -----Original Message-----
> From: owner-midrange-l@midrange.com
> [mailto:owner-midrange-l@midrange.com]On Behalf Of Rob Berendt
> Sent: Monday, July 24, 2000 7:38 AM
> To: MIDRANGE-L@midrange.com
> Subject: RE: Ready to scrap an AS/400
>
>
> And scrap any other C/S system you're thinking of.  Stick only with
> green screen.  And when you have to find a new job because they replaced
> all of your hardware with NT just smile smugly because you know how badly
> they've screwed up.
>
>
>
>
>
> bvstone@taylorcorp.com on 07/21/2000 02:44:25 PM
> Please respond to MIDRANGE-L@midrange.com@Internet
> To:   MIDRANGE-L@midrange.com@Internet
> cc:
> Fax to:
> Subject:      RE: Ready to scrap an AS/400
>
> Daniel,
>
> Your problem should be with domino, not the AS/400.  It's a pig.
> I've hard
> IBM reccomends a minimum of a 720 with mucho disk arms and memory.
>
> Scrap Domino.  Not the AS/400.  And ask for your money back.  They sold it
> to you so you'd buy a bigger machine.
>
> Brad
>
> > -----Original Message-----
> > From: Daniel Wesloskie [mailto:dwesloskie@altatennis.org]
> > Sent: Friday, July 21, 2000 12:29 PM
> > To: MIDRANGE-L@midrange.com
> > Subject: Ready to scrap an AS/400
> >
> >
> > Right now, my perception IS reality!  I'm about ready to
> > scrap this piece
> > of junk AS/400 for something else.  It just plain can't do
> > the job!  We
> > have a 170 (V4R4M0)that is running Domino(5.0.4) and a voice
> > response system
> > (VRS) as
> > the main applications along with the backend applications.
> > So there isn't
> > much to the system and it still can't handle it.  Domino
> > locks up and causes
> > problems for the VRS as the http server consumes more of the
> > cpu %.  This
> > means
> > members who are calling in to the VRS are either getting
> > kicked out of the
> > system halfway through their process, or they can't even get
> > in.  It also
> > means
> > that anyone who is trying to pay for memberships get locked
> > out.  I can only
> > imagine what these people think, but I'm almost sure it's not
> > good.  Since
> > many
> > of the membership include people who would be influential in
> > the decisions
> > to
> > their own companies systems, I would ask, How is this for an
> > advertisement
> > for
> > the AS/400? or How long can my business afford to be down with AS/400?
> >
> > Ticked off with the AS/400.
> >
> > -----Original Message-----
> > From: owner-midrange-l@midrange.com
> > [mailto:owner-midrange-l@midrange.com]On Behalf Of Rob Berendt
> > Sent: Thursday, July 20, 2000 11:24 AM
> > To: MIDRANGE-L@midrange.com
> > Subject: RE: So if there is no Hardware specific Advertising
> > then???????
> >
> >
> > I used to have a boss - different sort of fellow - anyways he
> > used to quote
> > John Lennon.
> > "Perception is 99% of reality."
> >
> >
> >
> > Nina,
> >
> > You've hit the nail on the head here. It is not that the 400
> > *can't* do
> > these things. The problem is the *Perception* is that it
> > can't do these
> > things.
> >
> >
> > +---
> > | This is the Midrange System Mailing List!
> > | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> > | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> > | To unsubscribe from this list send email to
> > MIDRANGE-L-UNSUB@midrange.com.
> > | Questions should be directed to the list owner/operator:
> > david@midrange.com
> > +---
> >
> +---
> | This is the Midrange System Mailing List!
> | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> | To unsubscribe from this list send email to
> MIDRANGE-L-UNSUB@midrange.com.
> | Questions should be directed to the list owner/operator:
> david@midrange.com
> +---
>
>
> +---
> | This is the Midrange System Mailing List!
> | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> | To unsubscribe from this list send email to
> MIDRANGE-L-UNSUB@midrange.com.
> | Questions should be directed to the list owner/operator:
> david@midrange.com
> +---
>
> +---
> | This is the Midrange System Mailing List!
> | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> | To unsubscribe from this list send email to
> MIDRANGE-L-UNSUB@midrange.com.
> | Questions should be directed to the list owner/operator:
> david@midrange.com
> +---
>
>
> +---
> | This is the Midrange System Mailing List!
> | To submit a new message, send your mail to MIDRANGE-L@midrange.com.
> | To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
> | To unsubscribe from this list send email to
> MIDRANGE-L-UNSUB@midrange.com.
> | Questions should be directed to the list owner/operator:
> david@midrange.com
> +---

+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---