Domino Utilities and Server Performance Issues

Hello Everyone,

I have been experiencing Lotus Notes performance issues since Monday
morning.  At first we thought the problem to be a run away process ( Router
task on a server was running at 80-90% for about 35min and it wasn't doing
anything ).  Since then ( Monday at 10:35am ), CPU for the entire LPAR has
been averaging 30% and peaking at 65%.  Still, as of the time of this
email, we are seeing intermittent performance issues. And by that I mean,
everything seems to run normally then for some seemingly random reason, it
take upwards of 4 minutes just to open an email that does not have an
attachment or switch to a different folder view, etc.  We sniffed the
network and found no network issues occurring during these moments that
Lotus Notes runs slow.  These moments tend to last about 5-10 minutes and
all goes back to normal.  The performance problems to not seem to tie into
any server task or event taking place.

After eliminating CPU we checked Memory.  The iSeries guru at our
organization informed me that there didn't appear to be any memory issues
but did notice that 6 of our disks ( of 22 ) were being hit hard.  When he
did some further digging he noticed that ALL of the data for all 10 server
instances on our LPAR were all on 6 of the 22 disks that we have.  This was
not the case last week.  It would seem that the performance issues were
related to disk and he is performing a task to evenly distribute the data
among the 22 disks by usage.  Now you are all probably wondering, "How did
all the data end up on just 6 disks when it was spread across 22 last
week?"......  That, my friends, is the $100,000 question.  If anyone has
any ideas I would love to hear them.  Some background info.

Saturday night we applied PTF's to the LPAR.
We also ran Fixup on 3 of our server, and Updall -r  on 2 of the 3 servers
that Fixup was run on.

That is the extent of the changes that occurred this weekend.  I have a
directors meeting in the morning at which they will be asking me why Lotus
Notes is running slow.  I believe I have determined what was the problem
but not the root cause.  If anyone can send me idea before 9am tomorrow.
Feel free to ask me any questions, I will provide as much detail as I can.

Thanks,

Jose del Risco
Corporate Domino Administrator
Oshkosh Truck Corporation

(920) 235.9151  x 2741


Although this e-mail and any attachments are believed to be free of any virus 
or other defect which might affect any computer system, it is the 
responsibility of the recipient to check that it is virus-free and the sender 
accepts no responsibility or liability for any loss, injury, damage, cost or 
expense arising in any way from receipt or use thereof by the recipient.

The information contained in this electronic mail message is confidential 
information and intended only for the use of the individual or entity named 
above, and may be privileged.  If the reader of this message is not the 
intended recipient, you are hereby notified that any dissemination, 
distribution or copying of this communication is strictly prohibited.  If you 
have received this transmission in error, please  contact the sender 
immediately, delete this material from your computer and destroy all related 
paper media.  Please note that the documents transmitted are not intended to be 
binding until a hard copy has been manually signed by all parties.
Thank you.