Re: The dreaded system full, now what.

Jack,

One of the biggest challenges when storage is maxed is getting signed on. The system will sometimes kill the system dispatch job when storage is overflowing (this is the job that assigns a table entry for new jobs). Putting 2+2 together, you cannot signon, including the console. When this job dies, the only way to restart it is with an IPL. In order to keep from having a "hard down", I will use existing FTP host jobs to log in, remove known areas where I can recoup space, then I will end a known job all while having my console session trying to connect. I am doing all this with FTP remote commands. With one job down, I can get the open table entry for my interactive console session. Now you can examine the system, remove stuff, kill rogue jobs, and prepare for a somewhat normal IPL. (Don't signoff once you get in!) Before you IPL its important to remember IBM might need a main storage dump. Once you are quiesced as good as you can, force the IPL and MSD
from the panel.(I would recommend Rochester support for all that.)

A couple of more recommendations:

1) Review your system value AUX storage lower limit, and the sysval AUX storage lower limit action. I would recommend at least sending a message and ensuring that Operators are monitoring for it.
2) Get some disk management software if this is an "important" system that has perpetual space issues. RTVDSKINF is ok, but there are much better 3rd party solutions.
3) If you do procure some disk management software, make sure they have a real-time monitor and corresponding alert. (ie. utilization grew 10% in one hour, something is wrong, alert Ops) I say this because detailed space analysis will always take time, usually 3+ hours for larger systems. So real-time monitor ---> emergency, full disk analysis ---> non-emergency, long term analysis/capacity planning, trending.



________________________________
From: Jack Kingsley <iseriesflorida@xxxxxxxxx>
To: Midrange Systems Technical Discussion <MIDRANGE-L@xxxxxxxxxxxx>
Sent: Mon, February 8, 2010 8:05:58 AM
Subject: The dreaded system full, now what.

I recently had a system that had filled up it's disk storage due to a run
away job and an errant qsysprt printer file set to nomax on records, the
system was basically locked up. Does anyone have specific instructions on
what they do when a problem such as this comes about. I did a normal IPL
and did not know what I had at the time until the IPL had completed and I
was able to gain access to a signon screen. Once signed on I did a quick
wrksyssts and could see what was going on. At that time I ended all
subsystems to get the box in a restricted state in order to diagnose what
was going on. I happened to stumble on the spooled file that had the
dreaded +++++ for pages. Once I deleted the spooled file the disk storage
went down, but then I get bit by the qrclsplstg system value being set to
*NONE, this caused a massive locking problem on members in QSPL library and
jobs that could not then run due to qpjoblog locks. If I had not stumbled
on the spooled file I am not sure how long it would have taken me to figure
out the(where has mydisk space gone) situation. I don't believe there is a
command to quickly diagnose previous rtvdskinf(s) either and or a command to
show which reports take up the most disk storage(not at least quckly )
anyways.