Runaway QHST

[Larry Bolhuis <lbolhuis@xxxxxxxxxx>]

Here's a strange one to watch for.

Customer calls wanting disk moved around on a 520 LPAR. They have gone 
from 60(ish) percent full to 90(ish) in the last few weeks and dont' 
know why. I get there this morning and am also told 'your backup (I 
wrote it for them) also failed the last two Wednesdays can you check 
that too?'.  Sure, so I hit DSPLOG starting at 2 AM  After 10 minutes 
still input inhibit. Hmmmm. Sysop tells me he quite using DSPLOG because 
it takes *WAYTOOLONG. Hmmmmm (again).  WRKOBJPDM QSYS/QHST* blows up 
with 'too many objects'. Hmmmmmmm!   DSPOBJD of QHST* and a query shows 
187,000 of them totaling 150+GB. Well now there's the disk space they've 
been missing!  Oldest file is 30 days old and matches CHGCLNUP values. 
Whysomuch? Turns out they had a damaged device description which the 
system was trying to vary on, that failed with damage, then it tried to 
delete it but the damage made the system think it was already gone, then 
it tried to make a new one which failed with 'already there' then it 
tried to vary it on which f....  This it did right around 1,500 times 
per second, creating a new QHST* file every 2 or 3 seconds. For about 
three weeks.  The backup failure turned out to be SAVSYS failing with 
too much stuff in QSYS!

 - Larry

ps: They thought the system had been running a tad sluggish as well. :-)