Re: IBM i POWER disk failure: our experience

On 3/8/11 7:50 AM, elehti@xxxxxxxxxxxxxxxxxx wrote:

<<SNIP>>
Approximately 12 Machine Check messages go to QSYSOPR at different
intervals.

<ed: reformat> CPF0937 03/05/11 22:50:49 "Machine check not
recoverable. Error code X'0000'. Cause . . . . . : The system
operation is being continued. The licensed internal code may have
a problem."
====================
Also, saving to tape produces message:
<ed: reformat> MCH3402 03/08/11 07:53:19.6
setspaceptrfromptr 000048 QDBDLTFI ?inst?

Seems more likely the msgCPF0937 would have been diagnosed during saves, and that the msgMCH3402 T/QDBDLTFI ?inst? occurred during DLTF. One would generally hope\expect that DLTF is not being run as part of a SAVLIB or SAVOBJ [except possibly to complete a prior, interrupted DLTF operation, but that would cause the save difficulties].?

=====================
IBM tech scans the 12 disk units on our 457GB system and discovers
numerous pages are bad.

</begin IBM explanation>
scanned all disk units this time and found 274 bad pages on unit 10.
Of those 110 were in free space and got cleaned up by the scan tool.
The remaining 174 bad pages are affecting the following four objects

Database file member PLANNING/F55NOTE2 INOTE
Service program QSYS/QJVAJNISYS
Database file member QSPL/Q04079N036Q651950294
Database file member MODEL/F3002 F3002
</end IBM explanation>

I delete bad objects and restore from good tape backups.

Good. Only delete object is recovery from damage. Note that the means to effect "delete object" for a member is RMVM; albeit possibly problematic if the next or prior member in a chain is also damaged, so DLTF may be the better choice sometimes. However, hopefully for the Q04079N036 file MBR(Q651950294) the RMVM was used, else there are likely many incidents to come, for problems with DSPSPLF [and other options from WRKSPLF]; as a side effect of the means of recovery, not as side effect of the original problem.

</Begin IBM explanation>
The damage was due to a hardware problem and it not specifically
related to single-level storage. Perhaps it could be answered better
by your hardware rep, but from my experience, when a disk goes bad,
it can put "garbage" on the SCSI bus which typically has multiple
disks attached, so data on other drives using that scsi bus can be
affected. Hardware problems can also happen at the cache, IOA, or
even IOP level. This is why when I work a call like this, I scan all
of the units under the storage controller for damage.

The MCH3402 messages you received look like the results of cleaning
up the addresses where the first page was bad. So now, instead of
getting a machine check error, you got better messages that
identified what object was affected.

Again this seems more likely to have been seen logged during "delete object" as recovery; DLTF to be specific. Since DLTF would have just ignored the bad page condition, a prior DLTF would have seen the error for the bad page and continue, but there would not have been any opportunity to see any other errors later since the *FILE would no longer exist to allow DLTF to be invoked. As described, the condition would seem likely to have occurred as a result of having chosen DLTF to effect the "delete object" as part of recovery, and the previous cleanup would have caused the DLTF to exhibit the MCH3402 instead of diagnosing a bad page as an unrecoverable machine check.

Tomorrow, after the IPL tonight, I'd like to get another remote and
rescan units 5 & 10 to make sure all of the bad pages have been
cleaned up. Then I recommend scheduling a full system save to check
for any further objects that may have pieces missing from those
addresses I removed (first page was bad).
</end IBM tech message >

Best of luck.

If any removed addresses had been a *FILE for which composite pieces like *MEM had thus been orphaned, the RCLSTG *ALL would place the *MEM under a manufactured *FILE from which data recovery could be attempted; an [especially a composite] object should never be "moved" from the QRCL, only exist for its data, attributes, authorities, and ?? used to apply to the equivalent object that was created or restored into the expected library.

Regards, Chuck