System crash w/scratch reload - what fun

Last Friday, I had a new experience with my iSeries 270.

The system operator came in and 3:30pm and said her screen (twinax 3180)
"looked just like it does during an IPL"  I assumed the display had died
because I've been waiting for that 3180 to die for years now.  Lo and
behold, none of my session or anyone else worked either.

The system had crashed.  To make a long story short it was the system
backplane.  There were none in town (Ft Wayne, IN) so one had to be
brought in from Chicago.

Sent everyone home and I met the CE (Rob: it was Dave Weber) Saturday
morning here at 7am.  After replacing the backplane, the system would
not IPL because the power supply in the expansion tower was toast.
After the CE got one from the parts depot here in town, the system still
would not IPL, even after repeated tries.  The cache controller was
reporting 51,000+ sectors with bad data (we have 5 physical disk drives,
RAIDed).

IBM Rochester suggested a scratch install but thought a slip install of
OS/400 might work.  So the CE opened a support call with software.  They
immediately said that was not a good idea.  It might not work at all,
and we could be putting out fires for months.  So we did a scratch
install, which I had never done before.

If anyone knows any gotchas to watch for, let me know.

I'm still taking a scheduled 1/2 day vacation this afternoon, so I may
not respond to any replies soon.

--------

In case it happens to you, here is a time line with approximate times.

Iseries 270, 3570 tape drive.

2:55pm  IPL from tape, install LIC, initialize load source
4:00pm  Disk config
4:30pm  OS/400 install
4:50pm  RSTLIB *IBM
5:05pm  RSTLIB *IBM again after QVFYOBJRST set to "1"
5:20pm  RSTUSRPRF
5:32pm  RSTCFG
5:42pm  RSTLIB *ALLUSR
6:35pm  RSTLIB QUSRSYS (not sure why we had to do this again . . .)
6:40pm  RCLDLO & RSTDLO
6:47pm  RST all other IFS objects (over 20,000 objects!)
7:32pm  RSTAUT
7:45pm  IPL
8:15pm  Go home
8:30pm  Drank 2 beers

-----------

Things I learned so far:

1) Man, am I glad we replaced that 8mm tape drive a few years back!

2) Man, am I glad I'm a stickler on backups.

3) I'm going to change the order I back things up so they are on the
tape in the order needed for a complete reload.

4) It appears the Advanced Job Scheduler may not be running, I'll have
to look into that.

5) The recovery manual says I will have to reinstall 5722-AC3 Crypto
Access Provider 128-bit for AS/400.  Software support has no idea why
that might be.  I'll have to look into that.

6) Might have to enter license info again - will have to look into that.

7) Power down all devices during the config restore or devices already
auto configged won't restore.

8) Man, am I glad it happened on a Friday afternoon and not some other
time of the week.

--
Jeff Crosby
Dilgard Frozen Foods, Inc.
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531

The opinions expressed are my own and not necessarily
the opinion of my company.  Unless I say so.