Re: LIC re-install - Confused on the steps to take

Alas! No reply from Dr. Franken. Must still be struggling with the laptop.

Here is the latest:
Swinging the dead cat got IBM mobilized to send out a "bunch of parts" so hopefully those parts will arrive today. They were also going to arrange for me to get a beta of the new VIOS release scheduled to be released Friday.

With IBM finally beginning to move and based on a hunch that perhaps the broken mirrored pair was the source of the LIC loading problem, I managed to get the partition running with the "magic" sequence: Shut down the blade, pull and reinsert the DSM, bounce the MM and voila! I have SAS drives again. Once the partition was running, I allowed the mirrored pair to re-sync. Then I uploaded the I_BASE_01 CD and mounted it as a virtual drive and then did a "D" manual IPL. It seemed to run just about as fast as before but this time when the partition IPL'd it no longer had the PTF's I had seen before. Then I bounced to the "B" side which DID work and then I mounted the image catalog of the latest CUME and somehow the thing managed to keep running long enough to install, load and apply the PTF's. In fact, the partition has been running for 14 hours at the moment, a new record!

I'll see which of the part's is the "true" fix. Plus, with the new release of VIOS I'll be able to use the media tray again AND be able to use the external tape drive directly from IBM i which will be a real step up!

Some progress at last.

Pete Helgren

Pete Helgren wrote:

Dr. Franken! I was hoping you would weigh in.

Two minutes for a LIC load seemed kinda short to me as well. The fact that the "A" side still shows PTF's needing to be applied and the "B" side won't IPL made me suspicious that all was not well here. This may be a clue: I remember seeing a message that mirroring had been suspended on one of the pairs. I don't recall which one. Would a suspended mirror cause the LIC not to load? I didn't seeing any errors, just warning about the suspended mirror. After starting the mirroring ("resuming", I think it said) the blade didn't last long.

At the core of this is the "disappearing" SAS drives. I can usually get them to appear by first pulling the DSM (Storage Module) and then reinserting it. The AMM then indicates that it can't read the firmware level of the DSM so I have to bounce the MM and, at that point, all is well. I can start the blade and, in the IVM or at the VIOS command line I can see the SAS drives as available. From here, the cause of the "disappearance" becomes difficult to pin down. The blade itself, just running VIOS stays up and running about 98% of the time. But, the blade itself can run without having the SAS drives in the DSM available because it uses the two onboard HDD's for storage. The problem occurs in two ways:

1. Start the i partition. It will run for minutes, hours, days (rarely) without a known trigger as to what causes it to fail. I'll just notice that the AMM starts reporting errors from the blade and then when I go to the i, the console is unresponsive. I have suspected that a large amount of I/O is what is causing it to fail but I can't prove it. When I resumed the mirroring on the pair for instance, the blade was dead within minutes.

2. Back up files on VIOS to tape (external SAS LTO4 drive). This also involves the SAS Connectivity module (where I suspect the problem is). I was successful in getting the i to do a savsys to virtual optical media this weekend so there is now a file in VIOS I can back up to tape. When I launched the backup, VIOS went dead stick, along with the blade, within a minute or two. This happened both times I attempted to back up to tape.

So I think all fingers point to a bad SAS Conn Module but IBM thinks otherwise. They *know* there is an issue with the media tray and the i partition, but this thing fails whether the media tray is inserted or not. This box has been casters up for almost 4 weeks (yeah, I have had a couple of low availability weeks to work on it but I have been trying at least once a day to keep this thing running). So my plan is now to start swinging the dead cat around the IBM support desk and see if I can't get them to send out a tech to fix this thing. Unfortunately, I leave town tomorrow so I think I have another week of "try this, try that" before I can get someone to fix this.

Pete

lbolhuis@xxxxxxxxxx wrote:

Pete,

Sounds like fun... :-( I'm just reading this thread after a
catastrophic laptop failure this weekend. Luckily, though my primary backup
is on Frankie III which is currently in Rochester, I have a backup for my
backup!

So if I understand your blade crashed during PTFs. This FUBARed the link
loader on the "B" side, so that side is toast. IPL to "A" side is (ok
"was") good. You first re-installed the OS (oops) and while that didn't
help it shouldn't hurt. Then you IPLed "D" again from an RSE disk and
installed the LIC with option 1. (Which is the correct path to take.) .
You said it took only 2 minutes. THAT Makes me go Hmmm. Installing the LIC
isn't a long process but given the blade center has no cache and the drives
are mostly 10K units and there is VIOS in there, two minutes seems very
quick, almost as if it didn't go where you expected, or maybe anywhere. The
fact that a "B" mode IPL fails with the same error seems to add credence to
this theory.

The one thing I might do at this point is the "D" IPL - install the LIC,
then from there Install the OS again and let that complete. Use the OS
Disks you downloaded with your RSE slic. This will give you a complete
replacement of software from the bottom up putting both LIC and the OS at
the same PTF level. From there all PTF processing should go normally.

But there is that pesky "can't see the drives" problem at this point. So
you are now stuck with no drives? Does VIOS See the drives or are they gone
completely?

- Larry


Larry Bolhuis IBM Certified Advanced Technical Expert:
Vice President System i Solutions
Arbor Solutions, Inc. IBM Certified Systems Expert:
1345 Monroe NW Suite 259 System i Technical Design and Implementation
V6R1
Grand Rapids, MI 49505
(616) 451-2500
(616) 451-2571 - Fax
(616) 260-4746 - Cell



(Embedded image moved to file: pic04760.gif)

If you can read this, thank a teacher....and since it's in English, thank
a soldier.



Pete Helgren <Pete@xxxxxxxxxx> Sent by: To midrange-l-bounce Midrange Systems Technical s@xxxxxxxxxxxx Discussion <midrange-l@xxxxxxxxxxxx> cc 05/10/2009 08:36 PM Subject Re: LIC re-install - Confused on the steps to take Please respond to Midrange Systems Technical Discussion <midrange-l@midra nge.com>



It's RSE. And yes, the first time I made the mistake of loading the OS
instead of the LIC. However, my later post to Lukas, I outlined the
steps I took, the first being loading the BASE CD and then IPL'ing from
D, after a 2 minute or so load, the i bounced and I ended up with a
manual A side IPL option. I was waiting to find out what the next is
that I should take.

Meanwhile, the blade crashed, again, and I can't get the partition to
see the SAS drives, no matter what I do. It will be a long day on the
phone with IBM tomorrow. I hope they send me a new blade/SAS module.
This POJ is taking up too much of my time.

Pete


Pete Massiello wrote:

Pete,

I think you reinstalled the OS. You need to reinstall the

LIC. Do

a "D" mode IPL from the I_BASE Cd. Make sure you have a current one. In
fact, you are best to get the latest one at

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss

You should then load this onto the "A" side, which will wipe

out all

your PTFs. Then just IPL that side up, and then you can do your PTFs.

If

you installed a new version of LIC, then you won't have any PTF

processing

as you state below.

I would make sure my I_BASE cd is either 610 RSD or RSE

Pete

Pete Massiello
iTech Solutions
http://www.itechsol.com


-----Original Message-----
From: midrange-l-bounces@xxxxxxxxxxxx
[mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Pete Helgren
Sent: Sunday, May 10, 2009 4:49 PM
To: Midrange Systems Technical Discussion
Subject: LIC re-install - Confused on the steps to take

I am trying to snuff a B side IPL issue where I get a B600 0901 (link
loader failure) error. So i decided to reinstall LIC/OS.

The first attempt was just a manual IPL to the A side (the side that
runs). I took the option to install the operating system and after an
hour or so had completed the install of the OS. I then moved on to
install the latest CUME taking the option to install a PTF package.
When I run that procedure I see the following:

"There is at least one temporarily applied fix for Licensed Internal Code
(LIC) on your system. If one of these fixes is superseded by a fix that
you are trying to install, some or all of the Licensed Internal Code
fixes will not install.

In order to ensure that the PTF installation will not fail because of a
superseded PTF, press F10 and restart the installation from the B
storage area.

Press F6 to continue with the installation.
Press F10 to stop the installation and to do an automatic IPL from
the B storage area."

OK. So, fingers crossed, I press F10. It IPL's to B and chokes with a
B600 0901 error again.

Next I try a manual "D" IPL. Taking the option to load the LIC. Load
completes and I manually step through the IPL, skipping the PTF apply
option. Then I go to load the CUME and get the same message: It wants me
to apply the temp PTF's before I run the PTF package install. If IPL to
the "B" side, it is still hosed up.

So I obviously don't understand what I am doing. Do I need to
permanently apply all PTF's on the A side in order to re-install the
CUME? Isn't there a danger in doing that? Somehow I thought the
re-install of the LIC/OS would "wipe out" the aborted CUME install from
before and allow me to start from scratch.

And, as a reminder if you aren't following the thread: The original CUME
install crashed when blade crashed. Since then I cannot IPL to the B
side. Re-installing was the supposed solution but I must not be doing
it correctly because I am making no progress.

What am I missing?

Pete

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.