Re: LIC re-install - Confused on the steps to take

Pete,

I sure can't disagree with your logic here. Seems like all fingers point
to the same failing component - The SAS module. It reminds me of an I/O
problem on our S/36 years back where it too went casters up at busy times.
To make a long story short, it was solved by the local SE saying words
like: "It has to be this card, SRC codes be damned." and so it was.

This is the system IBM would LOVE to sell to many 270, 800, 810, and 520
customers. They are NOT doing themselves any favors here by fighting you on
this. Heck if they swap the part and it's NOT bad they've proved that and
they STILL HAVE a good part! WHAT'S THE HARM?

Having heard your tale of woe on this for many weeks I'm very close to
copying someone in Rochester which will a) Get this fixed and b) Rain holy
heck down on me for doing so (Others reading this will know of whom I
speak.) Instead if you could send me privatly the PMR number you have open
on this (hardware and software if you got them.) then I will get the right
people involved. If not I will be in Rochester on Monday next week showing
off Frankie III. If they haven't behaved by then, I'll tip Frankie over on
them. :-)

- L



Larry Bolhuis IBM Certified Advanced Technical Expert:
Vice President System i Solutions
Arbor Solutions, Inc. IBM Certified Systems Expert:
1345 Monroe NW Suite 259 System i Technical Design and Implementation
V6R1
Grand Rapids, MI 49505
(616) 451-2500
(616) 451-2571 - Fax
(616) 260-4746 - Cell



(Embedded image moved to file: pic09137.gif)

If you can read this, thank a teacher....and since it's in English, thank
a soldier.




Pete Helgren
<Pete@xxxxxxxxxx>
Sent by: To
midrange-l-bounce Midrange Systems Technical
s@xxxxxxxxxxxx Discussion
<midrange-l@xxxxxxxxxxxx>
cc
05/11/2009 11:11
AM Subject
Re: LIC re-install - Confused on
the steps to take
Please respond to
Midrange Systems
Technical
Discussion
<midrange-l@midra
nge.com>






Dr. Franken! I was hoping you would weigh in.

Two minutes for a LIC load seemed kinda short to me as well. The fact
that the "A" side still shows PTF's needing to be applied and the "B"
side won't IPL made me suspicious that all was not well here. This may
be a clue: I remember seeing a message that mirroring had been suspended
on one of the pairs. I don't recall which one. Would a suspended
mirror cause the LIC not to load? I didn't seeing any errors, just
warning about the suspended mirror. After starting the mirroring
("resuming", I think it said) the blade didn't last long.

At the core of this is the "disappearing" SAS drives. I can usually get
them to appear by first pulling the DSM (Storage Module) and then
reinserting it. The AMM then indicates that it can't read the firmware
level of the DSM so I have to bounce the MM and, at that point, all is
well. I can start the blade and, in the IVM or at the VIOS command line
I can see the SAS drives as available. From here, the cause of the
"disappearance" becomes difficult to pin down. The blade itself, just
running VIOS stays up and running about 98% of the time. But, the blade
itself can run without having the SAS drives in the DSM available
because it uses the two onboard HDD's for storage. The problem occurs
in two ways:

1. Start the i partition. It will run for minutes, hours, days (rarely)
without a known trigger as to what causes it to fail. I'll just notice
that the AMM starts reporting errors from the blade and then when I go
to the i, the console is unresponsive. I have suspected that a large
amount of I/O is what is causing it to fail but I can't prove it. When
I resumed the mirroring on the pair for instance, the blade was dead
within minutes.

2. Back up files on VIOS to tape (external SAS LTO4 drive). This also
involves the SAS Connectivity module (where I suspect the problem is).
I was successful in getting the i to do a savsys to virtual optical
media this weekend so there is now a file in VIOS I can back up to
tape. When I launched the backup, VIOS went dead stick, along with the
blade, within a minute or two. This happened both times I attempted to
back up to tape.

So I think all fingers point to a bad SAS Conn Module but IBM thinks
otherwise. They *know* there is an issue with the media tray and the i
partition, but this thing fails whether the media tray is inserted or
not. This box has been casters up for almost 4 weeks (yeah, I have had
a couple of low availability weeks to work on it but I have been trying
at least once a day to keep this thing running). So my plan is now to
start swinging the dead cat around the IBM support desk and see if I
can't get them to send out a tech to fix this thing. Unfortunately, I
leave town tomorrow so I think I have another week of "try this, try
that" before I can get someone to fix this.

Pete

lbolhuis@xxxxxxxxxx wrote:

Pete,

Sounds like fun... :-( I'm just reading this thread after a
catastrophic laptop failure this weekend. Luckily, though my primary

backup

is on Frankie III which is currently in Rochester, I have a backup for my
backup!

So if I understand your blade crashed during PTFs. This FUBARed the

link

loader on the "B" side, so that side is toast. IPL to "A" side is (ok
"was") good. You first re-installed the OS (oops) and while that didn't
help it shouldn't hurt. Then you IPLed "D" again from an RSE disk and
installed the LIC with option 1. (Which is the correct path to take.) .
You said it took only 2 minutes. THAT Makes me go Hmmm. Installing the

LIC

isn't a long process but given the blade center has no cache and the

drives

are mostly 10K units and there is VIOS in there, two minutes seems very
quick, almost as if it didn't go where you expected, or maybe anywhere.

The

fact that a "B" mode IPL fails with the same error seems to add credence

to

this theory.

The one thing I might do at this point is the "D" IPL - install the

LIC,

then from there Install the OS again and let that complete. Use the OS
Disks you downloaded with your RSE slic. This will give you a complete
replacement of software from the bottom up putting both LIC and the OS at
the same PTF level. From there all PTF processing should go normally.

But there is that pesky "can't see the drives" problem at this point.

So

you are now stuck with no drives? Does VIOS See the drives or are they

gone

completely?

- Larry


Larry Bolhuis IBM Certified Advanced Technical Expert:
Vice President System i Solutions
Arbor Solutions, Inc. IBM Certified Systems Expert:
1345 Monroe NW Suite 259 System i Technical Design and Implementation
V6R1
Grand Rapids, MI 49505
(616) 451-2500
(616) 451-2571 - Fax
(616) 260-4746 - Cell



(Embedded image moved to file: pic04760.gif)

If you can read this, thank a teacher....and since it's in English,

thank

a soldier.

Pete Helgren

<Pete@xxxxxxxxxx>

Sent by:

To

midrange-l-bounce Midrange Systems Technical

s@xxxxxxxxxxxx Discussion

<midrange-l@xxxxxxxxxxxx>

cc

05/10/2009 08:36

PM

Subject

Re: LIC re-install - Confused on

the steps to take

Please respond to

Midrange Systems

Technical

Discussion

<midrange-l@midra

nge.com>

It's RSE. And yes, the first time I made the mistake of loading the OS
instead of the LIC. However, my later post to Lukas, I outlined the
steps I took, the first being loading the BASE CD and then IPL'ing from
D, after a 2 minute or so load, the i bounced and I ended up with a
manual A side IPL option. I was waiting to find out what the next is
that I should take.

Meanwhile, the blade crashed, again, and I can't get the partition to
see the SAS drives, no matter what I do. It will be a long day on the
phone with IBM tomorrow. I hope they send me a new blade/SAS module.
This POJ is taking up too much of my time.

Pete


Pete Massiello wrote:

Pete,

I think you reinstalled the OS. You need to reinstall the

LIC. Do

a "D" mode IPL from the I_BASE Cd. Make sure you have a current one.

In

fact, you are best to get the latest one at

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss

You should then load this onto the "A" side, which will wipe

out all

your PTFs. Then just IPL that side up, and then you can do your PTFs.

If

you installed a new version of LIC, then you won't have any PTF

processing

as you state below.

I would make sure my I_BASE cd is either 610 RSD or RSE

Pete

Pete Massiello
iTech Solutions
http://www.itechsol.com


-----Original Message-----
From: midrange-l-bounces@xxxxxxxxxxxx
[mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Pete Helgren
Sent: Sunday, May 10, 2009 4:49 PM
To: Midrange Systems Technical Discussion
Subject: LIC re-install - Confused on the steps to take

I am trying to snuff a B side IPL issue where I get a B600 0901 (link
loader failure) error. So i decided to reinstall LIC/OS.

The first attempt was just a manual IPL to the A side (the side that
runs). I took the option to install the operating system and after an
hour or so had completed the install of the OS. I then moved on to
install the latest CUME taking the option to install a PTF package.
When I run that procedure I see the following:

"There is at least one temporarily applied fix for Licensed Internal

Code

(LIC) on your system. If one of these fixes is superseded by a fix that
you are trying to install, some or all of the Licensed Internal Code
fixes will not install.

In order to ensure that the PTF installation will not fail because of a
superseded PTF, press F10 and restart the installation from the B
storage area.

Press F6 to continue with the installation.
Press F10 to stop the installation and to do an automatic IPL from
the B storage area."

OK. So, fingers crossed, I press F10. It IPL's to B and chokes with a
B600 0901 error again.

Next I try a manual "D" IPL. Taking the option to load the LIC. Load
completes and I manually step through the IPL, skipping the PTF apply
option. Then I go to load the CUME and get the same message: It wants me
to apply the temp PTF's before I run the PTF package install. If IPL to
the "B" side, it is still hosed up.

So I obviously don't understand what I am doing. Do I need to
permanently apply all PTF's on the A side in order to re-install the
CUME? Isn't there a danger in doing that? Somehow I thought the
re-install of the LIC/OS would "wipe out" the aborted CUME install from
before and allow me to start from scratch.

And, as a reminder if you aren't following the thread: The original CUME
install crashed when blade crashed. Since then I cannot IPL to the B
side. Re-installing was the supposed solution but I must not be doing
it correctly because I am making no progress.

What am I missing?

Pete

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing

list

To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.