Re: Post-mortem on v7r1 to v7r3 upgrade -- MIDRANGE-L

I did RTFM until my eyes were bleeding.
I have previous experience of NOT doing that on old systems a long time ago, and learned my lesson.
So, I downloaded and read the cover letters, memo to users (both v7r2 and v7r3), etc.

As for the latest I_BASE_01, that was the next thing IBM was going to have me do.
They had just put one up on their testbase download site.
I downloaded it and replaced the one in my image catalog (which was downloaded from the ESS exactly one week ago).
But they were able to finally get me around that D9002790 hard halt.

I confess I did NOT do the crossed fingers thing (but my tongue was sort of hanging out there for a long while).

Best Regards,

Thomas Garvey

On 11/2/2017 7:39 PM, Don Brown wrote:

I have done 3 upgrade
1 x V6R1 to V7R3 via V7R2 - Now that was fun.
2 x V7R1 to V7R3

All went perfectly and have had no downtime since the upgrades.

Each upgrade I updated to latest PTF's Cume, Groups, Hypers and especially
any pre-upgrade requirements from the upgrade planning documents.

Most important - RTFM or in this case the Upgrade planning documents.

Second most important step -RTFM or in this case the Upgrade planning
documents.

Ensure you complete all the pre-upgrade checks and verifications.

Also I recommend using the latest I-BASE version available.

I used Image catalogues for the upgrades and the PTF's

Also check the current status of all installed licensed programs and if
there are any errors do not start the upgrade until those are fixed.

Also check all PTF's are installed and made permanent to save space.

And finally make sure you cross your fingers and keep your tongue on the
left side of your mouth :-)

Don Brown

From: "Krill, Coy" <CKrill@xxxxxxxxxxx>
To: "Midrange Systems Technical Discussion" <midrange-l@xxxxxxxxxxxx>
Date: 03/11/2017 09:59 AM
Subject: RE: Post-mortem on v7r1 to v7r3 upgrade
Sent by: "MIDRANGE-L" <midrange-l-bounces@xxxxxxxxxxxx>

You guys are making me paranoid. I'm upgrading our DR box this weekend and
plan to do our production box next weekend after switching our environment
over to our DR box. We're going from V7R2 to V7R3, has anyone had better
experience with that upgrade?

Coy Krill
Core Processing Administrator/Analyst
Washington Trust Bank

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of
Justin Taylor
Sent: 2017 November 02 16:51
To: Midrange Systems Technical Discussion
Subject: RE: Post-mortem on v7r1 to v7r3 upgrade
Importance: Low

Our 7.1 to 7.3 upgrade was a nightmare. Our business partner did the
actual upgrade, and that went well. The hell began when we starting
actually running on 7.3.

In the two weeks following the upgrade we had:
8 hours of unplanned downtime
2 emergency IPL's
519 minutes of remote sessions with IBM support/devs
440 minutes on the phone with IBM support
1 untested, bleeding edge QSYS program to fix a breaking DB2 issue

Six weeks in, they're still sending me test PTF's in an attempt to fix
ongoing issues.

-----Original Message-----
From: Thomas Garvey [mailto:tgarvey@xxxxxxxxxx]
Sent: Thursday, November 02, 2017 5:25 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
Subject: Post-mortem on v7r1 to v7r3 upgrade

*I'm curious if our experience in upgrading from v7r1 to v7r3 was unusual,
or more common that expected.*

More than 6 months ago, I downloaded v7r3 to my PC and burned the
downloads to DVD's.
Those DVDs were used to create a NWSD intended for the purpose of creating
a hosted partition.
The plan was to use V7R1 to host a partition which would hold V7R3.
That never happened for unrelated reasons.
The plan changed to just upgrading the v7r1 partition to v7r3, so we used
the files, already in the IFS, to create an image catalog to do the
upgrade from.
After using the API to add all the files to the image catalog, the
LODIMGCLG repeatedly failed.
As each image entry failed, I would download the required file (the
download process was a nightmare in itself) and replace the entry in the
image catalog. This went file by file (IBASE to BGROUP1) until I gave up
on the previous DVD downloads, and downloaded EVERYTHING again (from the
ESS site).

Lessons learned: do NOT use IBM's Download Director (I NEVER got it to
succeed, at all), use the DownLoadThemAll add-on for FireFox instead

After following all the steps (verifying the image catalog, accepting
license agreements, use the pre-update PC application etc.) I started the
upgrade.
The upgrade from the image catalog ended in about 45 minutes, but I never
got a completion message because a *LNG object for the OAR licensed
software was not found.
We tried again, changing the software to install (skipping OAR) and we got
more errors in the loaded images.
How likely was that? Images that had previously been loaded and verified
and used for an install now failed? More downloading, etc.
Since we don't use/need the OAR feature, we finally proceeded as though
the install was complete.
Next we installed the latest cume we had downloaded (again to an image
catalog) and again never got a completion message.
From that point on, we were never able to IPL the system without a
failure.
For some reason the SCPF subsystem was corrupted and kept using more than
55% of the CPU for DAYS, while doing nothing (no entries in joblog, no
open files, etc.).
When system was restarted, it kept halting with D9002790 SRC code.

Got IBM involved and it appears the cume has a required PTF (from a
technical refresh) that must be applied before the cume can be applied.
Since this was all done from an image catalog, the required IPL would have
been automated, but could never complete, with the same halt.
We were finally able to get the PTF applied and performed several manual
IPLs to get past the halts.

Now, was the problem a flawed CUME download? Should I not have repeated
the LIC/OS install after de-selecting the OAR install?
Was the time between the initial availability of v7r3 and my attempt at
installing it too distant?
(all the v7r1 PTFs were applied, the PC pre-install application confirmed
everything was done and ready).

I was 'this close' to just going back to v7r1. And IBM concurred, if the
last effort they tried didn't work.

*I don't expect that anyone came close to this nightmare, but did
**_anyone_**get through it in a few hours without a hitch?* I know, much
too long a post, and maybe I'm venting, but I am curious.

/Also, BTW, much thanks to all those who helped by responding on this
forum to my earliest requests./

Best Regards,

Thomas Garvey