RE: Post-mortem on v7r1 to v7r3 upgrade

Our 7.1 to 7.3 upgrade was a nightmare. Our business partner did the actual upgrade, and that went well. The hell began when we starting actually running on 7.3.

In the two weeks following the upgrade we had:
8 hours of unplanned downtime
2 emergency IPL's
519 minutes of remote sessions with IBM support/devs
440 minutes on the phone with IBM support
1 untested, bleeding edge QSYS program to fix a breaking DB2 issue

Six weeks in, they're still sending me test PTF's in an attempt to fix ongoing issues.




-----Original Message-----
From: Thomas Garvey [mailto:tgarvey@xxxxxxxxxx]
Sent: Thursday, November 02, 2017 5:25 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
Subject: Post-mortem on v7r1 to v7r3 upgrade

*I'm curious if our experience in upgrading from v7r1 to v7r3 was unusual, or more common that expected.*

More than 6 months ago, I downloaded v7r3 to my PC and burned the downloads to DVD's.
Those DVDs were used to create a NWSD intended for the purpose of creating a hosted partition.
The plan was to use V7R1 to host a partition which would hold V7R3.
That never happened for unrelated reasons.
The plan changed to just upgrading the v7r1 partition to v7r3, so we used the files, already in the IFS, to create an image catalog to do the upgrade from.
After using the API to add all the files to the image catalog, the LODIMGCLG repeatedly failed.
As each image entry failed, I would download the required file (the download process was a nightmare in itself) and replace the entry in the image catalog.  This went file by file (IBASE to BGROUP1) until I gave up on the previous DVD downloads, and downloaded EVERYTHING again (from the ESS site).

Lessons learned: do NOT use IBM's Download Director (I NEVER got it to succeed, at all), use the DownLoadThemAll add-on for FireFox instead

After following all the steps (verifying the image catalog, accepting license agreements, use the pre-update PC application etc.) I started the upgrade.
The upgrade from the image catalog ended in about 45 minutes, but I never got a completion message because a *LNG object for the OAR licensed software was not found.
We tried again, changing the software to install (skipping OAR) and we got more errors in the loaded images.
How likely was that? Images that had previously been loaded and verified and used for an install now failed? More downloading, etc.
Since we don't use/need the OAR feature, we finally proceeded as though the install was complete.
Next we installed the latest cume we had downloaded (again to an image
catalog) and again never got a completion message.
From that point on, we were never able to IPL the system without a failure.
For some reason the SCPF subsystem was corrupted and kept using more than 55% of the CPU for DAYS, while doing nothing (no entries in joblog, no open files, etc.).
When system was restarted, it kept halting with D9002790 SRC code.

Got IBM involved and it appears the cume has a required PTF (from a technical refresh) that must be applied before the cume can be applied.
Since this was all done from an image catalog, the required IPL would have been automated, but could never complete, with the same halt.
We were finally able to get the PTF applied and performed several manual IPLs to get past the halts.

Now, was the problem a flawed CUME download? Should I not have repeated the LIC/OS install after de-selecting the OAR install?
Was the time between the initial availability of v7r3 and my attempt at installing it too distant?
(all the v7r1 PTFs were applied, the PC pre-install application confirmed everything was done and ready).

I was 'this close' to just going back to v7r1. And IBM concurred, if the last effort they tried didn't work.


*I don't expect that anyone came close to this nightmare, but did
**_anyone_**get through it in a few hours without a hitch?*
I know, much too long a post, and maybe I'm venting, but I am curious.


/Also, BTW, much thanks to all those who helped by responding on this
forum to my earliest requests./

Best Regards,

Thomas Garvey