RE: Post-mortem on v7r1 to v7r3 upgrade

We appear to have been in the same boat, our upgrade went smoothly, but
we've had issues since.

We too have had an emergency IPL to clear up issue, have noticed a slow-down
in job completion times (have tried to address by adding additional indices
noted by the index advisor but still no major increase in throughput time
and some of our DB2 WebQuery jobs are just tanking now) and increasing disk
consumption (not all related to the additional indices).

This last one has the primary responsible for the system pulling his hair
out as he's not able to pin point what/where the additional space usage is
coming from. Though we have seen an uptick in temporary space usage but not
enough to warrant the big jump in disk usage. He is about ready to throw in
the towel and call IBM on this last one.

Ken

-----Original Message-----
From: Justin Taylor [mailto:JUSTIN@xxxxxxxxxxxxx]
Sent: Thursday, November 02, 2017 7:51 PM
To: Midrange Systems Technical Discussion
Subject: RE: Post-mortem on v7r1 to v7r3 upgrade

Our 7.1 to 7.3 upgrade was a nightmare. Our business partner did the actual
upgrade, and that went well. The hell began when we starting actually
running on 7.3.

In the two weeks following the upgrade we had:
8 hours of unplanned downtime
2 emergency IPL's
519 minutes of remote sessions with IBM support/devs
440 minutes on the phone with IBM support
1 untested, bleeding edge QSYS program to fix a breaking DB2 issue

Six weeks in, they're still sending me test PTF's in an attempt to fix
ongoing issues.




-----Original Message-----
From: Thomas Garvey [mailto:tgarvey@xxxxxxxxxx]
Sent: Thursday, November 02, 2017 5:25 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxx>
Subject: Post-mortem on v7r1 to v7r3 upgrade

*I'm curious if our experience in upgrading from v7r1 to v7r3 was unusual,
or more common that expected.*

More than 6 months ago, I downloaded v7r3 to my PC and burned the downloads
to DVD's.
Those DVDs were used to create a NWSD intended for the purpose of creating a
hosted partition.
The plan was to use V7R1 to host a partition which would hold V7R3.
That never happened for unrelated reasons.
The plan changed to just upgrading the v7r1 partition to v7r3, so we used
the files, already in the IFS, to create an image catalog to do the upgrade
from.
After using the API to add all the files to the image catalog, the LODIMGCLG
repeatedly failed.
As each image entry failed, I would download the required file (the download
process was a nightmare in itself) and replace the entry in the image
catalog. This went file by file (IBASE to BGROUP1) until I gave up on the
previous DVD downloads, and downloaded EVERYTHING again (from the ESS site).

Lessons learned: do NOT use IBM's Download Director (I NEVER got it to
succeed, at all), use the DownLoadThemAll add-on for FireFox instead

After following all the steps (verifying the image catalog, accepting
license agreements, use the pre-update PC application etc.) I started the
upgrade.
The upgrade from the image catalog ended in about 45 minutes, but I never
got a completion message because a *LNG object for the OAR licensed software
was not found.
We tried again, changing the software to install (skipping OAR) and we got
more errors in the loaded images.
How likely was that? Images that had previously been loaded and verified and
used for an install now failed? More downloading, etc.
Since we don't use/need the OAR feature, we finally proceeded as though the
install was complete.
Next we installed the latest cume we had downloaded (again to an image
catalog) and again never got a completion message.
From that point on, we were never able to IPL the system without a failure.
For some reason the SCPF subsystem was corrupted and kept using more than
55% of the CPU for DAYS, while doing nothing (no entries in joblog, no open
files, etc.).
When system was restarted, it kept halting with D9002790 SRC code.

Got IBM involved and it appears the cume has a required PTF (from a
technical refresh) that must be applied before the cume can be applied.
Since this was all done from an image catalog, the required IPL would have
been automated, but could never complete, with the same halt.
We were finally able to get the PTF applied and performed several manual
IPLs to get past the halts.

Now, was the problem a flawed CUME download? Should I not have repeated the
LIC/OS install after de-selecting the OAR install?
Was the time between the initial availability of v7r3 and my attempt at
installing it too distant?
(all the v7r1 PTFs were applied, the PC pre-install application confirmed
everything was done and ready).

I was 'this close' to just going back to v7r1. And IBM concurred, if the
last effort they tried didn't work.


*I don't expect that anyone came close to this nightmare, but did
**_anyone_**get through it in a few hours without a hitch?* I know, much too
long a post, and maybe I'm venting, but I am curious.


/Also, BTW, much thanks to all those who helped by responding on this
forum to my earliest requests./

Best Regards,

Thomas Garvey