Re: P7 to P9 test migration/recovery using BRMS - issues/resolutions -- MIDRANGE-L

I've always found the BRMS spool file support to be, well, not worth the effort. They've added some nice features but the base save/restore code, as you point out, has horrible performance so I always just save the out queues with the library and restore from there, sometimes using native commands instead.

You are correct the policy changes should be persistent, I think a PMR is in order there.

As to the problems with recovery, I don't share the same experience. I've done 100s of BRMS recoveries and while some present a challenge or two, nothing like what your experience has been.

I've always considered data queue data to be transitory so the application should be able to recreate the transaction. Where it cannot then journal it. Use that for recovery.

Data areas also hold transitory data so same rules apply.

During a split recovery where you have to force recovery from the complete save, that's where humans have to apply their expertise. I just force that stuff to restore from the base save, then move on with the daily.

BRMS is an excellent tool, but like a hammer, there are many options to use it. Some times a tack hammer is needed vs. a framing hammer. Your recovery plan needs to be set up for your system and application. Each system is different in the data flows and usage, and that might include manual steps as well as native vs. BRMS commands.

Your experience is a classic case where recovery tests take at least three tries before the recovery plan is solid. Trying to build a recovery plan from a back up almost always spells trouble in capitals, which brings me to the two main points I put in all my BRMS user groups sessions.

Build a recovery plan, and then the backup to support it, not the other way around.
If you don't test your recover plan, you don't have a recoverable back up.

You are well on your way to successful recoveries.

Jim Oberholtzer
Agile Technology Architects

On Dec 31, 2018, at 8:14 AM, Steinmetz, Paul <PSteinmetz@xxxxxxxxxx> wrote:

When setting/Changing BRMS Recovery Policy - WRKPCYBRM *RCY, shouldn't these changes be permanent.
I'm finding their owning valid for your current interactive session.
If you sign off/on, Recovery Policy settings are back to default.

Also, any feedback on my other issues, 1,2,3 noted below.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Steinmetz, Paul
Sent: Friday, December 28, 2018 1:54 PM
To: 'Midrange Systems Technical Discussion'
Subject: RE: P7 to P9 test migration/recovery using BRMS - issues/resolutions

1) New migration issue, spoolfiles.
When I got to STEP 021 : Recover Spooled Files for all ASPs
I found that WRKSPLFBRM was not working.
After a PMR with IBM, it was discovered that in order to use WRKSPLFBRM, BRMS must be configured to Retain object level detail, which I changed to *YES to *ERR back in 2009 due to performance issues.
Retain Object level detail adds much extra time (hours) and space to a BRMS save (Output File Processing) and is NOT recommended by IBM.
I'm not sure if this has changed over the years, anyone have input/feedback related to this?

Also discovered that if I had changed by STRRCYBRM recovery defaults, Restore spooled file data from *NONE to *NEW, all my spoolfiles would have been restored with the outqs for those libraries.
Most of my outqs reside in either QUSRSYS or QGPL, others in some 3rd party products.

I tested this, changing recovery default Restore spooled file data from *NONE to *NEW, re restoring those libraries containing outqs, all spoolfiles that were saved were restored.

IBM concluded that the BRMS recovery report, Step 10, needs be improved in regards to changing/specifying the correct recovery defaults.
Because this is language sensitive, it wouldn't be PTF'd, but done in a future release.

2) Most of my migration/recovery issues were due to BRMS recovery defaults, that if set differently, would have simplified and improved the recovery.
I'm was thinking of CHGCMDDFT for STRRCYBRM, a better and preferred method by IBM was to set and store the BRMS recovery defaults. WRKPCYBRM *RCY.
I didn't know this existed.
I'm currently setting these on all my current LPARS, one less thing to deal with on a migration/recovery.

Device . . . . . . . . . . . . . . TAPMLB01
End of media option . . . . . . . . *LEAVE
Data base member option . . . . > *ALL
Spooled file data . . . . . . . > *NEW
Allow object differences . . . . > *COMPATIBLE
System resource management . . . . *NONE

Also, using the defaults for "Time period for recovery" may not be the best choice.
Selecting a recent date did solve some issues.
Start time and date:
Beginning time . . . . . . . . *AVAIL
Beginning date . . . . . . . . *BEGIN
End time and date:
Ending time . . . . . . . . . *AVAIL
Ending date . . . . . . . . . *END

What are others in the group using for BRMS recovery defaults?

3) Finally, I've discovered a serious issue with a BRMS recovery, if I ever had to recover my system, with BRMS, using the last dedicated system save and my last full nightly (which is a SWA - Save While Active),
BRMS is not smart enough to restore items from the last dedicated system save that were either omitted for skipped due to locks during the nightly save.
This issue is only with BRMS, a Restore 21 restores entire system from last system save.

PF, LF, pgms are all good from the nightly save.
Problem areas are with objects that an application (mostly 3rd party) were written with exclusive locks and thus not saved during the nightly.
ALCOBJ OBJ((XXX/YYY *DTAARA *EXCL))
Examples:
DTAQ, USRSPC, DTAARA, MSGQ

How are others in the group handling this issue?
Stopping/starting the applications is not an option.

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Jim Oberholtzer
Sent: Wednesday, December 26, 2018 10:41 AM
To: 'Midrange Systems Technical Discussion'
Subject: RE: P7 to P9 test migration/recovery using BRMS - issues/resolutions

Paul:

While RTVDSKINF will give you generalized information about the IFS. The RTVDIRINF can give you very detailed information about all the mounted directory objects on the system. I say mounted since we can now have UFDS that may or may not be mounted when you run your command.

My guess is you want both commands and then you'll have a complete database (along with the DSPOBJD command) of everything that is on the system. More information than you really need, but you can cull out what you don't want with SQL easily enough.

Keep in mind that a significant amount of the IFS is logs, and similar type files that will never be the same except on a replicated system, and even then they will be different in some areas.

--
Jim Oberholtzer
Agile Technology Architects

-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxx> On Behalf Of Steinmetz, Paul
Sent: Wednesday, December 26, 2018 9:22 AM
To: 'Midrange Systems Technical Discussion' <midrange-l@xxxxxxxxxxxx>
Subject: RE: P7 to P9 test migration/recovery using BRMS - issues/resolutions

Jim / Rob,

< Clearly a DSPOBJD *all/*all into a file on both systems the use SQL to compare libraries is an option too.

I like your idea.
At first, I was simply thinking of comparing library totals.
But a match of all objects in the libraries (and IFS), source against target, would give a fast, accurate, confirmation of the restore.

I run a RTVDSKINF daily, takes about 7 minutes, which I use for multiple other purposes.
RTVDSKINF creates QAEZDISK PF, about 2,000,000 records(objects).

From a previous post, Rob also suggested a RTVDIRINF.
RTRDIRINF takes about 3 hours to run, I only run this as needed.
RTVDIRINF creates QAEZD000#O and #D PF, about 2,500,000 records(objects).

1) Before I create this, what additional detail is RTVDIRINF reporting that RTVDSKINF is lacking?

2) Has anyone from the group ever used either of these to confirm any missing objects on the target system from a migration or a system recovery?

3) Any thoughts from the group?

Paul

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Jim Oberholtzer
Sent: Wednesday, December 19, 2018 9:07 AM
To: 'Midrange Systems Technical Discussion'
Subject: RE: P7 to P9 test migration/recovery using BRMS - issues/resolutions

OK, starting from the top:

Always download the latest code especially for LIC from ESS and use that. And since you did not use the LIC from the tape, you really need to reapply all your PTFs. Better yet, get all new ones and get them applied. You may have done this already. Even if you think this is resolved, it might not be. I've had this bite in the past, hard. Since you can apply all new PTFS to save grief down the road.

Since you told BRMS to use multiple drives on the restore, and it did not, that's an error and a PMR should be called in. I'm glad you manually corrected the problem.

To confirm the restore was complete, I always use the *PRINT on the restores, then search for "not restored" in the spool file. Clearly a DSPOBJD *all/*all into a file on both systems the use SQL to compare libraries is an option too.

--
Jim Oberholtzer
Agile Technology Architects

-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxx> On Behalf Of Steinmetz, Paul
Sent: Tuesday, December 18, 2018 3:02 PM
To: 'Midrange Systems Technical Discussion' <midrange-l@xxxxxxxxxxxx>
Subject: P7 to P9 test migration/recovery using BRMS - issues/resolutions

1) 5770SS1 *BASE *ERROR IBM i
CHKPRDOPT PRDID(5770SS1)
Product 5770SS1 release V7R3M0 L00 option *BASE load 2924 correctly installed.
Required PTF 5770999-MF65144 V7R3M0 not in correct status for PTF 5770SS1-SI68084 V7R3M0.
Required PTF 5770999-MF65297 V7R3M0 not in correct status for PTF 5770SS1-SI68338 V7R3M0.
PTF requisite errors found for product 5770SS1.

Resolution.
CHKPRDOPT PRDID(5770SS1) VFYREQPTF(*NO)
Resolved the issue.
5770SS1 *COMPATIBLE IBM i
Issue resolved.

2) Because LIC was installed from I_BASE_01 RS-73--F DVD that shipped with my P9, I was missing some PTFS that were temp applied on the source system are appearing as Cover Letter Only on the target.

Resolution.
slip install LIC
http://www-01.ibm.com/support/docview.wss?uid=nas8N1018561
Issue Resolved.

3) When doing an D-IPL with HMC profile set to alternate install deice, (FC tape library), and then setting the alternate IPL device during the IPL,
Unable to allocate resources - installation failed
Alternate Installation Device Failed
The alternate installation device or media failed.

Resolution
When HMC profile is set to alternate IPL device ( (FC tape library), DO NOT set the alternate install options during the IPL, simply continue.
Issue resolved.

4)All the *IBM libraries, which were being restored form a four volume system save, were experiencing significant tape seeking, unloading, loading during the recovery.
I had BRMS recovery set to use all 4 drives, I'm not sure why this is occurring.

Temp Resolution – Open item.
I only selected the restore items from the same volume which were in sequence.
Restore streamed with no rewind, etc.
I repeated this process for each volume listed in the recovery.
Issue temporarily resolved.
Waiting for answers from IBM development on how to perform the restore when the save had multiple, concurrent volumes.

5) STRRCYBRM OPTION(*SYSTEM) (report shows correct volume) is showing a duplicated volume not in the library, as opposed to the original volume which is in the library.

Found an old PMR which had us create BRMS dtaara , QUSRBRM/Q1AIGNRDUP.
I don’t remember the specifics on why we had to do this.
https://www.ibm.com/developerworks/community/wikis/home - !/wiki/IBM%20Backup%2C%20Recovery%20and%20Media%20Services%20%28BRMS%29%20for%20i/page/Include%20both%20original%20save%20media%20and%20duplicate%20save%20media%20in%20Recovery%20Volume%20Summary%20Report%20-%20Q1AIGNRDUP<https://www.ibm.com/developerworks/community/wikis/home#!/wiki/IBM%20Backup%2C%20Recovery%20and%20Media%20Services%20%28BRMS%29%20for%20i/page/Include%20both%20original%20save%20media%20and%20duplicate%20save%20media%20in%20Recovery%20Volume%20Summary%20Report%20-%20Q1AIGNRDUP>
We also changed the command defaults for USEDUPMED from *NO to *YES.
Use duplicate media . . . . . . USEDUPMED *YES

Resolution.
Removed BRMS datara , QUSRBRM/Q1AIGNRDUP and changed command defaults for USEDUPMED from *YES to *NO.
Issue resolved.

5) STRRCYBRM OPTION(*SYSTEM) is included 3 libraries for recovery that were deleted over 6 years ago.

Resolution
Changing Show deleted libraries from default of *YES to *NO DID NOT resolve this issue.
Either expire the old tape or change the Beginning date .from *BEGIN to a date after 6/3/12.
Issue resolved.

6) STRRCYBRM OPTION(*SYSTEM) is including a 6 year old LTO5 volume, 2 sequences, (currently LTO7).

Saved Save Save Save Parallel Volume File Expire
Item Date Time Type Devices Serial Sequence Date
*NONSYS 6/03/12 12:24:59 *FILE 001060 681 *PERM
QDOC 6/03/12 12:41:40 *FILE 001060 1015 *PERM

Resolution.
Either expire the old tape or change the Beginning date .from *BEGIN to a date after 6/3/12.
Issue resolved.

7) Several thousand X-Lib LF initially not restored, later restored via deferred object.
How does one confirm that all X-LIB deferred LF got restored.

Resolution.
Confirmed by looking at WRKF FILE(QRECOVERY/QADBRSDFR) No records exist, all deferred objects were restored.
Issue resolved.

8) How does one easily determine what objects, if any, did not get restored?
Verifying whether objects are restored successfully https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/rzarm/rzarmvfyrst.htm

Resolution
Open, 100% restore not confirmed.

Thank You
_____
Paul Steinmetz
IBM i Systems Administrator

Pencor Services, Inc.
462 Delaware Ave
Palmerton Pa 18071

610-826-9117 work
610-826-9188 fax
610-349-0913 cell
610-377-6012 home

psteinmetz@xxxxxxxxxx<mailto:psteinmetz@xxxxxxxxxx>
http://www.pencor.com/

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com