Yesterday I had to force restart two LPARs from HMC, due to LPAR Ethernet comm issue due to a PTF apply issue.
This could have been much worse.
One damaged object *DTAQ, Two access paths corrupted that required manual intervention.
The iSeries recovery is amazing, keeps getting better.
I haven't had to do one of these since the old S/38 , AS/400 days.
Every recovery could result in different errors/issues.
My MPLUS monitoring isn't configured for these type of errors/issues.
Has anyone created monitoring for these types of recovery messages/issues?
Not all messages were in QHST, some were from JOBLOGs.
Any thoughts from the group?
Below are clips and quantiles of the various recovery messages NOT seen during a normal IPL.
Production LPAR - V7R1 P7 740 - 8205-E6C - 100% SSD
1) qty 15 - System journal receivers being recreated, with damage but ignored.
CPC7011 00 COMPLETION Journal receiver QSQTTJ0025 created in library QRECOVERY.
SCPF QSYS 000000 QJOIPL 0000 03/09/16 02:48:49.661576 QSYS
CPF7001 30 DIAGNOSTIC Damage was detected but has been ignored.
SCPF QSYS 000000 QJOIPL 06A8 03/09/16 02:48:49.661609 QSYS
CPF7020 00 INFO Journal receivers QSQTTJ0024 and *N detached.
SCPF QSYS 000000 QJOIPL 0000 03/09/16 02:48:49.661686 QSYS
2) qty 53 - : CPI8356 - IPL recovery of commitment definition *DFTACTGRP for job
Message ID . . . . . . : CPI8356
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:48:57
Message . . . . : IPL recovery of commitment definition *DFTACTGRP for job
148293/IC_GTWYP1/APISTART started.
Cause . . . . . : Conditions that were detected during an IPL or ASP devic
vary on indicate that commitment control was active in job
148293/IC_GTWYP1/APISTART for commitment definition *DFTACTGRP at the time
the system failed. The system will attempt to rollback the last transactio
for the job, unless the system failed during a commit operation. In that
case, the system will either attempt to complete the commit operation or
attempt to prepare the commitment definition for an eventual commit or
rollback operation.
Recovery . . . : No recovery is required. If the system attempts to
complete a commit operation, this message is followed by CPI8350. If the
3) qty 1139 - CPF3124 File QAPMRESP in QMPGDATA in use at abnormal system end.
Message ID . . . . . . : CPF3124
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:49:11
Message . . . . : File QAYPSJET in QMGTC2 in use at abnormal system end.
Cause . . . . . : Member QAYPSJET file QAYPSJET in library QMGTC2 was in use
when the system abnormally ended. 0 is the last relative record number.
4) qty -1 CPI3225 System access path protection processing has started.
Message ID . . . . . . : CPI3225
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:49:12
Message . . . . : System access path protection processing has started.
Cause . . . . . : This message is issued during an IPL, at the start of
system access path protection.
Recovery . . . : This is an informational message and no recovery is
required.
5) qty 85 - CPF3123 Access path for member QADBXTCDEF recovered or built during IPL in 00:00:01.
Message ID . . . . . . : CPF3123
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:49:12
Message . . . . : Access path for member QADBXTCDEF recovered or built
during IPL in 00:00:01.
Cause . . . . . : The access path for member QADBXTCDEF file QADBXTCDEF in
library QSYS was recovered or built for one of the following reasons:
-- The file RECOVER attribute is *IPL.
-- The RECOVER attribute was overridden to *IPL for this IPL.
-- The access path is journaled and is MAINT(*IMMED).
-- The file was damaged.
6) qty 1 - CPI 3226 - System access path protection processing has completed.
Message ID . . . . . . : CPI3226
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:49:13
Message . . . . : System access path protection processing has completed.
Cause . . . . . : This message is issued during an IPL, at the end of system
access path protection.
Recovery . . . : This is an informational message and no recovery is
required.
7) qty 1 - CPF9898 RECOVERY OF JOURNALED ACCESS PATHS WITH MAINT(*IMMED) IS COMPLETE.
Message ID . . . . . . : CPF9898
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:49:13
Message . . . . : RECOVERY OF JOURNALED ACCESS PATHS WITH MAINT(*IMMED) IS
COMPLETE.
Cause . . . . . : This message is used by application programs as a general
escape message.
8) qty - 40 CPF 3272 Damage ignored for file IGI7024A in QTEMP
Message ID . . . . . . : CPF3272
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:50:21
Message . . . . : Damage ignored for file IGI7024A in QTEMP.
Cause . . . . . : Either member *N or file IGI7024A in library QTEMP was
found damaged during processing. The requested operation was partially
completed. Related files or members may be logically damaged.
Recovery . . . : If the request is not for a delete file operation (DLTF
command), the damaged file IGI7024A should be deleted and restored (RSTOBJ
or RSTLIB command). Then try your request again.
9) one damaged object, an application *DATAQ, which I was able to delete/recreate from a copy.
From HSTLOG
CPF8147 40 INFO Full damage on data queue MOPRSCUR in CBWORKDATA. VLOG-080163E4.
QSYSARB2 QSYS 149793 QRCDMGLG 0000 03/09/16 05:05:39.595864 QSYS
CPF8198 80 INFO Damaged object found.
QSYSARB2 QSYS 149793 QRCDMGLG 0000 03/09/16 05:05:39.596072 QSYS
10) an access path reporting Unique access path problems prevent updates to member which we resolved by cpyf the PF records out/in using *replace.
From JOBLOG
CPF5090 Notify 30 03/09/16 04:05:31.434764 QDBSIGEX QSYS 0252 DAILYPROC IC08XXPGMS *STMT
To module . . . . . . . . . : DAILYPROC
To procedure . . . . . . . : DAILYPROC
Statement . . . . . . . . . : 180
Message . . . . : Unique access path problems prevent updates to member
CCE8CPL0.
11) one access path reporting records with duplicate keys, which we resolved restoring from previous night's backup to temp file, then cpyf the PF records out/in using *replace.
From JOBLOG
CPF4012 Diagnostic 10 03/09/16 06:13:04.518849 QDBSIGEX QSYS 0EAB QDMCOPEN QSYS 0892
Message . . . . : Unique keyed access path for member CDRZCPL0 not built.
Cause . . . . . : The unique keyed access path for member CDRZCPL0 file
CDRZCPL0 in library CBWORKDATA could not be built because the keys were not
unique. Record 2290688 format #RZCPI6 member number 1 has the same key as
record 5806717 format #RZCPI6 member number 1. Recovery . . . : Either
remove one of the records with a duplicate key or change the key of either
record. Then try your request again.
CPF4161 Escape 50 03/09/16 06:13:04.518938 QDBSIGEX QSYS 05A4 IGI1355 IC08XXPGMS 3B9D
Message . . . . : Records in member CDRZCPL0 have duplicate keys.
Cause . . . . . : Duplicate keys were found in the records of member
CDRZCPL0 file CDRZCPL0 in library CBWORKDATA when the access path was being
built. Recovery . . . : See message CPF4012, which was previously listed,
to see which records in member CDRZCPL0 file CDRZCPL0 in library CBWORKDATA
have duplicate keys. Change the keys on all records that have the same key
as another record and try your request again.
Message ID . . . . . . : CPF0C4D
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:48:48
Message . . . . : Error occurred while processing object QSZPAVLI in library
QUSRSYS.
Cause . . . . . : While processing object QSZPAVLI type *PRDAVLI in library
QUSRSYS an error occurred which prevented the operation from completing
successfully. The return code is 0099-DAMAGED.
Recovery . . . : See the previously listed messages. Correct any errors,
and try the request again.
Message ID . . . . . . : CPF7001
Date sent . . . . . . : 03/09/16 Time sent . . . . . . : 02:48:49
Message . . . . : Damage was detected but has been ignored.
Cause . . . . . : Damage was detected during the requested operation.
However, the operation was completed.
Recovery . . . : Look at the job log (DSPJOBLOG command) for the messages
that identify the cause of the damage.
Thank You
_____
Paul Steinmetz
IBM i Systems Administrator
Pencor Services, Inc.
462 Delaware Ave
Palmerton Pa 18071
610-826-9117 work
610-826-9188 fax
610-349-0913 cell
610-377-6012 home
psteinmetz@xxxxxxxxxx<mailto:psteinmetz@xxxxxxxxxx>
http://www.pencor.com/
midrange.com
