SRC10001520 - Power Supply AC fault
SRC78D1152A - A fatal error occurred on power supply 2.
SRC11001540 - Detected AC loss
One lpar got the first.
The hosting lpar got all of them.
The second one is a "call home" incident. IBM informed me that numerous customers in the area (Kentwood MI) had similar call home incidents at the same time which indicate a power failure in the area.
We didn't lose any function as our Power 9 there had dual power supplies. Each power supply goes to different UPS, generator, power supplier (I think on that last one) in one of those lights out data center colocation facilities.
Well there was a single power source firewall which was an issue but I digress...
Everything seems to be working fine. IBM says there really is no way to remotely tell if the power supply is really having an issue. The data here:
WRKPRB
System
Problem ID Ref Code Date Time
2112800286 SRC78D1152A 05/08/21 00:04:58
2112800285 SRC78D11529 05/08/21 00:04:58
2112800248 SRC78D1152A 05/08/21 00:04:18
2112800247 SRC78D11529 05/08/21 00:04:18
2112800126 SRC10001520 05/08/21 00:02:12
2112800070 SRC11001540 05/08/21 00:01:13
May all be related to a temporary power loss to one of the power supplies in the unit.
From IBM:
<IBM>
Hey Rob,
So as far as for us there's no really way to tell if it's the source or the supply itself. That being said we have had multiple tickets in the area for power outage related issues. If it's managed and you can remote into your hmc, you may be able to see if it came back up when it swapped back, but even then it can give similar symptoms on both sides of the coin. That being said I just received a ticket for the psu, so the box is still up, just has one of the redundant psus we can order and have on-hand for Monday if you would like.
</IBM>
How do I tell if "if came back up when it swapped back"?
Do I need to go onsite and look for amber lights on the power supply itself?
No system attention lights are indicated on the HMC.
From the HMC:
Problem #, PMH #, Reference code, Status, Last reported time, Failing MTMS
124, ,78D1152A,Open,May 8, 2021 12:03:26 AM,ESLS-001
123, ,78D11529,Open,May 8, 2021 12:03:25 AM,ESLS-001
122, ,78D1152A,Open,May 8, 2021 12:02:46 AM,ESLS-001
121, ,78D11529,Open,May 8, 2021 12:02:45 AM,ESLS-001
120, ,10001520,Open,May 8, 2021 12:00:38 AM,78CD-001
119, ,11001540,Open,May 7, 2021 11:59:40 PM,9009-41A
Back to the single power source firewall...
Networking guy said: Came back online after I went to bed at 1:27AM.
Which, for a colocation data center which is supposed to have all sorts of UPS, generators, etc is a long time to be without power. Networking guy is in the same timezone.
Rob Berendt
--
IBM Certified System Administrator - IBM i 6.1
Group Dekko
Dept 1600
Mail to: 7310 Innovation Blvd, Suite 104
Ft. Wayne, IN 46818
Ship to: 7310 Innovation Blvd, Dock 9C
Ft. Wayne, IN 46818
http://www.dekko.com
midrange.com
