Re: Power outage unfun

I will take all these suggestions and ponder them in my heart over the next
several weeks.

We lost 1 freezer compressor for sure, and maybe a 2nd. Lost a number of
fans on freezer compressors.

Electrician hasn't even got to the air conditioner yet.



On Mon, Dec 24, 2012 at 10:22 AM, Pete Massiello - ML <
pmassiello-ml@xxxxxxxxxxxx> wrote:

Jeff,

You could do a few things.

First, you don't need SAVSECDTA & SAVCFG if you are doing a SAVSYS, they
are included in that. Secondly, once the SAVSYS is done, you don't need to
be in Restricted state, so you could do a BRMS EXIT and bring up the
monitoring software. If you do that, I would backup that library
beforehand as a single item, then start the monitoring software. You can
then exclude the one library on the *ALLUSR.

This way, you only won't be monitoring during the SAVSYS which is
probably 20 minutes.

Merry Christmas,

Pete

--
Pete Massiello
iTech Solutions
http://www.itechsol.com
http://www.iInTheCloud.com





-----Original Message-----
From: midrange-l-bounces@xxxxxxxxxxxx [mailto:
midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Jeff Crosby
Sent: Monday, December 24, 2012 9:30 AM
To: Midrange Systems Technical Discussion
Subject: Re: Power outage unfun

BRMS, run from the console. The control group entries are:

*SAVSYS
*SAVSECDTA
*SAVCFG
*IBM
*ALLUSR
QMPGDATA
*ALLDLO
*LINK

Do not currently do save while active.



On Mon, Dec 24, 2012 at 8:52 AM, Jack Kingsley <iseriesflorida@xxxxxxxxx

wrote:

Are you running your monitor program in QCTL?? What type of backup
did you have running (21 save) ??

On Mon, Dec 24, 2012 at 8:34 AM, Jeff Crosby
<jlcrosby@xxxxxxxxxxxxxxxx

wrote:

All,

We had a power outage at work Fri 3:45am that lasted till 1pm. The

outage

was discovered by the Plant Manager when he arrived at 4:30am. (The

night

crew had finished loading trucks and left at 3:15am.) He got
everything opened, the drivers arrived, and hit the streets to
deliver. All in the dark.

The first thing that was really scary about this was the timing. We

have a

UPS that will power all the servers for 20-30 minutes. Software on
the System i monitors the UPS and does an orderly shut down in an

outage.

But

that software isn't running during the backup, which requires a
dedicated system. That backup starts at 3am and is finished at
4-4:05am. With 20+ minutes on the battery we should be OK, but . .
. So we had no idea what state the System i would be in.

When I&M (Indiana Michigan Power) "repaired" the blown fuse on a
pole

down

the street, restoring power, they didn't look for the root cause of
the outage. Never came into our building to check things or talk to

anyone.

So when power came back on, things were really strange. Brown out,

lights

were flashing like strobe lights. In rhythm. Like people who have

outside

Christmas lights timed to music. Never seen anything like it. At
least

1

freezer compressor started to smoke. At $20,000 per freezer
compressor, that was scary. We didn't start any computers at all
for fear of frying them.

We even thought we were going to lose the UPS. It kept cycling
on/off battery with some not so nifty panel messages about power
module

failures.

I finally made a call to the UPS people. They had me look at the

voltages

the UPS was receiving. We were getting 80-160 volts instead of 240

volts.

That explained a lot. The UPS tech arrived an hour or so later and
said he'd never seen a unit act like that. He was first afraid the
main

section

of the UPS was going because of the buzzing sound it made. Thought
he would have to have one couriered in overnight. After some
thought, his hunch was the low voltage was causing the issues at the
UPS, so we had to wait on the power company. We called in the
electrician who does our

work

and he confirmed the low voltage was an I&M issue.

I&M was no help over the phone on this. They couldn't give us any
idea when power would be restored. When another I&M line truck
finally

arrived

at 6pm (one guy in the truck), he found the real problem. The wires
between 2 poles across the street were too loose. The high winds
that

day

were causing them to touch, sending backfeeds into our building.
That

one

guy gave us more customer service that all the other I&M people put
together. He explained exactly what the problem was, what he was
going

to

do to fix it, and what the next step would be if that didn't resolve
the issue.

After he fixed that, all was well. He came back to our building to
confirm. (And he was none too happy with the crew that had been out
earlier.) We were getting 250 volts and the UPS went back to
normal. We brought up all the Windows servers and SAN and all was
well there. Our System i maintenance is with ServIT. That tech had
already been there twice during the day and was now at home. I
called him on his cell and

he

volunteered to come in right then, Friday night. I said let's wait
until Saturday morning when we'll both be fresh.

Saturday we brought the System i up in manual mode and everything
looked good. Brought it up normally and everything looked good. I
checked the BRMS backup log and the very last thing recorded was
"Save of list *LINK complete". I looked at the log from the
previous morning's backup and

the

very next thing in the log was the saving of media information. So,
in effect, we had a complete backup of the system. Hey hey. We had
all the sales people send in their Monday orders and staged on the

System i.

Sunday, management and several office people came in. We ran all of
Monday's business, did the A/R, deposited Friday's collections from
the drivers in about 2 hours. We have some really good people.
More people volunteered to come in than we needed. All hourly
people that came in

are

getting paid more hours than they actually worked plus we have
another surprise for them in the next week or 2.

It's Monday and there are no computer issues that I have found. One
freezer compressor won't come on. The newer A/C unit in the
computer

room

won't work, so we're using the old A/C unit for now. Glad it's not
summertime. Electrician on the way.

A couple of years back, the CEO had me get a quote for an automatic

backup

generator. Due to the expense, we didn't do it at that time. We
may

very

well do it now. With that generator running the office and the

computers,

we could have worked a normal day Friday and saved a lot of grief.
The

CEO

says he is going to sell it to the board.

One question: Is there any way whatsoever that we can get a
complete, full, don't-miss-anything backup and still have that UPS
software

running?

All in all, last Friday was the most harrowing day I've spent here
in 38+ years.

--
Jeff Crosby
VP Information Systems
UniPro FoodService/Dilgard
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531
www.dilgardfoods.com

The opinions expressed are my own and not necessarily the opinion of
my company. Unless I say so.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing

list

To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at
http://archive.midrange.com/midrange-l.

--
Jeff Crosby
VP Information Systems
UniPro FoodService/Dilgard
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531
www.dilgardfoods.com

The opinions expressed are my own and not necessarily the opinion of my
company. Unless I say so.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.