MIDRANGE dot COM Mailing List Archive



Home » MIDRANGE-L » December 2012

Re: Power outage unfun



fixed

Roberto,

Thanks for that info. I will look into that as well.

To all who responded, when I get ready to work on this in the weeks ahead,
I will start new threads. For example, there could be separate threads on
backup and power handling. This thread was about my company's experience.

For the first time ever, all the family Christmas get togethers and church
services occurred _before_ Christmas Day. So this will be a nice, relaxing
one.

Merry Christmas, everybody!


On Mon, Dec 24, 2012 at 3:54 PM, Roberto José Etcheverry Romero <
yggdrasil.raiker@xxxxxxxxx> wrote:

You mention 240V and some quite power consuming devices (freezers
whose compressors cost 20k) so it might not be feasible, but...
Here in Argentina you can find "voltage elevators and autoconverters"
which take line voltage (in our case, from 140 to 260V) and get it to
the right level (220V in our case). If it's too low or too high it
just cuts the output, thereby protecting whatever is behind them.
I've seen them rated for tens of KiloVoltAmper so it might be possible
to protect eveything with those. For my very small datacenter (2
racks, 2 disk enclosures, 1 bladecenter) i'm installing 3 units, one
on each phase, each able to bear the whole load (about 10kv leaving
room for upgrades).
In any case, with or without generator you DO need a way to cut the
power when this brownout situation occurs. And for that you need
something measuring the line voltages.
Best regards,
Roberto

On Mon, Dec 24, 2012 at 3:34 PM, Jeff Crosby <jlcrosby@xxxxxxxxxxxxxxxx>
wrote:
I will take all these suggestions and ponder them in my heart over the
next
several weeks.

We lost 1 freezer compressor for sure, and maybe a 2nd. Lost a number of
fans on freezer compressors.

Electrician hasn't even got to the air conditioner yet.



On Mon, Dec 24, 2012 at 10:22 AM, Pete Massiello - ML <
pmassiello-ml@xxxxxxxxxxxx> wrote:

Jeff,

You could do a few things.

First, you don't need SAVSECDTA & SAVCFG if you are doing a SAVSYS, they
are included in that. Secondly, once the SAVSYS is done, you don't
need to
be in Restricted state, so you could do a BRMS EXIT and bring up the
monitoring software. If you do that, I would backup that library
beforehand as a single item, then start the monitoring software. You
can
then exclude the one library on the *ALLUSR.

This way, you only won't be monitoring during the SAVSYS which
is
probably 20 minutes.

Merry Christmas,

Pete

--
Pete Massiello
iTech Solutions
http://www.itechsol.com
http://www.iInTheCloud.com





-----Original Message-----
From: midrange-l-bounces@xxxxxxxxxxxx [mailto:
midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of Jeff Crosby
Sent: Monday, December 24, 2012 9:30 AM
To: Midrange Systems Technical Discussion
Subject: Re: Power outage unfun

BRMS, run from the console. The control group entries are:

*SAVSYS
*SAVSECDTA
*SAVCFG
*IBM
*ALLUSR
QMPGDATA
*ALLDLO
*LINK

Do not currently do save while active.



On Mon, Dec 24, 2012 at 8:52 AM, Jack Kingsley <
iseriesflorida@xxxxxxxxx
wrote:

Are you running your monitor program in QCTL?? What type of backup
did you have running (21 save) ??

On Mon, Dec 24, 2012 at 8:34 AM, Jeff Crosby
<jlcrosby@xxxxxxxxxxxxxxxx
wrote:

All,

We had a power outage at work Fri 3:45am that lasted till 1pm. The
outage
was discovered by the Plant Manager when he arrived at 4:30am. (The
night
crew had finished loading trucks and left at 3:15am.) He got
everything opened, the drivers arrived, and hit the streets to
deliver. All in the dark.

The first thing that was really scary about this was the timing. We
have a
UPS that will power all the servers for 20-30 minutes. Software on
the System i monitors the UPS and does an orderly shut down in an
outage.
But
that software isn't running during the backup, which requires a
dedicated system. That backup starts at 3am and is finished at
4-4:05am. With 20+ minutes on the battery we should be OK, but . .
. So we had no idea what state the System i would be in.

When I&M (Indiana Michigan Power) "repaired" the blown fuse on a
pole
down
the street, restoring power, they didn't look for the root cause of
the outage. Never came into our building to check things or talk to
anyone.
So when power came back on, things were really strange. Brown out,
lights
were flashing like strobe lights. In rhythm. Like people who have
outside
Christmas lights timed to music. Never seen anything like it. At
least
1
freezer compressor started to smoke. At $20,000 per freezer
compressor, that was scary. We didn't start any computers at all
for fear of frying them.

We even thought we were going to lose the UPS. It kept cycling
on/off battery with some not so nifty panel messages about power
module
failures.
I finally made a call to the UPS people. They had me look at the
voltages
the UPS was receiving. We were getting 80-160 volts instead of 240
volts.
That explained a lot. The UPS tech arrived an hour or so later and
said he'd never seen a unit act like that. He was first afraid the
main
section
of the UPS was going because of the buzzing sound it made. Thought
he would have to have one couriered in overnight. After some
thought, his hunch was the low voltage was causing the issues at the
UPS, so we had to wait on the power company. We called in the
electrician who does our
work
and he confirmed the low voltage was an I&M issue.

I&M was no help over the phone on this. They couldn't give us any
idea when power would be restored. When another I&M line truck
finally
arrived
at 6pm (one guy in the truck), he found the real problem. The wires
between 2 poles across the street were too loose. The high winds
that
day
were causing them to touch, sending backfeeds into our building.
That
one
guy gave us more customer service that all the other I&M people put
together. He explained exactly what the problem was, what he was
going
to
do to fix it, and what the next step would be if that didn't resolve
the issue.

After he fixed that, all was well. He came back to our building to
confirm. (And he was none too happy with the crew that had been out
earlier.) We were getting 250 volts and the UPS went back to
normal. We brought up all the Windows servers and SAN and all was
well there. Our System i maintenance is with ServIT. That tech had
already been there twice during the day and was now at home. I
called him on his cell and
he
volunteered to come in right then, Friday night. I said let's wait
until Saturday morning when we'll both be fresh.

Saturday we brought the System i up in manual mode and everything
looked good. Brought it up normally and everything looked good. I
checked the BRMS backup log and the very last thing recorded was
"Save of list *LINK complete". I looked at the log from the
previous morning's backup and
the
very next thing in the log was the saving of media information. So,
in effect, we had a complete backup of the system. Hey hey. We had
all the sales people send in their Monday orders and staged on the
System i.

Sunday, management and several office people came in. We ran all of
Monday's business, did the A/R, deposited Friday's collections from
the drivers in about 2 hours. We have some really good people.
More people volunteered to come in than we needed. All hourly
people that came in
are
getting paid more hours than they actually worked plus we have
another surprise for them in the next week or 2.

It's Monday and there are no computer issues that I have found. One
freezer compressor won't come on. The newer A/C unit in the
computer
room
won't work, so we're using the old A/C unit for now. Glad it's not
summertime. Electrician on the way.

A couple of years back, the CEO had me get a quote for an automatic
backup
generator. Due to the expense, we didn't do it at that time. We
may
very
well do it now. With that generator running the office and the
computers,
we could have worked a normal day Friday and saved a lot of grief.
The
CEO
says he is going to sell it to the board.

One question: Is there any way whatsoever that we can get a
complete, full, don't-miss-anything backup and still have that UPS
software
running?

All in all, last Friday was the most harrowing day I've spent here
in 38+ years.

--
Jeff Crosby
VP Information Systems
UniPro FoodService/Dilgard
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531
www.dilgardfoods.com

The opinions expressed are my own and not necessarily the opinion of
my company. Unless I say so.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
mailing
list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please
take a moment to review the archives at
http://archive.midrange.com/midrange-l.


--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take
a moment to review the archives at
http://archive.midrange.com/midrange-l.




--
Jeff Crosby
VP Information Systems
UniPro FoodService/Dilgard
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531
www.dilgardfoods.com

The opinions expressed are my own and not necessarily the opinion of my
company. Unless I say so.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxx To subscribe,
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx Before posting, please take a
moment to review the archives at http://archive.midrange.com/midrange-l
.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.




--
Jeff Crosby
VP Information Systems
UniPro FoodService/Dilgard
P.O. Box 13369
Ft. Wayne, IN 46868-3369
260-422-7531
www.dilgardfoods.com

The opinions expressed are my own and not necessarily the opinion of my
company. Unless I say so.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.









Return to Archive home page | Return to MIDRANGE.COM home page

This mailing list archive is Copyright 1997-2014 by MIDRANGE dot COM and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available here. If you have questions about this, please contact