Re: Power outage unfun

Jeff,

That sounds like a nightmare to live through, I'm glad it was all resolved. The power handling program will not be running during a full system save. If needed you can mitigate the downtime by only doing a *SAVSYS (the only part that truely needs restricted state) when applying PTFs since that's really the only time you need to do it. I usually recommend once or twice a year unless PTFs have been applied.

Also in your control group below you are getting security and configuration twice, no real need to spend the extra time.

I would suggest you change to Save While Active and use the same control group you have below, except drop the *SAVSYS (only run when needed) and don't bother with QMPGDATA, or QPFRDATA. Neither are needed for a recovery anyway. I would even exclude them from the *ALLUSR. There will be some files in the IFS that are log files and flight recorders (BRMS has one) that will not back up either, but again, those will be recovered by the software that uses them and they are not needed in the event of a full recovery anyway.

Now you can end all the subsystems that have user work going, perform the back up and restart while the power handling program is still running.

Jim Oberholtzer
Chief Technical Architect
Agile Technology Architects


On 12/24/2012 8:29 AM, Jeff Crosby wrote:

BRMS, run from the console. The control group entries are:

*SAVSYS
*SAVSECDTA
*SAVCFG
*IBM
*ALLUSR
QMPGDATA
*ALLDLO
*LINK

Do not currently do save while active.



On Mon, Dec 24, 2012 at 8:52 AM, Jack Kingsley<iseriesflorida@xxxxxxxxx>wrote:

> Are you running your monitor program in QCTL?? What type of backup did you
> have running (21 save) ??
>
> On Mon, Dec 24, 2012 at 8:34 AM, Jeff Crosby<jlcrosby@xxxxxxxxxxxxxxxx

> >wrote:

>

> > All,
> >
> > We had a power outage at work Fri 3:45am that lasted till 1pm. The

> outage

> > was discovered by the Plant Manager when he arrived at 4:30am. (The

> night

> > crew had finished loading trucks and left at 3:15am.) He got everything
> > opened, the drivers arrived, and hit the streets to deliver. All in the
> > dark.
> >
> > The first thing that was really scary about this was the timing. We

> have a

> > UPS that will power all the servers for 20-30 minutes. Software on the
> > System i monitors the UPS and does an orderly shut down in an outage.

> But

> > that software isn't running during the backup, which requires a dedicated
> > system. That backup starts at 3am and is finished at 4-4:05am. With 20+
> > minutes on the battery we should be OK, but . . . So we had no idea what
> > state the System i would be in.
> >
> > When I&M (Indiana Michigan Power) "repaired" the blown fuse on a pole

> down

> > the street, restoring power, they didn't look for the root cause of the
> > outage. Never came into our building to check things or talk to anyone.
> > So when power came back on, things were really strange. Brown out,

> lights

> > were flashing like strobe lights. In rhythm. Like people who have

> outside

> > Christmas lights timed to music. Never seen anything like it. At least

> 1

> > freezer compressor started to smoke. At $20,000 per freezer compressor,
> > that was scary. We didn't start any computers at all for fear of frying
> > them.
> >
> > We even thought we were going to lose the UPS. It kept cycling on/off
> > battery with some not so nifty panel messages about power module

> failures.

> > I finally made a call to the UPS people. They had me look at the

> voltages

> > the UPS was receiving. We were getting 80-160 volts instead of 240

> volts.

> > That explained a lot. The UPS tech arrived an hour or so later and said
> > he'd never seen a unit act like that. He was first afraid the main

> section

> > of the UPS was going because of the buzzing sound it made. Thought he
> > would have to have one couriered in overnight. After some thought, his
> > hunch was the low voltage was causing the issues at the UPS, so we had to
> > wait on the power company. We called in the electrician who does our

> work

> > and he confirmed the low voltage was an I&M issue.
> >
> > I&M was no help over the phone on this. They couldn't give us any idea
> > when power would be restored. When another I&M line truck finally

> arrived

> > at 6pm (one guy in the truck), he found the real problem. The wires
> > between 2 poles across the street were too loose. The high winds that

> day

> > were causing them to touch, sending backfeeds into our building. That

> one

> > guy gave us more customer service that all the other I&M people put
> > together. He explained exactly what the problem was, what he was going

> to

> > do to fix it, and what the next step would be if that didn't resolve the
> > issue.
> >
> > After he fixed that, all was well. He came back to our building to
> > confirm. (And he was none too happy with the crew that had been out
> > earlier.) We were getting 250 volts and the UPS went back to normal. We
> > brought up all the Windows servers and SAN and all was well there. Our
> > System i maintenance is with ServIT. That tech had already been there
> > twice during the day and was now at home. I called him on his cell and

> he

> > volunteered to come in right then, Friday night. I said let's wait until
> > Saturday morning when we'll both be fresh.
> >
> > Saturday we brought the System i up in manual mode and everything looked
> > good. Brought it up normally and everything looked good. I checked the
> > BRMS backup log and the very last thing recorded was "Save of list *LINK
> > complete". I looked at the log from the previous morning's backup and

> the

> > very next thing in the log was the saving of media information. So, in
> > effect, we had a complete backup of the system. Hey hey. We had all the
> > sales people send in their Monday orders and staged on the System i.
> >
> > Sunday, management and several office people came in. We ran all of
> > Monday's business, did the A/R, deposited Friday's collections from the
> > drivers in about 2 hours. We have some really good people. More people
> > volunteered to come in than we needed. All hourly people that came in

> are

> > getting paid more hours than they actually worked plus we have another
> > surprise for them in the next week or 2.
> >
> > It's Monday and there are no computer issues that I have found. One
> > freezer compressor won't come on. The newer A/C unit in the computer

> room

> > won't work, so we're using the old A/C unit for now. Glad it's not
> > summertime. Electrician on the way.
> >
> > A couple of years back, the CEO had me get a quote for an automatic

> backup

> > generator. Due to the expense, we didn't do it at that time. We may

> very

> > well do it now. With that generator running the office and the

> computers,

> > we could have worked a normal day Friday and saved a lot of grief. The

> CEO

> > says he is going to sell it to the board.
> >
> > One question: Is there any way whatsoever that we can get a complete,
> > full, don't-miss-anything backup and still have that UPS software

> running?

> >
> > All in all, last Friday was the most harrowing day I've spent here in 38+
> > years.
> >
> > --
> > Jeff Crosby
> > VP Information Systems
> > UniPro FoodService/Dilgard
> > P.O. Box 13369
> > Ft. Wayne, IN 46868-3369
> > 260-422-7531
> > www.dilgardfoods.com
> >
> > The opinions expressed are my own and not necessarily the opinion of my
> > company. Unless I say so.
> > --
> > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing

> list

> > To post a message email:MIDRANGE-L@xxxxxxxxxxxx
> > To subscribe, unsubscribe, or change list options,
> > visit:http://lists.midrange.com/mailman/listinfo/midrange-l
> > or email:MIDRANGE-L-request@xxxxxxxxxxxx
> > Before posting, please take a moment to review the archives
> > athttp://archive.midrange.com/midrange-l.
> >
> >

> --
> This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
> To post a message email:MIDRANGE-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit:http://lists.midrange.com/mailman/listinfo/midrange-l
> or email:MIDRANGE-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> athttp://archive.midrange.com/midrange-l.
>
>

-- Jeff Crosby VP Information Systems UniPro FoodService/Dilgard P.O. Box 13369 Ft. Wayne, IN 46868-3369 260-422-7531 www.dilgardfoods.com The opinions expressed are my own and not necessarily the opinion of my company. Unless I say so.
--