Re: Horror stories (was New Mach -- MIDRANGE-L

From
MacWheel99@aol.com (Alister Wm Macintyre) (Al Mac)

Well I think I told you the funnyiest part of it ... the rest is just kind of 
sad repetition of variations on the same nonsense.

We IBM customers are familiar with preventative maintenance contracts.

We buy some equipment with a guarantee - if it fails, IBM will fix it, but we 
pay some monthly fees to IBM to maintain this promise.  IBM monitors rates of 
service calls & when some equipment has a lot of trouble, IBM works on what 
it takes to lower failure rate.  There is also stuff on the 400 that can call 
IBM before the end user sees the malfunction happening, because there are 
indicators that IBM can see.

Apparently a lot of customers of IBM competition do not follow this practice. 
 They pay for no preventative maintenance whatsoever (which reminds me of 
ANOTHER story at a major University, which I may tell you later - a much 
shorter one), and only pay to have things fixed when they break down.

Thus, trying to sell a management structure on the idea of having an 
inspection of all their equipment to see which is in SOON need of 
maintenance, and fixing any problems BEFORE they happen, can be a mental leap 
in a place that has never done preventative maintenance.

I have been trying without success my entire career almost to persuade a 
string of employers that a computer audit every 5-10 years should be a 
standard event to inventory stuff that needs fixing & prioritize the fixes.  
Places that do not do certain kinds of audits generally do not see the value 
in starting them.

If I told you the name of the Drug Store Chain & if you were a customer of 
Drug Stores 10-20 years ago, you would instantly recognize the name.  They 
were all over the USA & may also have been in Canada.  But they are all gone 
now.

I do not know if the broom handles were metal, I would have assumed wooden or 
plastic ... I got this story in pieces second and third hand ... I knew a 
person who worked at the place but some of what he knew was because he was 
not there when it happened, only in the build up & the aftermath.  

This was at a time that jobs were scarce for people who knew this stuff that 
had this kind of problem.  So my friend stuck around a while & kept me 
supplied with some juicy updates.

Apparently it was the laws in Pennsylvania that constrained them the most.

In other states, at the time, when there is some problem with the computer 
system that keeps track of the prescription drug data base, they could shut 
down the pharmacy part of the store & keep the rest of the store going, but 
in Pennsylvania they had to shut down the whole store, and they had hundreds 
of outlets all over the state, so it cost them a bundle for every hour that 
these 24x7 convenience stores were not open for business.

(This also reminds me of the story of a major bank in New York that issued 
government bonds by computer & ran out of numbers.  I think it was the most 
expensive computer disaster prior to Y2K because they were also constrained 
by law ... they had to keep issuing money to the Fed, but they could not sell 
the bonds, so they had to borrow money from other banks to stay afloat until 
teams of Cobol programmers fixed a problem that anyone could have seen coming 
in ample time to avoid a crisis - anyone who was paying attention to what was 
happening with the numbers ... which came to a week of carrying the US 
national debt on the shoulders of high interest short term loans.)

The event occurred with 3rd shift crew, who phoned their bosses once they 
hacked their way to a working telephone & the bosses called their bosses & 
the CEO was on site at like 4 am & there were long distance phone calls to 
the vendors of all their equipment & general air charter flights to get 
relevant technicians on site ASAP.  The scenario was spare no expense to get 
us operational again, but ... hey nothing in the budget for anything other 
than getting operational again ASAP.

Not in the budget was fixing the air conditioning plumbing which caused the 
problem in the first place.  It is my understanding that some employees 
figured out that the company could not afford to have this happen more than 
once, in other words IT IS GOING TO HAPPEN AGAIN & when it does it will 
bankrupt the company & we will all be out of our jobs, so the air 
conditioning plumbing was fixed through public subscription to an employee 
fund, without going through the process of getting something into the budget, 
considered a lost cause.

Not in the budget was preventative maintenance on the hard drives which had 
been shaken up pretty badly in this crisis.  They had over 100 of them total 
& over 50 failed within the next six months, in which it was estimated by the 
technicians that most of  them could have been saved by expending proper 
maintenance ASAP.  

This was in the days when a hard drive cost a pretty penny.

When you have a hard disk failure, it brings down more than the actual 
hardware that is failing, and when you have a lot of these events, it has a 
cascade effect of messing up equipment that was not messed up by the big one, 
including some of the replacement hard drives that arrived after the big one.

But the mandate was not to to take reasonable measures to avoid problems, the 
mandate was to do whatever was neccessary to get those stores back open 
again, and spend no more money than absolutely neccessary to accomplish that.

Each time one of those disk drives crashed, they had several hours shut down 
... 50 some events in the 6 months after the big one.  Middle Management was 
incapable of learning from this.  As far as they were concerned, each event 
was a total surprise, in which we needed to get it resolved as quickly as 
possible, then move on.

My informant got a new job & was in touch with co-workers still there, but 
our contacts dried up, so for all I know there were more such events after 
the 6 months but I was no longer in the information loop.

I know some other stories ... years ago I used to do a seminar at computer 
conferences "Learning from Computer Disasters" in which each disaster that I 
shared had some lesson of stuff you should be doing to avoid this kind of 
thing happening to you & the disaster stories drove in the lesson of why some 
stuff really important like backup.

For example there was this company that was a neighbor of where I worked 
years ago.  They never had anything serious go wrong, so they could not be 
bothered to do backups.  Then they had a hard disk crash & lost all their 
on-line data.

They did however have print-outs associated with end-of-month & other stuff 
associated with their business.  They hired an army of data entry people to 
reconstruct their data base from the reports.  I once had some statistics 
about the volume of data involved & the rate at which people transcribe & 
this was before scanners were widely available.

Some stories there was no obvious lesson, or kinda libelous to identify the 
players.
I want to be sharing these stories ... I do not want to be getting in trouble 
with the players.  Thus I take pains to disguise precisely where this or that 
happened.

Some stories rooted in technology we not have around any more, but some are 
human nature that could happen any place.

I think bears repeating are the stories about 
Payroll at the Law Office
Punched Cards in Voting
State Tax Agencies
Legalizing driving with expired license plate
Infinite Theft of same Auto
Infinite Crooked Cash Flow
Membership Numbers
Flunked Computer Science

The above lines are reminders where I should remember which story that was 
... Hey Mac do you have time to tell us the story about ....

------------------------- Original Message --------------------------
From:   D.BALE@handleman.com

>>the computer operators were forever mopping up 

Geez, the first time that happened, I'da been looking for a new job.

Wow.  I thought I've seen a lot.  And that was only the beginning?  Sounded
like the very end to me!  Please, daddy, finish the story...  (the line I get
most nights)

Dan Bale

-------------------------- Original Message --------------------------
From
MacWheel99@aol.com (Alister Wm Macintyre) (Al Mac)

I have another story like that without quite the same happy ending.

A friend of mine worked at a place that sounded like they needed a mainframe
as they had obviously outgrown what could be done with what they had from a
non-IBM supplier.  Each controller could handle 8 disk controllers & each
disk controllers could handle 8 disk drives & each disk drive had two
connections for data access & they had 3 computers - the live one used to
manage a chain of drug stores that is no longer in business, which is no
surprise to me, the one used for development work, and the backup computer.

Under the laws of states they doing business in, they could not be issuing
prescriptions unless connected to computer data bases of drug interactions &
what other stuff the patients taking.  When the primary computer went down,
which was quite often, they ran around the computer room like crazy
connecting key disk drives to a second computer to take over the load,
rotating which was backup, and stopping development when long time no repair.

This stuff was in a sub basement with an air conditioner that leaked into the
false floors, and the computer operators were forever mopping up down there &
begging management to have the air conditioner plumbing fixed, but were
always told by the pointy haired executives that there was no budget for this
& to quit bothering them for this when they had previously been told there
was no budget for it.

Well one day there was a short circuit thanks to this.
A lot of equipment died.
The UPS thought external power had gone out & kicked in but failed because
the short circuit was on the computer side of the power supply.
There was a momentary surge through the short circuit that brought a lot of
hard drives up temporarily, then they died again.
The emergency generator thought the same thing.
history repeated.
There was a third deal ... the UPS was backed up like the computers.

This led over the months ahead to a series of hard disk crashes, and
attendant disruptions, because there was no budget for hard disk maintenance
to see which needed work in the aftermath of this.  The ultimate cost to the
corporation was in the millions of dollars, because there was no budget to
fix the air conditioning which could have avoided the whole mess, and it was
kind of foolish to put the computer center in a sub basement in the first
place.  I am accustomed to living places that occasionally have flooding, or
bad weather that drowns basements.

People in the computer room reported a red glow from the floor, comparing it
later to some Star Trek episode.

The building used a security system where you needed key cards to go through
doors, but that was out because it ran off the computer system that was now
down.  Fortunately there was a fire axe accessible to the computer room,
which they used to hack their way OUT of the building & phone call for help.

This is a long story & this is only the beginning of it.
I am laughing too hard from this little memory to tell you any more.

MacWheel99@aol.com (Alister Wm Macintyre) (Al Mac)


+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---