RE: Restrict ability to alter variables in debugger on production -- MIDRANGE-L

> From: Jamie Coles
> 
> A bug may appear due to lack of thorough testing of an application -
not
> because production data wasn't used.

In a system of non-trivial complexity, the combinations of data
situations are high enough that not all combinations of testing paths
can ever be tested completely.  Each unit test may complete
successfully, but an unforeseen combination of data may produce
unforeseen results at the system level, usually because of business
rules that were not completely specified.
 

> I sell functional testing software - so whether the test is carried
out
> with live data or an extracted subset of live data that has the field
> contents scrambled should make no difference.  I can't immediately see
> where a bug would appear ONLY if live data were used, but would be
> willing to learn.

Simple situation, just to illustrate.  A company has certain guidelines
for orders: you must have filled out certain forms, you must be in our
special phone list database, and you must have a credit card with us.
OR, you must be a "special customer", with a flag set by the President
of our company, which says we can bypass these tests.

Unfortunately, the programmer read the specification to say "bypass all
tests for special customers".  One of the tests bypassed was credit hold
authorization.  Nobody ever decided what would happen if a President's
special customer was on credit hold, thus nobody even bothered to put
that test in a test scenario.  It would never be caught by any level of
unit testing, BECAUSE NOBODY THOUGHT IT COULD HAPPEN.

This is a very simplistic situation, but I hope you can extrapolate that
out to the myriads of business logic conditions that exist in any
non-trivial application and see how there are situations that simply
don't occur until the system goes live and users start actually using
the system.


> I can see possibilities of errors being found in load or stress
testing
> scenarios - but again I can't see the need to use live data as opposed
> to a scrambled copy of the complete real database.

This whole scrambling issue is a little strange to me.

It can only work at all if by scrambled you mean "scrambling key
identity fields" and "not scrambling data fields", but then how do you
determine which fields are scrambled and which are not?  If you scramble
the account number but not the account name, how hard is it for me to
figure out which company is which?  And if you DO scramble the identity
fields, how does the operator report to me which account is causing the
problem so that I can recreate the failure in the scrambled database?
And if the operator knows which account I'm dealing with, and I have the
scrambled data, and the operator tells me which account I'm REALLY
looking at, aren't I in effect looking at unscrambled data?

I'm so confused.

Joe