Re: Test environments (was: upgrade)

A couple generic comments in-line...


midrange-l-request@midrange.com wrote:

>From: "Jamie Coles"
>
>  So the old tests have been recorded and can be played back whilst you record 
>new
>ones, and then these get played back when you record the next set.

For many basic tests, something as simple as a recorded ClientAccess macro 
could be used to automate testing. Not for everything, but for testing that 
each menu option actually invoked a program, that the logged-on user didn't run 
into authority errors, etc.

Simple macro recording and playback isn't used nearly to full advantage for 
testing. With modification, the macros could be made very intelligent. Much of 
this only needs doing once.


>One of the main problems with testing or running parallel with the old and the 
>new system is the extra amount of time needed to
>check the test results.

On contract, I rewrote a Local Improvement District billing application for a 
City a number of years ago. The IS manager resigned to take a better position 
towards the end of the project, just before full system testing began. The new 
manager insisted on full parallel testing until everything checked out in 
detail (which had much merit in concept) as the basis for "Acceptance".

Because a reconciliation between old and new took major time for City staff and 
because LID billing was on a monthly cycle, parallel testing was a major 
problem. Surprise! The old and new didn't match by significant margins at many 
points. A few items were found easily by me and corrected in the new system, 
but major items remained that took a lot of detail checking over years of 
history. They started seeing that the big problems were mostly because the old 
system had always had major flaws that were never detected.

So, the month would pass while reconciliation went on. A new month-end would 
arrive and a new parallel also. And corrections for old errors would allow new 
errors to be uncovered. After the first parallel, no significant errors were 
ever discovered in the new version; all were errors in the old. In one major 
case, an LID was found that had _NEVER_ been billed to the property owners for 
a number of years. You can imagine the political difficulties for the City in 
getting things current there.

But the parallels continued to be mandated. Meanwhile, my contract payment is 
delayed for months awaiting "Acceptance".

Ever since, I've been extremely leery of the value of parallel testing without 
clear guidelines.


>Once an error has been spotted the next difficulty is providing the developer 
>with the actual scenario that led to the error being
>created so that it can be replicated and then fixed - not always an easy task 
>- because us users never really know if we pressed
>"enter" or "space" !!

A very good point to bring up. Worth keeping in mind. We've all had users say 
"No, I'm _POSITIVE_ I hit <F16>, not <F4>." Automated testing bypasses this.


Tom Liotta

--
Tom Liotta
The PowerTech Group, Inc.
19426 68th Avenue South
Kent, WA 98032
Phone  253-872-7788
Fax    253-872-7904
http://www.powertechgroup.com



__________________________________________________________________
Your favorite stores, helpful shopping tools and great gift ideas. Experience 
the convenience of buying online with Shop@Netscape! 
http://shopnow.netscape.com/

Get your own FREE, personal Netscape Mail account today at 
http://webmail.netscape.com/