RE: Liars Figure and Figures Lie - Sybase ad (LONG)

  Well, I did some checking on this ad.  Here is the table from the press
release.  It will
probably be a mess but I will reference the AS/400 numbers again:


>      Database                    TpmC      $per TpmC     CPUs
Hardware Platform
>      Sybase A S E 11.5    52,117     82                      16
            HP 9000 V-Class Enterprise Server
>      Oracle                           51,871     135
44            Sun Enterprise 6000 cluster
>      Oracle                           34,117     N/A                   18
            Fujitsu/ICL GRANPOWER 7000
>      IBM DB2                       25,150     128                    12
            AS/400 e
>      MS SQL Server 6.5   14,900     60                         8
           NEC Express5800 HV8000

Again, from the press release there is the following statement:

>With this result, HP and Sybase have achieved the fastest transaction
processing
>capabilities in the industry (based on published Transaction Processing
Performance
>Council Benchmark C (TPC-C) results)

The key here is the word 'published'.  I checked the TPC site (www.tpc.org)
 for information.
Checking for published results as of 3/20/98 here is what I found.  Two
AS/400 models, the
S40 and S30 which had published dates of 8/18/97.  The machine quoted in
the ad is indeed
an S40-2261 (12-way).  At the TPC site the S40 had numbers of 25,149.75 and
 $128 for TPC-C
throughput and Price/Performance.  Compare this to the 25,140 and 128
listed in the ad and
you do have a match.

I'd love to hear IBM's explanation about this......but you and I both know
that these bench mark
numbers must take a lot into consideration.  For example, the cost of the
HP/Sybase machine
is $4,229,894.  The cost for the S40 is $3,217,385.  I don't know about you
 but that seems to take
the relevance of the number for most customers and place it somewhere near
Pluto.  Take a
look at the site and you will be amazed with the configurations.  I'm not
an expert on TPC-C
by any stretch but these configurations are enormous.  The HP/Sybase
configuration is the
HP9000 C2250 front ended by 12 HP9000 D230.  The S40 is front ended by 64
150's!  The common
complaint with benchmarks is that, while it may tell us about certain
characteristics
of a machine that are not very real world.

Some random thoughts.

The typical TPC-C configuration is typically as follows:

Driver System -------->  Client System ------------> Database Server
(PC/Term)

>From a cost perspective the IBM cost does get driven up by the 150's.  Of
course IBM is using
COBOL and CICS.....doesn't that stuff slow down any AS/400?  :-)

I am a little perplexed as to why someone would use 150's and an S40 in a
OLTP benchmark.
Wouldn't traditional systems work better?

I remember a very good point made by Dr. Soltis at San Antonio regarding
artificial optimizations - ie: vendors modifying their HDW/SFW specifically
 for the
purpose of improving a benchmark.

TPC-C was approved sometime in 1992.......relevance?

Quoting from the TPC-C Benchmark Specification I would offer the following:

>TPC-C uses terminology and metrics that are similar to other benchmarks,
>originated by the TPC or others. Such similarity in terminology does not
in
>any way imply that TPC-C results are comparable to other benchmarks.  The
>only benchmark results comparable to TPC-C are other TPC-C results
>conformant with the same revision.  Despite the fact that this benchmark
>offers a rich environment that emulates many OLTP applications, this
>benchmark does not reflect the entire range of OLTP requirements. In
>addition, the extent to which a customer can achieve the results reported
>by a vendor is highly dependent on how closely TPC-C approximates the
>customer application. The relative performance of systems derived from
>this benchmark does not necessarily hold for other workloads or
environments.
>Extrapolations to any other environment are not recommended.
>Benchmark results are highly dependent upon workload, specific application

>requirements, and systems design and implementation. Relative system
>performance will vary as a result of these and other factors. Therefore,
>TPC-C should not be used as a substitute for a specific customer
application
>benchmarking when critical capacity planning and/or product evaluation
>decisions are contemplated.

Of course, we should remember that the AS/400 holds 7 of the top 10
RPMarks!

Benchmarketing and benchmark wars can really turn into a, excuse my
language,  pissing contest with no real winners.  What I'm most worried
about is
how many in-flight magazines this will show up in.......

Oh well, I've taken up too much time and hopefully not too much
bandwidth....


Michael Crump
Technical Project Leader
Ball-Foster Glass Container Corp.



+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---