Recently, Charles Levine of Microsoft posted a response on the SQL Server performance blog, entitled TPC-E: Raising the bar on OLTP Performance, to my article from 3 October entitled The State of TPC-E. In this post, I’d like to respond to some of the statements Mr. Levine made in his commentary, which are indented below:
TPC-C is 16 years old and has changed little since 1992. Saying that we should continue using TPC-C because we know it so well is like saying that we should drive horse and buggies because we have a lot of expertise in blacksmithing. This is a mindset trapped in the past and doesn’t serve our customers.
TPC-C has stood the test of time. But today it is outdated, over-optimized, and of questionable relevance. Customers hold onto TPC-C because it is familiar and available, not because it is better. Database vendors need to exercise leadership.
Mr. Levine’s rhetoric is curious, given that Microsoft has out-published all other database vendors combined with TPC-C results at nearly a 2:1 ratio over the past eight years (163:86, see chart below). Moreover, Mr. Levine neglected to mention in his blog commentary that Microsoft has continued to publish TPC-C results — lots of them — since TPC-E was approved. Since January 2007, Microsoft has published 16 TPC-C results, while their TPC-E count stands at 18. The downward trend in the number of TPC-C benchmarks is evident from the chart below, but clearly Microsoft continues to support hardware vendors in their TPC-C benchmarking efforts using SQL Server 2005. The ten most recent TPC-C results, published since June 2008, include results on Dell, IBM, Hewlett-Packard, and Groupe Bull hardware and are composed of four tests of IBM DB2 UDB 9.5, one of Sybase ASE 15, one of Oracle 11g, one of SQL Anywhere 11, and three of Microsoft SQL Server 2005, including the most recent, published on 15 September 2008.
This recent history illustrates that both hardware and database vendors, including Microsoft, continue to see value in publishing TPC-C results. In this IBM whitepaper, TPC-E past-chairperson Trish Hogan states:
The move from TPC-C results to TPC-E results will be gradual. As clients and Marketing organizations become more familiar with and understand the value of TPC-E results, they will stop requesting TPC-C results. Over time the number of TPC-C results published will dwindle, clients won’t request them, and vendors will be reluctant to invest the money and resources required to publish them when a more affordable and more relevant benchmark is available.
We are not at that point yet; both hardware and database system vendors, including Microsoft, continue to find value in publishing TPC-C results and I stand by my opinion that we are still early in the transition away from TPC-C.
Now, is TPC-E a better benchmark than TPC-C? Mr. Levine, critical of my position on “E”, tries to imply that I am defending TPC-C on technical grounds. I am not. The additional complexity of TPC-E is welcome, particularly the introduction of additional data skew to the database instance. However, I believe that claims of TPC-E being “realistic” in comparison to production workloads in customer environments is overstated, as such claims were with TPC-C. TPC-C, being a write-intensive workload, stresses a database system’s logging, concurrency control and checkpoint/recovery systems. These are worthwhile components to test in a benchmark, but the overemphasis of update transactions makes the TPC-C benchmark unlike the vast majority of customer workloads . As Mr. Levine points out, TPC-E offers a larger and more complex schema, and an increase in the number of transactions and the total number of DML statements (assuming an implementation closely follows the benchmark’s pseudo-code). Since the transaction mix of TPC-E is different from TPC-C, TPC-E stresses server components differently than TPC-C.
I am not aware of a study similar to that of Hsu et al. that compares TPC-E workloads with a current sampling of production ones. I am also well aware that TPC-E, as it is a benchmark, is “composed of a set of basic operations designed to exercise systems functionalities, in a manner representative of most implementations, within the range of applications targeted by the benchmark” . Nonetheless, however, I believe that the low proportion of multi-way join queries in the TPC-E workload mix renders the benchmark unrepresentative of most customer workloads. Query processing — both optimization and execution, including memory management — is an extremely important component of any database management system. Consequently, I believe that for database vendors, moving to TPC-E as the “de facto” OLTP benchmark is, perhaps, not as compelling as it might be otherwise.
None of the transactions has changed in any way that impacts performance. All spec revisions have been classified as “minor” changes by the TPC and results across all spec revisions are comparable. The number of revisions to the spec since it was first released actually reflects a deep commitment by the members of the TPC-E committee to clean up rough edges and address areas of ambiguity before they become issues in published results. A better gauge of the high quality of the TPC-E spec is that to-date 18 results have been published by six vendors spanning 15 months, but there have been no compliance challenges.
Given the lack of TPC-E results from the rest of the database vendor community it is unsurprising to me that no compliance challenges have been raised, at this time. Moreover, while it is true that there have been few changes to the definitions of the benchmark’s transactions over the past 18 months, there has been considerable editing of the other sections that are more likely causes of such challenges; for example, the changes to Clause 7, Transaction and System Properties, with version 1.3.0 in September 2007.
This is not at all a criticism of the TPC-E committee; it is completely understandable that issues with the original benchmark specification would arise once formal attempts at publication are made. However, I believe that the complexity of the acidity and durability requirements as laid out in the TPC-E specification are impediments to its acceptance within the database community, and I know that on that point I am not alone.
Curiously, he doesn’t say why Sybase hasn’t published TPC-E results. Since he is, presumably, in a position to know, one can only conclude that he would rather not say. Readers can reach their own conclusions about what that might mean.
For Sybase iAnywhere’s benchmarking efforts over the past 12 months, we decided to exploit the TPC-C expertise within Sybase, and publish a competitive TPC-C result using the benchmark with the greatest representation from across the industry — including Microsoft.
Will TPC-E replace TPC-C as the de-facto OLTP benchmark? In all probability, the answer is yes. When that may occur, however, is anyone’s guess.
 W. W. Hsu, A. J. Smith, and H. C. Young (2001). Characteristics of production database workloads and the TPC benchmarks. IBM Systems Journal 40(3), pages 781-802.
 Francois Raab (1991). An Overview of the TPC Benchmark C: A Complex OLTP Benchmark. In The Benchmark Handbook, Jim Gray, ed., 2nd edition. Morgan Kaufmann Publishers, San Francisco.