"Cache-off" implies testing several independent implementations of similar products, taking place within a short period of time and usually at the same location. Every product is tested under the same conditions. Cache-off results are used to compare product performance. In our case, we will be evaluating the performance of Web caching proxies.

There are many reasons for the cache-off format. Our primary objectives are fair competition and audited tests.

Test labs, audited on-site tests, and even SPEC-like reports are considered to be ``fair''. That is, they give equal opportunities to participants to win. So what makes Caching cache-offs special?

The primary reason (as far as fairness is concerned) is highly competitive and unstable environment. New product releases and even companies appear virtually every month. Benchmarking workload improvements are also quite common. In such atmosphere, two test results obtained a few month apart are by default unfair to compare. A vendor who gets the last test ``slot'', essentially has a big advantage over the vendor who opened the test sequence. Thus, we have to test a lot of products in a short time interval.

Web proxy benchmarking often requires complex test setup that involves many sophisticated components like client-server simulators, L4 switches, and clusters of caches. Our experience shows that expertise and a lot of extra effort is required to guarantee the correctness of the setup. The auditing requires physical presence of the auditor during all stages of the tests.

To summarize, fair competition objective implies semi-concurrent execution of tests while test auditing requires human monitoring of test execution. It is currently infeasible to provide high quality auditing for dozens of isolated companies within a short period of time. Thus, only cache-off format produces fair and high quality results.

There are several nice side effects of the cache-off model. It is much easier to guarantee equal test conditions for everybody when the tests are executed under one roof and using the same equipment. It is also easier to get many results on exactly the same workload, especially keeping in mind frequent workload improvements. Finally, since proxy benchmarking rules are only being developed, they are not perfect and sometimes require adjustments or clarifications. It is often essential to have an ability to consult all participants at once and make a balanced decision about a questionable rule. Any other form of tests would make this virtually impossible because rule modification would be unfair to those who already completed the tests.

The unfortunate side-effects of cache-offs include the requirement for a participant to commit to a given cache-off date (which may not be in sync with product development cycle) and significant management problems for the hosting team.

We believe that the benefits of the cache-off model outweight its disadvantages. Cache-offs are currently the best option to give fair, high quality performance data to the customer.

January 17th, 2000.

At the cache-off meeting, vendors voted down the original Fall 1999 target. The majority of vendors (7 out of 13) felt that more time (about 8 weeks) is needed to verify and get used to the new workload and adjust their products if necessary.

Houston, Texas. A 50,000 ft² facility for the event was gratuitously provided by Compaq Corporation.

Anticipated duration of the cache-off is from two to three weeks, depending on the number of participants. We plan to have 20 - 40 hours worth of official tests per Box. Our experience shows that vendors need a lot of extra time to setup their caches and make sure they work OK.

The first cache-off had 10 hour test sequence. The last participant finished the tests after at least 72 hours, about 48 of which were under direct Polyteam supervision (three 16 hour shifts).

Any company or organization with a Web proxy product can request to be put on the participant list. Polyteam may decline any registration based on space and other reasons. Our goal is to test a wide variety of products within limitations of our resources.

The results will be published in the Network World magazine. Please contact us if you want to publish the results as we can split the publication into several semi-independent parts.