This informational document discusses our intentions regarding presentation of the bake-off results.

2. Approach

We should try to come up with a compact but informative presentation format. The presentation should depict major (strong and weak) points of tested Boxes.

Targeted audience for our presentation is people with at least intermediate level of knowledge about Web proxies. It is not our goal to describe the differences between URLs and Pages, Response Time and Throughput, etc. We expect the readers to understand the importance of Document Hit Ratios and Price Performance Ratios. Hence, our task is to present the data in a form that assists knowledgeable readers in making their conclusions rather than dictating those conclusions to them.

There are many ways to interpret the results. No presentation format will satisfy everybody. We do not claim that our format is the only correct one. Vendors may choose to present the results differently. Readers may choose to consult raw data.

3. Intended Format

Here is a tentative outline of the Polyteam report. The report will be published in Proceeding of the Fourth Web Caching Workshop and will be available on Bake-off Web site.

1. Introduction.
2. Benchmarking methodology.
3. Benchmarking strategy chosen for the bake-off (including pros, cons, and reasons for selecting a particular test). Limitations of these tests; why the test workload does not match real-world situations (aka the ``laundry list'').
4. The Setup (what hardware/software was used, etc.)
5. The Results. Comparisons of different products. Graphs where several Participants are compared with each other. Table of Price vs Performance for each Participant. Discussion.
6. Plans (Future Work).
7. Conclusions.
8. Appendix: Executive summaries for each Participant. (see subsection below)

Besides publishing ``Executive summaries'', we would also like to prepare a one page summary for all Participants combined. However, it may be infeasible to fit all essential info into one page. The feasibility will depend, in part, of the actual results.

3.2 Details of Results

For every participant, we may present the following:

• Raw TCP performance results (with ttcp or similar). Baselines the Cluster's network performance.
• No-Proxy test results. Also baselines the Cluster's network performance, but in a manner very similar to the cache macro benchmark.
• Macro benchmark results. Mean or Median response time versus offered load.

To compare all Participants, Polyteam will generate a table of Price/Performance ratios. The Performance figure is taken from the macro test results. This is the request rate which the Box sustained, with an mean response time of 6 seconds.

The performance is divided by the total cost of all equipment in the Participant Zone (Box, router, switch, etc). The units of the price/performance number is: requests per second per thousand dollars.

3.3 Executive summaries

Executive summaries that are described here are generated for each Participant using similar format. We limit ourselves to a one page summary per Participant.

We assume that there will be only one major test during the first bake-off (the PolyMix#1 macro test).

The final presentation form will depend on the results. Here are a few items that we intend to show based on our current knowledge.

• Cost Report (a table with the cost for major components of the equipment in the Participant Zone). This table can also be used to understand what kind of proxy was tested (if a Vendor is offering more than one product).
• Mean or median Response Time (msec) versus Offered Load (req/sec) graph. X-axis may also have MBytes/sec ticks. Y-axis (or individual data points) may also have measured Document Hit ratio (%). We can also show confidence intervals (error-bars) for the measured data.
• A table or graph with the number of xaction errors during the test.
• Expected availability date for the Box.
• Participant comments (one short paragraph from the Participant Comments in the Results distribution) if any
• Polyteam comments (one short paragraph from the Polyteam Comments in the Results distribution) if any
• If space allows, we may include ``trace'' graphs that show variation of reported averages with time during the test.
• If space allows, we may include measured response time distribution; perhaps with separate curves for hits, misses, and ``all''.