| Second Cache-off: Meeting (Old) |
|---|
| IRCache Cache-Offs |
Note: meeting results are discussed elsewhere.
Polyteam is hosting an organizational meeting for potential participants of the Second IRCache Cache-Off. If you intend to participate in the cache-off, please send us an e-mail and register for the meeting by August 8. You must register to attend. One person per cache-off participant please.
The meeting is scheduled for Tuesday, August 17th in Boulder, Colorado. This is a 9 hour event, starting 9:00am. Directions to NCAR Mesa Lab where the meeting will be held are available here. On-site lunch will be provided.
Participation in the meeting implies your intention to come to the second cache-off, but there is no obligation. We do not want to waste time accommodating Vendor V's custom wishes or complains if V has no intention to participate in the cache-off anyway. On the other hand, meeting attendees may get priority in cache-off slot assignment (given all other factors are equal) as a reward for their contribution.
In general, constructive input from non-participants is very welcome via Web Polygraph list or other channels, but we want to keep the meeting focused.
As of August 8, the following companies intend to participate in the second cache-off and are sending a representative to the meeting:
IBM and Netstor intend to come to the cache-off but cannot attend the meeting.
We intend to launch the second cache-off on Monday, October 25th. The duration of the event will depend on the number of participants and test rules, but is likely to take at least a week, with 3-4 day slots for each participant. We do not have a confirmed location yet. If you can provide a warehouse or similar size space with lots of power and appropriate cooling, please contact us.
We will rent Intel based PCs to build testing Clusters. The number of PCs will depend on the participants requirements and number of concurrent testing slots. We hope to find a reasonable balance between the cost of renting PCs and space/time limitations.
Web Polygraph version 2.x running on FreeBSD 3.1 will be used for the tests.
The Workload will consist of several independent runs, featuring performance and supplementary tests. The proposed test suite is described elsewhere. See also Polygraph workloads page for information about PolyMix-2, the performance workload. We will need to discuss parameters and details of the workload at the meeting.
Here are a few rules that should be discussed at the meeting.
Limiting the number of participants
Imagine a company A that develops caching software S and has N OEMs that sell that software bundled with their hardware. If all OEMs want to participate in the cache-off, we will have N entries running S. The boxes may or may not differ a lot in terms of hardware configuration (these differences are often hard to quantify). N may be as large as 6.
Technically, there is nothing wrong with more boxes from company A participating at the cache-off (each participant covers the cost of their participation). However, from marketing perspective, A's name may dominate the charts (regardless of the performance results). Such domination may be viewed by ``single Box'' vendors as an unfair advantage.
The questions for the meeting are:
- Should we place any limits on number of boxes running software developed by one vendor? What those limits should be?
- Should we place any limits on number of boxes running on hardware assembled by one vendor? What those limits should be?
- Should we place any limits on the total number of boxes participating in the cache-off?
Polyteam does not know good answers to the questions above. We hope to reach some consensus at the meeting.
Bail-out option
The first cache-off allowed a participant to bail at any time prior to publication of the results. There was no official penalty for doing so. Several vendors did choose to bail. When the bail-out option was discussed before the first cache-off, several vendors said they will not come to the cache-off if there is no bail-out provision (some of those vendors did not come anyway). However, participation in several trade magazine tests was not affected by the absence of bail-out option.
A bail-out provision allows a vendor to prevent publication of ``poor'' results. This is arguably a good side of the rule, especially if poor results were due to unexpected and unusual hardware or software problems. However, the same rule allows a vendor to cancel publication of ``good'' results because of some political/marketing reasons. The latter has negative impact on the cache-off image and contradicts the entire idea of open and friendly competition.
- Should there be an unconditional bail-out provision?
Polyteam suggests that we find a wording that allows a vendor to quit only in the event of clear hardware, software, or human failures. We suggest that the nature of those failures becomes known to the public, but no performance results are published if a vendor qualifies for a bail-out status.
Box maturity and minimum feature set
In general, cache-offs are open to any participant. IRCache cache-offs concentrate on performance and robustness features of the product and do not pay much attention to other important "features". The latter makes it possible for a participant to bring a box that can do little more than process simple Polygraph requests. It is probably a bad practice to compare such a crippled box with products that meet real life requirements. We call this issue a minimum feature set problem.
A similar problem exists with entries that have had no exposure to real workloads yet. A ``startup'' company may bring a product that is assumed to be ready for production use, but is likely to have many bugs just due to its limited exposure. Some of those bugs may be invisible for Polygraph, but may provide a performance boost for the product, effectively penalizing mature products.
- Should there be a minimum feature set requirement?
- How can we measure and/or report box maturity?
In attempt to address the product maturity issue, Polyteam suggests to report (for each participant) an approximate number of similar boxes in production use. We do not have a good solution for the minimum feature set problem besides publishing a ``product features'' table.
Boxes availability
The first cache-off rules require products to be available within six months from the data of the test. Our experience shows that the web-caching industry changes a lot within this six month period. It seems to be logical to introduce more rigid requirements on box availability.
- When boxes under test should be available for purchase?
Polyteam suggests a 2 month cut-off.
Polyteam has not received any offers to sponsor the event. Unless the situation changes, we are likely to follow the first cache-off pricing model. Each participant will be required to pay a participation fee. The fees will be proportional to the number of PCs required to create appropriate load conditions for a given product. The money will be used to pay for the rented PCs, space, Polyteam time and other cache-off expenses.
It is hard to accurately estimate the price tag at this point because we have limited knowledge about test conditions. We expect the fees to be around $2K-$3K per client-server pair.
Trade magazines often have to select winners when they publish our reports. Some magazines also have to assign weights to various categories. We think that such selections are arbitrary and misleading (at best) because they are based more on hand-waving than on actual performance results and product features.
Having cache-off reports published in magazines helps to reach larger audience and increases the likelihood of vendor participation. Note that magazines seem to have no money to sponsor events like cache-offs so money is not an issue here.
Two magazines has expressed their interest in publishing the second cache-off report: Boardwatch and Network World. Polyteam wants to hear vendor opinions on the presentation matters before finalizing the agreements with the magazines.
If you have any suggestions or comments relevant to the meeting please share them with us. Additions to the agenda are also welcome.
$Id: meeting.sml,v 1.5 1999/08/12 18:00:39 rousskov Exp rousskov $