| Second Cache-off: The Rules |
|---|
| IRCache Cache-Offs |
If you have any suggestions on improving the rules, please contact us.
A Product (as defined in the ``Terminology'' section) must satisfy all of the following criteria to be considered for the cache-off.
Each Participant must disclose the following:
This documentation must be made available prior to the start of the official test sequence. Polyteam will not start official tests unless the Participant provides all necessary details.
There will be no private rooms for test benches. Participants will be able to see each other equipment.
Working hours are 9am till 8pm with an hour lunch break. Experiments can be run overnight.
A Bench is logically divided into a Polyteam zone and a Participant zone.
Zones are connected using 100BaseT Ethernet cables. Specific requirements for
address allocation and network configuration are outside of the scope of this
document and are discussed elsewhere.
4.1 Polyteam Zone
Polyteam provides a number of machines to operate as clients and servers. Client machines have similar hardware and software configuration (identical to the extent possible). Server machines have similar hardware and software configuration (identical to the extent possible).
To manage the tests, Polyteam provides monitoring stations, one per Bench. The configuration of monitoring stations does not affect the tests.
The number of client and server pairs in a cluster is determined based one the desired workload (request rate). Some Benches may require more clients and servers than other Benches.
Polyteam is responsible for setup, operation, and maintenance of the client and server machines. The exact hardware and software specs will be included in Polyteam's cache-off report.
The Participant zone includes the Vendor's caching Product, cables, and any networking hardware (switch, router, hub) used to connect the client and server machines to the cache.
All equipment in the Participant zone must meet the ``Product Requirements'' described previously.
All equipment in the Participant zone, including networking hardware, is included in the list price of the Participant's configuration. Substitutions are not allowed. For example, a participant brings a 24-port switch and uses 8 ports, the price cannot be scaled down to an 8 port switch of the same manufacturer. Removing ``blades'' or ``cards'' from network gear is allowed if the gear can be bought without those cards.
The Participant is responsible for setup and maintenance of all the equipment in the Participant zone.
The Participant should specify any special accommodations (extra cooling, extra power outlets, etc.) as early as possible.
Participant zone must allocate at least one 10/100BaseT Ethernet port for the monitoring station.
Participant zone setup must provide IP level connectivity between the monitoring station and (clients and servers). For example, the monitoring station must be able to ``ping'' or ``ssh'' to all clients and servers.
Participant may request that their cables are used for networking clients and servers. Otherwise, Polyteam will provide cables.
In the event of performance, compatibility, or other problems between Polyteam and Participant zones, Polyteam and the Participant must work together to isolate and eliminate the problem.
In order to facilitate centralized management of all client and server machines, Polyteam may utilize a routed network that connects all Benches. This network is to be physically dismantled during performance tests.
All benches will be physically separated, and no connections between benches will be allowed (except for temporary network connections described above).
A sequence of tests is run independently and concurrently on every Bench. This section describes the test execution procedure for one Participant.
Official tests (discussed elsewhere) are:
The four tests listed above define the official test sequence. All other tests are unofficial.
Official tests must be executed in order listed above. Failure to complete a single test, invalidates the entire sequence.
A participant may repeat the entire sequence and choose the best results among all sequences executed. All sequences must start and finish within the time slot allocated for the participant (5 days).
Polyteam will treat requests to repeat a sequence with a lower priority than the request to execute the first sequence. Rerunning unexpectedly failed sequences will get higher priority that rerunning other tests. Polyteam may refuse to repeat the sequence if at least 3 attempts were made and/or there are reasons to believe further trials will not change the outcome of the tests.
A participant must choose one sequence to be reported as the cache-off result at the end of the participant slot. The results of other trials will be discarded.
Participants are not guaranteed any access to the benchmarking results during the tests. Reasonable efforts will be made by Polyteam to share run-time performance with Participants.
Polyteam archives all the logs from the official test sequence chosen by the vendor. On Participant request, Polyteam releases these results to the Participant after the cache-off is over (or earlier, time permitting).
In the event of equipment failure, the failed parts can be replaced, but the interrupted sequence (if any) is invalidated. The fact of the failure is not reported unless the failure leaves a participant with valid results.
Polyteam prepares an official cache-off report. The report is made available to participants no later than February 23rd. Polyteam will make reasonable efforts to discuss the content of the report with Vendors. The report, or its variation, will be published in at least one industry journal/magazine.
On February 28th, Polyteam makes all cache-off results publicly available on the Web at http://cachoff.ircache.net/. Polyteam's archive becomes an ``official site'' for distribution of the results. Polyteam must keep the archive available for at least one year.
Cache-off participants agree not to make public (or publicly discuss) the results of the cache-off prior to February 28th.
A Participant may request an inclusion of a ``Participant Comments'' entry into the report. The comments must not contain any references to external documents. The comments should only discuss matters relevant to the cache-off. Vendors are strongly discouraged to include speculations and accusations in the comments.
The comments are limited to 350 words and must be submitted in plain ASCII text format.
Once submitted, the comments cannot be modified.
Polyteam reserves the right to not include the Participant Comments.
Polyteam may add their own comments, up to three months after the final day of the Cache-off.
The Comments become a part of the official result distribution. These
Rules do not require any part of the results to be included in the derived
works.
6.2 Derivative works, publications, and such
Anyone can publish or otherwise make public any derivate work based on the cache-off results, provided that the ``Terms of use'' clause is satisfied.
Anyone can get a free copy of a Participant result from the
Official Site, provided they agree to the ``Terms of use''.
Terms of use
Wessels, Rousskov, Chisholm. The Second Caching Cache-off.
Houston, Texas; January 2000. Raw data and independent analysis at
http://cacheoff.ircache.net/
After the start of the first official sequence, a participant must not bailout or prevent their results from being published.
In the event of any form of conflict, including these Rules interpretation, Polyteam will be the final arbiter and judge. A reasonable effort on the part of a Polyteam will be made to find a compromise solution.