If you have any suggestions on improving the rules, please contact us.

Table of Contents

1. Terminology
2. Requirements for Participation
    2.1 Requirements on Products
    2.2 Autogenerated Report
    2.3 Disclosures
3. Access to the cache-off facility
4. Benches
    4.1 Polyteam Zone
    4.2 Participant Zone
    4.3 Inter-Zone communication
    4.4 Inter-Bench Network
5. Testing
    5.1 Official tests
    5.2 Tests execution
    5.3 Results selection
    5.4 Equipment failures
6. Cache-off Report
    6.1 Comments on the Results
    6.2 Derivative works, publications, and such
7. Bailout and Failures
8. Conflict Resolution

Vendor

An organization bringing a caching product to the cache-off. To simplify the terminology all commercial, non-profit, virtual, etc. organizations are labeled as ``vendors.''

Product under test

A caching product/solution/box being tested as a single proxy cache. A product includes software, hardware, network gear, etc. required for proxy operation. A product may include several caching units.

Entry

A (Vendor, Product) tuple.

Participant

A Vendor associated with a particular Entry.

Test Bench

A group of machines and network gear dedicated to benchmarking an entry, including the Product.

Polyteam

The team responsible for Web Polygraph development and cache-off organization.

A Product (as defined in the ``Terminology'' section) must satisfy all of the following criteria to be considered for the cache-off.

• The Product must be available for purchase no later than three months after the start of the cache-off.
• The Product proxy must have a way of completely flushing (emptying) the cache so that no old cached objects remain stored.
• The Product proxy must have an interface to report the portion of cache disk space in use.
• The Product's TCP stack must use a Maximum Segment Lifetime (MSL) value of 30 seconds, which corresponds to a ``TIME_WAIT'' timeout (2*MSL) value of 60 seconds.
• The Product's vendor, specifications, and all configuration details must be public at the start of the cache-off.
• Product participation fee must be paid (received by Polyteam) in full before the start of the first official test for the corresponding entry.

A hard copy of an auto-generated Polygraph report should be presented to Polyteam before the start of the first official test for the corresponding entry. Polyteam reserves the right to make public the fact that the report was (or was not) submitted. The contents of the report will not be made public. The report must be generated with the same Polygraph version and workload as used at the cache-off and correspond to the same (to the extent possible) Product as tested at the cache-off.

If such a report is not presented, the entry will receive the lowest handling priority during the cache-off, including troubleshooting and technical support or advice.

The first official performance test will use request rate measured in the submitted report or a lower request rate.

Each Participant must disclose the following:

• The list price of the Product being tested. The given price must be the list price when the Product is initially available or when the cache-off starts, whichever comes later. Accurate estimations are acceptable for Entries not currently on the market.
• The list price of all additional networking equipment that the Participant uses in conjunction with the Entry. This includes hubs, switches, and routers. Network cabling will not be included in these costs. Separate prices should be given for each piece of hardware.
• Exact model numbers and/or specifications of all major hardware systems in use. For example:
  • Digital Alphaserver 1200; (2) 533 MHz CPU's; 1024 MB RAM; (9) 9.1 GB disks (RZ1DF-CB)
  • Alteon ACEswitch 110 plus
  • Cisco 7204 router with (2) 4-port 100BaseTX Ethernet cards.
• Exact version numbers of all major operating systems in use. For example:
  • ProxyOS 4.0D (REV-133)
  • Cisco IOS 11.1(20)CA
  • Alteon ACEswitch Software Version 4.0.42
• Full description of any and all changes made to default configurations and ``out-of-the-box'' states of hardware and software.
• Other details required to reproduce the results acheived at the cache-off.

This documentation must be made available prior to the start of the official test sequence. Polyteam will not start official tests unless the Participant provides all necessary details.

There will be no private rooms for test benches. Participants will be able to see each other equipment.

• A Participant must delegate one or two people to set up equipment and monitor all the tests.
• Polyteam may invite technical support people to help with the setup and such.
• Press is not allowed into the cache-off room(s) during the tests, but may be invited to cover the event in other ways (special conference room, interviews and tours prior to or after the cache-off, etc.). Polyteam coordinates press coverage.

Working hours are Monday through Friday, 9am till 8pm with an hour lunch break. Experiments can be run overnight.

A Bench is logically divided into a Polyteam zone and a Participant zone. Zones are connected using 100BaseT Ethernet cables. Specific requirements for address allocation and network configuration are outside of the scope of this document and are discussed elsewhere.

4.1 Polyteam Zone

Polyteam provides a number of machines to operate as clients and servers. Client machines have similar hardware and software configuration (identical to the extent possible). Server machines have similar hardware and software configuration (identical to the extent possible).

To manage the tests, Polyteam provides monitoring stations, one per Bench. The configuration of monitoring stations does not affect the tests.

The number of client and server pairs in a cluster is determined based one the desired workload (request rate). Some Benches may require more clients and servers than other Benches.

Polyteam is responsible for setup, operation, and maintenance of the client and server machines. The exact hardware and software specs will be included in Polyteam's cache-off report.

The Participant zone includes the Vendor's caching Product, cables, and any networking hardware (switch, router, hub) used to connect the client and server machines to the cache.

All equipment in the Participant zone must meet the ``Product Requirements'' described previously.

All equipment in the Participant zone, including networking hardware, is included in the list price of the Participant's configuration. Substitutions are not allowed. For example, a participant brings a 24-port switch and uses 8 ports, the price cannot be scaled down to an 8 port switch of the same manufacturer. Removing ``blades'' or ``cards'' from network gear is allowed if the gear can be bought without those components.

The Participant is responsible for setup and maintenance of all the equipment in the Participant zone.

The Participant should specify any special accommodations (extra cooling, extra power outlets, etc.) as early as possible.

Participant zone must allocate at least one 10/100BaseT Ethernet port for the monitoring station.

Participant zone setup must provide IP level connectivity between the monitoring station and (clients and servers). For example, the monitoring station must be able to ``ping'' or ``ssh'' to all clients and servers.

Participant may request that their cables are used for networking clients and servers. Otherwise, Polyteam will provide cables.

In the event of performance, compatibility, or other problems between Polyteam and Participant zones, Polyteam and the Participant must work together to isolate and eliminate the problem.

In order to facilitate centralized management of all client and server machines, Polyteam may utilize a routed network that connects all Benches. This network is to be physically dismantled during performance tests.

All benches will be physically separated, and no connections between benches will be allowed (except for temporary network connections described above).

A sequence of tests is run independently and concurrently on every Bench. This section describes the test execution procedure for one Participant.

Official tests are:

1. MSL test
2. PolyMix-3 ``performance'' test
3. Downtime test

All other tests are unofficial.

All tests must start and finish within the time allocated for the cache-off. A total of 55 hours of ``busy'' time is guaranteed for each participant. ``Busy'' time includes the time when official tests are running (or can be running as far as Polyteam is concerned). ``Busy'' time also includes 15 minutes to prepare each test. If the participant shows up late, or is not ready for some reason, time is subtracted from their 55 hours until they are ready for testing. Practice shows that many vendors will have more than 55 hours of test time.

Polyteam will treat requests to repeat a test with a lower priority than the request to execute the first test. Rerunning unexpectedly failed tests will get higher priority that rerunning other tests. Polyteam may refuse to repeat the test when 55 hours run out and/or there are reasons to believe further trials will not change the outcome of the tests.

Test execution rules also include the following provisions.

• Tests are executed manually by Polyteam and/or automatically.

• All equipment should be rebooted before each peformance test (or before the MSL test if that comes first).

• Any two performance tests are separated by at least 10 minutes of inactivity unless the Participant requests to shorten that idle time.

• Participants are allowed to monitor the performance of their Product using any tool that does not interfere with benchmarking activities. Monitoring tools must work in read-only mode.

• MSL test should be executed before the performance test. There should be no reboots or Product changes between the MSL test and the performance test.

• Downtime test should be executed after the performance test. There should be no reboots or Product changes between the performance test and the downtime test.

• MSL and downtime tests must be executed at least once.

• The entire Participant zone setup and configuration must remain intact during an official test.

Reasonable efforts will be made by Polyteam to share run-time performance with Participants.

A participant may repeat any test and choose the best results among all tests executed.

Polyteam archives all the logs from the official tests. On Participant request, Polyteam releases these results to the Participant after the cache-off is over (or earlier, time permitting).

In the event of equipment failure, the failed parts can be replaced, but the interrupted test (if any) is invalidated. The fact of the failure is not reported unless the failure leaves a participant without valid results.

Polyteam prepares an official cache-off report. Polyteam will make reasonable efforts to discuss the content of the report with Vendors before making the report public. Polyteam makes all cache-off results publicly available on the Web. Polyteam must keep the cache-off archive available for at least one year.

Cache-off participants agree not to make public (or publicly discuss) the results of the cache-off prior to public release of the official report.

A Participant may request an inclusion of a ``Participant Comments'' entry into the report. The comments must not contain any references to external documents. The comments should only discuss technical matters relevant to the cache-off. Vendors are strongly discouraged to include speculations and accusations in the comments.

The comments are limited to 300 words and must be submitted in plain ASCII text format.

Once submitted, the comments cannot be modified.

Polyteam reserves the right to not include the Participant Comments.

Polyteam may add their own comments, up to three months after the fpublicationof the report.

The Comments become a part of the official result distribution. These Rules do not require any part of the results to be included in the derived works.

Anyone can publish or otherwise make public any derivate work based on the cache-off results, provided that the ``Terms of use'' clause is satisfied.

Anyone can get a free copy of a Participant result from the Official Site, provided they agree to the ``Terms of use''.

Terms of use

1. Any derived work from, distribution, publication, presentation, etc. of the results must include the following explicit reference in its entirety:

   Alex Rousskov and Duane Wessels. The Third Cache-off. Houston, Texas; September 2000. Raw data and independent analysis at http://www.measurement-factory.com/results/

2. All modifications of the copies of the results are prohibited. Only verbatim copies may be distributed.

After the start of the first official test, a participant must not bailout or prevent their results from being published.

A Participant may request that an entry is considered ``failed''. Polyteam may declare an entry ``failed'' if the entry cannot produce reasonable results despite Polyteam and Participants efforts such as decreasing request rate by 50% of the registered peak load.

Failed entries (if any) will be counted in the report preamble and listed in the executive summary table with measurements replaced with ``n/a''.

A special section at the end of the report will contain details necessary for a reader to understand why no results are available for each failed entries (and, possibly, links to re-test results; see below). Vendors of failed entries may submit ``vendor comments'' sections.

Failed entries will be given an option to be re-tested at Polyteam lab within two weeks after the end of the cache-off. The results of those tests will be linked from the official cache-off report, but will not be included into the cache-off report.

In the event of any form of conflict, including these Rules interpretation, Polyteam will be the final arbiter and judge. A reasonable effort on the part of a Polyteam will be made to find a compromise solution.