Polyteam

IRCache/UCSD team responsible for Web Polygraph development and bake-off organization.

Vendor

An organization bringing a caching product to the bake-off. To simplify the terminology all commercial, non-profit, virtual, etc. organizations are labeled as ``vendors.''

Box

A caching product/solution/box being tested as a single ``proxy'' unit. A Box includes software, hardware, network gear, etc. required for proxy operation.

Participant

A (Vendor, Box) tuple.

Cluster

A group of machines and network gear dedicated to benchmarking a Box, including the Box itself.

Any company or organization (Vendor) with a caching Web proxy product (Box) can register for participation.

There is a minimum participation fee of $5000 (US dollars) for each Participant. In some cases, the participation fee may be higher if the Participant requires additional hardware to drive their Box.

A finite number N (e.g. 10) is placed on the number of Participants. If space allows, some Vendors may be able to test multiple Boxes. The Vendor must pay the participation fee for each Box.

Vendors must make an official request to participate by February 26th. Early requests will gladly be accepted. The first N Vendors that request participation before February 26th will be allowed to test at least one Box.

Vendors that request to bring more than one Box will be notified exactly how many Boxes they may bring by March 1st. If space allows, Polyteam will allocate unused Clusters to Vendors in a fair manner. The i-th Box from vendor A has higher priority than the j-th Box from vendor B, for any A,B,j,i where i < j.

Polyteam reserves the right to deny any request for participation.

2.2 Requirements on Boxes

A Box (as defined in the ``Terminology'' section) must satisfy all of the following criteria to be considered for the bake-off.

• The Box must be available for purchase no later than six months after the Bake-off date.
• The Box proxy must have an interface that can reliably and verifiably flush (empty) the cache.
• The Box cache must be fillable by Polygraph software in the specified Cluster environment in at most eight hours. If the cache is not full after eight hours of filling-the-cache run, the Box is disqualified. Please see the ``Test Suite'' document for details of the filling-the-cache run.
• The Box's TCP stack must use a Maximum Segment Lifetime (MSL) value of 30 seconds, which corresponds to a ``TIME_WAIT'' timeout (2*MSL) value of 60 seconds.

Each Participant must disclose the following:

• The list price of the Box being tested. The given price must be the list price when the Box is initially available. If already available, the given list price must be the Box's price when the Bake-off is held (March 15). If the Box will not be available for the time allowed above (e.g. six months), then the given list price must be the price at which the Box will first be sold to customers.
• The list price of all additional networking equipment that the Participant uses in conjunction with the Box. This includes hubs, switches, and routers. Network cabling will not be included in these costs. Separate prices should be given for each piece of hardware.
• Exact model numbers and/or specifications of all major hardware systems in use. For example:
  • Digital Alphaserver 1200; (2) 533 MHz CPU's; 1024 MB RAM; (9) 9.1 GB disks (RZ1DF-CB)
  • Alteon ACEswitch 110 plus
  • Cisco 7204 router with (2) 4-port 100BaseTX Ethernet cards.
  • ...?
• A diagram that clearly shows the networking topology of all hardware.
• Exact version numbers of all major operating systems in use. For example:
  • Digital Unix 4.0D (REV-133)
  • Cisco IOS 11.1(20)CA
  • Alteon ACEswitch Software Version 4.0.42
• Full description of any and all changes made to default configurations and ``out-of-the-box'' states of hardware and software.

Deadlines occurring prior to the bake-off are relative to the first day of the bake-off (negative values). Deadlines occurring after the event, are relative to the last day of the bake-off (positive values).

Relative Time (days)	Absolute Date	Description
-35	Feb 8	Draft Rules, etc documents due to potential Participants.
-28	Feb 15	FINAL Rules, etc documents due to potential Participants.
-21	Feb 22	Polygraph code freeze
-17	Feb 26	Box specifications and cost reports are due. Registration cut-off. The list of participating Vendors is announced.
-14	March 1	Vendors are notified how many Boxes they may bring to the Bake-off, and what their total participation fee is.
-8	March 8	Bake-off payments are due
-0	March 15	Vendor Boxes and benchmarking gear must be installed at the bake-off site. The bake-off begins
+0	March 17	Last day of the bake-off.
+1	March 18	Results are released to the participants
+8	March 25	Participants Comments and Disclosure/Bailout notifications are due
+14	March 31	Preliminary Polyteam report is due

3.1 Working hours

Working hours are 8am till 8pm. Reasonable exceptions are possible (e.g., if absolutely necessary, caches may be filled at night).

4. Access to the bake-off facility

• A Participant must delegate one or two people to set up equipment and monitor all the tests.
• Polyteam may invite technical support people to help with clusters' setup and such.
• Press is not allowed into the bake-off room during the tests, but may be invited to cover the event in other ways (special conference room, interviews and tours prior to or after the bake-off, etc.). Polyteam coordinates press coverage.

A Cluster is logically divided into a Polyteam zone and a Participant zone. Zones are connected using necessary networking gear.

5.1 Polyteam Zone

Polyteam provides a number of machines to operate as client-server pairs. A Cluster has the same number of client machines and server machines. Polyteam guarantees that every client-server pair (across all Clusters) will be identical.

The number of client-server pairs in a cluster is determined by the corresponding Participant. Some Clusters may have more client-server pairs than other Clusters. A Participant may have to pay an extra fee to cover the costs of additional machines required for Participant's Cluster configuration. The default number of polymachines is given in the ``Logistics'' document.

Polyteam is responsible for setup, operation, and maintenance of the client-server machines. The exact model numbers, version numbers, and configuration of the client-server machines will be included in Polyteam's bake-off report.

The actual Polyteam zone equipment to be used at the bake-off is outside of the scope of these Rules and is described in the ``Logistics'' document.

5.2 Participant Zone

The Participant zone includes the Vendor's caching Box, and any networking hardware (switch, router, hub) used to connect the client-server machines to the cache.

All equipment in the Participant zone must meet the ``Requirements on Boxes'' described previously.

All equipment in the Participant zone, including networking hardware, is included in the list price of the Participant's configuration.

The Participant is responsible for setup and maintenance of all the equipment in the Participant zone.

The Participant should specify any special accommodations (extra cooling, extra power outlets, etc.) as early as possible.

A Participant is responsible for network gear setup and maintenance.

In the event of performance, compatibility, or other problems between Polyteam and Participant zones, Polyteam and the Participant must work together quickly to isolate and eliminate the problem.

5.3 Inter-Cluster Network

In order to facilitate centralized management of all client-server machines, Polyteam may utilize a routed network that connects all Clusters. This network is only to be used for Pre-Testing and Post-Testing activities. It is to be physically dismantled during the Testing Phase.

Polyteam requires one network port (10/100 Ethernet) on each Cluster to create this network. The networking hardware (hub, switch router) provided by the Vendor must have a spare port for this purpose.

6. Testing Preparation

Monday, March 15th:

Polyteam configures all equipment that it is responsible for, especially the client and server PC's.

Participants arrive with their Boxes. Each Box is assigned to a Cluster.

Polyteam verifies that each Cluster's network is functioning properly in terms of Layer 2 (Ethernet) and Layer 3 (TCP/IP) connectivity and performance. Polyteam will use ttcp or similar to measure raw TCP throughput. These raw TCP throughput measurements may be used in Polyteam's bake-off report.

Each Participant will be made aware of the TCP throughput test results, and provided the opportunity to investigate potential problems.

Polyteam executes the No-Proxy runs, as described in the ``Test Suite'' document. Participants are made aware of the No-Proxy test results and given a final opportunity to investigate potential problems.

Boxes are flushed and Polygraph is used to fill each Box from an empty state. The filling process is not a part of the test sequence. Any data that is collected during this phase remains the property of the Vendor and will not be included in Polyteam's bake-off report.

7. Testing Phase

Tuesday, March 16th and Wednesday, March 17th:

The actual set of experiments to be used at the bake-off is outside of the scope of these Rules and is documented in the ``Test Suite'' document.

Testing phase starts with the start of the first test (across all Clusters). A sequence of tests is run independently and concurrently on every Cluster. Testing phase ends with the end of the last test (across all Clusters).

There is no synchronization between test executions for any two Participants. This section describes the test execution procedure for one Participant.

• The test sequence is executed manually by Polyteam and/or automatically.
• A test starts with a 10 minute ``cooling'' period when no requests are sent to the Box.
• Test terminating conditions for each test are documented in the ``Test Suite'' document.
• Participants are prohibited to change proxy configuration or tuning parameters during the test phase (from the start of the first test till the end of the last test).
• Participants are allowed to monitor the performance of their Box using any tool that does not interfere with benchmarking activities. Monitoring tools should work in read-only mode.
• Participants are not guaranteed any access to the benchmarking results during the tests. Reasonable efforts will be made by Polyteam to share run-time performance with Participants.
• Boxes may not be turned off or rebooted during the Test Phase unless absolutely necessary.

After completing the entire set of the experiments (successfully or not), a Participant or Polyteam may wish to re-run some of the tests.

Results of all runs will be released to the corresponding Participant.

The Participant may choose which of the repeated runs to include in the results set.

Polyteam will give a lower priority to reruns than to first-run experiments.

Participant initiated reruns

A Participant may request a rerun of any number of tests. A given test will be rerun if

1. the entire set of the experiments has been completed, and
2. there is a reasonable doubt that rerun results will differ from previous results, and
3. a rerun will finish within the bake-off time frame

Polyteam may rerun any number of tests provided there is a reason to believe that new results will differ from original ones.

7.2 Post-testing Activities

Polyteam archives all the results from the testing phase and releases them to Participants.

Participants have the choice to receive the results in one of the following ways:

• On floppy disks
• Copied (FTP, scp) to one of their own systems
• Other? Polyteam may be able to accommodate special requests

This subsection covers events that affect several Vendors rather than a single Participant or a single Vendor.

Vendor boxes and benchmarking equipment may fail for reasons beyond the Vendor or Polyteam control. A reasonable effort will be made to re-run the failed tests. However, we may run out of time doing so. If the participants and Polyteam cannot agree on disclosing partial results, the whole bake-off will have to be canceled. A participant and Polyteam has the right to veto partial disclosure.

8. Bake-off Results

1. All output produced by the benchmarking software and
2. Cost Report and
3. Box availability date and
4. Participant Comments if any and
5. Polyteam Comments if any

1. Bailout Notice

Polyteam must provide documentation or tools sufficient to convert all logged output to a human readable form.

8.1 Results Disclosure

After the bake-off results are released to a Participant:

• A Participant has to choose one of the only two options:
  1. allow the results to be disclosed
  2. prohibit disclosure of the results
• If the results are to be disclosed, ``Results Publication'' rules apply.
• If the results are not to be made public, ``Bailout'' clause applies.

If a Participant prohibits disclosure of the results (see ``Results Disclosure'' section), then

1. No information about the results will be released without explicit Participant's consent.
2. The fact that a Participant has prohibited disclosure will be made public. See below.
3. Polyteam may keep all the results in their private archive unless the Participant explicitly requests that the results must be deleted. Polyteam must destroy its copy of the results if such a request is made.

If Participant (V, B) prohibits disclosure of the results, the following notice must be added to the official results distribution:

All the data related to the tests of B has been removed from the official bake-off results based on V's request. The Rules of the Bake-off prohibit the inclusion of any other comments regarding V's decision.

This section applies if and only if the results were disclosed (see the "Results Disclosure" section).

• A Participant may request an inclusion of a ``Participant Comments'' entry into the results. These Rules impose no limitations on the content or nature of those comments. The comments must be of reasonable size and in a reasonable electronic format.
• After the results are released, a Participant has one week to submit their comments for inclusion. After this waiting period, no addition or modifications are allowed.
• Polyteam must include the Participant Comments in the results if requested by the Participant.
• Polyteam may add their own comments, up to three months after the final day of the Bake-off.

NOTE: The Participant Comments become a part of the official result distribution. These Rules do not require any part of the results to be included in the derived works.

8.4 Results Publication and Dissemination

This section applies if and only if the results were disclosed (see the "Results Disclosure" section).

8.5 Official Distribution Site

• Polyteam makes all disclosed results publicly available on the Web. Polyteam's archive becomes an ``official site'' for distribution of the results.
• There will be only one Official Site.
• The location of the Official Site is to be determined by Polyteam. The location might change with time.
• Polyteam must keep the archive available for at least three years.

• Polyteam prepares a report based on the bake-off results.
• The report is presented during the Fourth Caching Workshop and is published in the Workshop Proceedings.
• Polyteam will make reasonable efforts to discuss the content of the overview with Vendors prior to the presentation/publication.

Anyone can publish or otherwise make public any derivate work based on the bake-off results, provided that the ``Terms of use'' clause is satisfied.

Anyone can get a free copy of a Participant result from the Official Site, provided they agree to the ``Terms of use''.

8.8 Terms of use

1. Any derived work from, distribution, publication, presentation, etc. of the results must include the following explicit reference in its entirety:

   Wessels, Rousskov, et. al. The First Caching Bake-off. In Proceedings of the Fourth Caching Workshop, San Diego, California, USA, April 1999. Raw data and independent analysis at http://bakeoff.ircache.net/

2. All modifications of the copies of the results copies are prohibited. Only verbatim copies may be distributed.

In the event of any form of conflict, including these Rules interpretation, Polyteam will be the final arbiter and judge. A reasonable effort on the part of a Polyteam will be made to find a compromise solution.