Benchmark Results

Presented are descriptions of three selected benchmarks used in our suite to quantify Linux kernel performance: database query, VolanoMark, and SPECweb99. For all three benchmarks, we used 8-way machines, as detailed in the figures presenting the benchmark results.

Figure 1. Database query benchmark results

Figure 1 shows the database query benchmark results. Also included is a description of the hardware and software configurations used. The figure graphically illustrates the progress we have made in achieving our target. Some of the issues we have addressed have resulted in improvements that include adding bounce buffer avoidance, ips, io_request_lock, readv, kiobuf and O(1) scheduler kernel patches, as well as several DB2 optimizations.

The VolanoMark benchmark (see Resources) creates 10 chat rooms of 20 clients. Each room echoes the messages from one client to the other 19 clients in the room. This benchmark, not yet an open source benchmark, consists of the VolanoChat server and a second program that simulates the clients in the chat room. It is used to measure the raw server performance and network scalability performance. VolanoMark can be run in two modes: loopback and network. The loopback mode tests the raw server performance, and the network mode tests the network scalability performance. VolanoMark uses two parameters to control the size and number of chat rooms.

The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the server to take turns broadcasting all of the clients' messages to the group. At the end of the loopback test, it reports a score as the average number of messages transferred per second. In the network mode, the metric is the number of connections between the clients and the server. The Linux kernel components stressed with this benchmark include the scheduler, signals, and TCP/IP.

Figure 2. VolanoMark benchmark results; loopback mode

Presented in Figure 2 are the VolanoMark benchmark results for loopback mode. Also included is a description of the hardware and software configurations used and our target for this benchmark. We have established close collaboration with the members of the Linux kernel development team on moving forward to achieve this target. Some of the issues we have addressed that have resulted in improvements include adding O(1) scheduler, SMP scalable timer, tunable priority preemption and soft affinity kernel patches. As illustrated, we have exceeded our target for this benchmark; however, there are some outstanding Linux kernel component-related and Java-related issues we are addressing that we believe will further improve the performance of this benchmark.

Please note that the SPECweb99 benchmark work was conducted for research purposes only and was non-compliant, with the following deviations from the rules:

1. It was run on hardware that does not meet the SPEC availability-to-the public criteria. The machine was an engineering sample.
2. access_log wasn't kept for full accounting. It was written, but deleted every 200 seconds.

This benchmark presents a demanding workload to a Web server. This workload requests 70% static pages and 30% simple dynamic pages. Sizes of the Web pages range from 102 to 921,000 bytes. The dynamic content models GIF advertisement rotation. There is no SSL content. SPECweb99 is relevant because Web serving, especially with Apache, is one of the most common uses of Linux servers. Apache is rich in functionality and is not designed for high performance. However, we chose Apache as the Web server for this benchmark because it currently hosts more Web sites than any other Web server on the Internet. SPECweb99 is the accepted standard benchmark for Web serving. SPECweb99 stresses the following kernel components: scheduler, TCP/IP, various threading models, sendfile, zero copy and network drivers.

Figure 3. SPECweb99 benchmark results using the Apache Web server

Presented in Figure 3 are our results for SPECweb99. Also included is a description of the hardware and software configurations used and our benchmark target. We have a close collaboration with the Linux kernel development team and the IBM Apache team as we make progress on the performance of this benchmark. Some of the issues we have addressed that have resulted in the improvements shown include adding O(1) and read copy update (RCU) dcache kernel patches and adding a new dynamic API mod_specweb module to Apache. As shown in Figure 3, we have exceeded our target on this benchmark; however, there are several outstanding Linux kernel component-related issues we are addressing that we believe will significantly improve the performance of this benchmark.