Test tool

The Linux Test Project (LTP; see Resources for links and more information) is a joint project with SGI, IBM, OSDL, Bull, and Wipro Technologies with a goal to deliver test suites to the open source community that test the reliability, robustness, and stability of Linux. The Linux Test Project is a collection of tools for testing the Linux kernel and related features. The goal is to help improve the Linux kernel by bringing test automation to the kernel testing effort.

Currently, there are over 2000 test cases within the LTP suite, covering the majority of kernel interfaces such as syscalls, memory, IPC, I/O, filesystems, and networking. The test suite is updated and released monthly and runs on multiple architectures. There are 11 known LTP test suite tested architectures including i386, ia64, PowerPC, PowerPC 64, S/390, S/390x (64bit), MIPS, mipsel, cris, AMD Opteron, and embedded architectures. We used LTP version 20030514 -- the latest available at the time -- in our reliability testing.

Test strategy

There were two unique phases in the baseline run: a 24-hour "initial test," followed by the stress reliability run phase, or "stress test."

Passing the initial test was an entry requirement. The initial test consisted of a successful 24-hour run of the LTP test suite on the hardware and operating system that would be used for reliability runs. The driver script runalltests.sh, which comes with the LTP test suite package, was used to validate the kernel. This script runs a group of packaged tests in sequential order and reports the overall result. It also has the option to launch several instances running in parallel simultaneously. By default, this script executes:

• Filesystem stress tests
• Disk I/O tests
• Memory management stress tests
• IPC stress tests
• Scheduler tests
• Commands functional verification tests
• System call functional verification tests

The stress test verified the robustness of the product during high system usage. In addition to runalltests.sh, a test scenario called ltpstress.sh was specially designed to exercise a wide range of kernel components in parallel with networking and memory management and to create a high-stress workload on the testing system. ltpstress.sh is also part of the LTP test suite. The script runs similar test cases in parallel and different test cases in sequence in order to avoid intermittent failures caused by running into the same resources or interfering with one another. By default, this script executes:

• NFS stress tests
• Memory management stress tests
• Filesystem stress tests
• Math (floating point) tests
• pthread stress tests
• Disk I/O tests
• IPC (pipeio, semaphore) tests
• System call functional verification tests
• Networking stress tests