The Network Simulator ns-2: Validation Tests

Daily status of the validation suite

The test suite (described below) is run every day on the ns snapshot. Output of the daily validation is sent to the nsnam-commits mailing list (browse the list archive to see whether other platforms are encountering problems).

Classes of Ns Protocols

The program ``validate'' in the root directory of the ns distribution runs all current standard tests. Protocols covered in validate represent the most stable core of ns. We insure that validate passes on several different systems for each ns release, and we run it over the daily snapshot (see below).

We encourage you to report problems with validated protocols to us. We try to resolve these problems rapidly (as resources allow).

Even though we consider these protocols ``validated'', our test suite coverage is not complete. You are advised to look at what aspects of the protocols are tested in the test suite before drawing research conclusions from these protocols.

Protocols and modules covered at least in part by validate include the following:
Application-level:

Transport protocols (UDP, TCP, RTP, SRM): Routing: Router Mechanisms (scheduling, queue management, admissions control, etc.): Link-layer mechanisms: Other:

In addition there are a number of protocols in the standard ns distribution which are not covered by validate. Because they cannot be automatically tested, bit-rot sometimes breaks these protocols.

We attempt to keep non-validated protocols working and welcome bug reports. Becuase of difficulties maintaining code that we did not write and for which we may not know ``ground truth'', we cannot promise that these protocols will remain working. We strongly encourage people using these protocols in their research to examine their output carefully and implement test suites for them so that we can move them into the ``validated'' category.

Protocols and modules in the core but not validated include:

Finally, there are a number of contributed protocols described on their own web page. These protocols are often for specific releases of ns and may not work in the current release.

We welcome discussion of contributed protocols on the ns-users mailing list, but we are unlikely to be able to provide support for them. We welcome outside efforts to move these protocols into the core; we will add protocols that include test suites and work with the current release at their authors or users request. In some cases we are actively integrating contributed code on a case-by-case basis (for example, we're currently adding wireless extensions).

Specific Ns Validation Tests

The basic test

The Simulator Tests document illustrates a basic suite of validation tests that mainly tests TCP congestion control algorithms. These tests are run with the command ./test-all-simple in ns-2/tcl/test (or ./test-all quiet to suppress graphs). Comments.

Extended versions of this test suite are in tcl/test/test-suite-routed.tcl. This version also separates out the topology as a single library (tcl/test/topology.html), and common drivers for the different test suites (tcl/test/misc.tcl).

The test-all-simple suite, but using the tcl code for that test suite from ns-1, can be run in ns-2 in ``backward compatibility mode'' with the command test-all-v1 in ns-2/tcl/test (or ./test-all-v1 quiet to suppress graphs).

The SSF TCP validation tests are adapted from the TCP validation tests in ns-2 to validate the TCP implementation in the SSFNET simulator. The SSFNET web page also includes a detailed description of the behavior in each test.

More TCP tests

The SACK TCP validation tests -1 can be run with the command test-all-sack in ns-2/tcl/test (or ./test-all-sack quiet to suppress graphs). The SACK TCP validation tests from ns-1 can be run with the command test-all-sack-v1 in ns-2/tcl/test. Comments. The TCP implementations in these tests use one-way TCP without SYN/FIN packets.

Tests comparing Tahoe, Reno, SACK, and Vegas TCP can be run with the command test-all-tcpVariants in ns-2/tcl/test. Some of these tests are described in Simulation-based Comparisons of Tahoe, Reno, and SACK TCP . Comments. The TCP implementations in these tests use both one-way and two-way TCP.

Tests with Reno FullTCP (with SYN/FIN packets and two-way data flow) can be run with the command test-all-full in ns-2/tcl/test. These tests are described in Ns Simulator Tests for Reno FullTCP.

Additional tests of TCP's retransmit timer and ECN functionality can be run with the command ./test-all-tcp in ns-2/tcl/test (or ./test-all-tcp quiet to suppress graphs). This test is discussed in the file test-suite-tcp.txt.

Tests for TCP-Vegas (ported from ns-v1) can be run tcl/test/test-all-vegas-v1.

Tests for TCP with rate-based-pacing are in tcl/test/test-all-rbp (see Improving Restart of Idle TCP Connections for a description of RBP). This script also tests Reno and Vegas behavior after an idle period, with and without the slow-start restart option.

Queue management

Ns Simulator Tests for Random Early Detection (RED) Gateways illustrates the validation tests for the implementation of RED queue management. These tests can be run with ./test-all-red in ns-2/tcl/test (or ./test-all-red quiet to suppress graphs). The ns-1 tests are run with the command ./test-all-red-v1 in ns-2/tcl/test. Comments.

The validation tests for ECN are described in the note ECN Implementations in the NS Simulator (postscript, PDF).

Scheduling

Ns Simulator Tests for Class-Based Queueing illustrates the validation tests for the implementation of CBQ in the ns simulator. These tests are run with the command ./test-all-cbq in ns-2/tcl/test (or ./test-all-cbq quiet to suppress graphs). The command test-all-cbq-v1 in ns-2/tcl/test runs the ns-1 cbq test suite in backward compatibility mode. All of the tests run in ns-2 except for the text showing the ``old'' implementation of formal link-sharing, which is not implemlented in ns-2.

The Deficit Round Robin (DRR), Fair Queueing (FQ), and Stochastic Fair Queueing (SFQ) scheduling algorithms are contrasted with FIFO scheduling in the scheduling validation tests, run with the command test-all-schedule in ns-2/tcl/test (or ./test-all-schedule quiet to suppress graphs).

Multicast routing

The validation tests for multicast routing can be run with the command test-all-mcast in ns-2/tcl/test. Currently the test suites consist of 2 tests for Centralized Multicast, 2 tests for a basic dense mode protocol, DM (does not respond to topology changes) and 7 tests for a more complete version of the dense mode protocol, detailedDM.

A test suite for exhaustive selective loss on lans can be run with the detailedDM multicast in two simulation runs. The first run uses the script ns-2/ex/newmcast/pim-no-loss.tcl, which runs the simulations without loss to determine which packets/links will lose the packets in later runs. A second simulation is then run using the script ns-2/ex/newmcast/pim-loss.tcl. This example script targets Join messages for selective loss.

Demos

(Many of these demos use nam.)

Unicast routing

Multicast routing

Multicast transport

Traffic

Other

ns [email protected]