Setup

Physical Setup

The cluster can be devided into three parts. Support and monitoring infrastructure, device under test (DUT) and traffic generation (tgen). The tgen, nodes 1-4, are PCs with both an e1000 and a r8169 based NIC. The e1000s are being used for traffic generation and the r8169s are unused for this study. The DUTs are a pair of machines with matched pairs of NICs. Node 5 has two r8169 nics, and node 6 has a pair of e1000 based nics. The support and monitoring are a kickstart server/firewall, a pile of switches and a testmaster machine. All the cluster nodes and support infrastructure is also tied together on a management network.

All of the cluster nodes, both DUT and TGEN are based on the Brookdale-G chipset, 2 GHz Celeron CPUs with 128 KBytes cache, 512 Megs of memory, and are using 32 bit PCI buses. All of the NICs used in the cluster are desktop versions of the cards with 32 bit PCI buses.

There are two types of Intel NICs in the cluster, the 8254lGI/PI and the 'Unknown device 107c'. The 107c seems to be a slightly different version of the card sold as the Pro 1000 GT. So to be clear, only the '82541GI/PI' or pro 1000 MT Desktop, and the 'RTL-8169' AirLink gigabit network adapters are on the DUTs, and are thus the only ones being tested.

A more detailed description of each machine can be found in the appendix.
Node # CPU Cache RAM OS Kernel Traffic NICs
1 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.11 i386 
01:00.0 Ethernet controller: Intel Corp.: Unknown device 107c (rev 05)
01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
2 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.11 i386 
01:00.0 Ethernet controller: Intel Corp.: Unknown device 107c (rev 05)
01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
3 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.11 i386 
01:00.0 Ethernet controller: Intel Corp.: Unknown device 107c (rev 05)
01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
4 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.11 i386 
01:00.0 Ethernet controller: Intel Corp.: Unknown device 107c (rev 05)
01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
5 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.12 i386 
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
01:01.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)
6 1 x Intel(R) Celeron(R) CPU 2.00GHz 128 KB 513972 kB Fedora Core release 3 (Heidelberg) Linux 2.6.11.12 i386 
01:00.0 Ethernet controller: Intel Corp. 82541GI/PI Gigabit Ethernet Controller
01:01.0 Ethernet controller: Intel Corp. 82541GI/PI Gigabit Ethernet Controller

Logical Setup

This test uses two differnet network topologies. The first setup we have devided the nodes into traffic generation and DUT. The traffic generaion is further devided into two pairs of two hosts.

Two are connected to the 'top' vlan while the other two are connected to the 'bottom' vlan. To test different DUTs, the DUT is simply configured into the vlan and allowed to pass traffic. In parallel with the DUTs is a patch_loop. A wire is the exact embodyment of what the DUTs are atempting to model so it will serve as our ideal control case.

The second setup takes a pair of traffic generation hosts uses them for round trip time (rtt) monitoring.

Traffic generation

There are three types of traffic generation used. Simple wgets, wgets with vairable MTU and tcpreplay.

The first is a simple pair of wget and apache. The FC3 installed version of apache on each of the machines is configured to serve a set of various sized zero filled web pages. Then wgets are driven against the webservers. By selecting different sized files a different average file size can be obtained, and thus the average packet size can be modulated. Currently this traffic stream is a function of number of wgets in parallel, avg filesi The second configuration can do at least 1/2 as much. ze, and content type. For this test we are using zero filled files.

Each TGEN host is configured with multiple IPaddresses on there service networks. Then each server randomly picks urls and random IPs on the oposite side of the link. In effect simulating a mesh of connections from all the TOP hosts and IPs to all the bottom hosts and IPs.

The wget with variable MTU is just the above wget/apache pairing with the MTU of the service interface changing. Modifying the MTU can help produce more fragmented traffic, and can help differentiat between the cost of packets and the cost of connections.

Tcpreplay is used to generate UDP ethernet frames of arbitrary size. The hosts are seperated into pairs of top and bottom hosts. Each member of the pair sends out a uniq MAC address to help inform the infrastructure switch and the DUTs MAC tables. Then the host on the oposite side sends traffic to that MAC. This traffic stream is a function of both the number of tcpreplays to start, the size of frame to send, and the number of frames to send in each loop of tcpreplay.

In the first configuration the cluster can generate up to 900+ Mbits / sec or 300 Kpkts / sec across a DUT. The second configuration can do at least 1/2 as much.

DUT configuration

The DUTs were configured first as standard two port ethernet briges. Then iptables was instructed to direct all FORWARD traffic that came in br0 to the ip_queue kernel module. Snort_inline, then picked up packets and returned them to the kenrel as nessisary.

Snort_inline 2.3.0 Build 10 (snort_inline-2.3.0-RC1) was used for the durration of testing. As an ideal version of a userland packet filter the passthrough example from the libipq package was used.

Data Aquisition and Definitions

Metrics and statistics can be a ficle beast. In most cases any measurements of bandwith or pktrate are taken from the snmp counters on the infrastructure switch. Aditional measurements are taken from the DUTs and the Tgen hosts.

Word Defintion
outofdut A byte,bit or packet that has been emited from the DUT.
intodut A byte,bit or packet that was sent to the DUT.
total outofdut + intodut
snmp Simple Network Management Protocol, values derived from snmp quantites will tipicaly be labled as snmp
eth as in networking device eth2, values derived from /proc/net/dev will be tipicaly labled with 'eth'.