Performance Evaluation

Forwarding performance measurements

We continuously test the forwarding performance of the qualified hardware platforms. To determine the forwarding performance we run a set of tests for each hardware. The different test cases are described in separate sections below.

Depending on the processing that is tested a different topology is used:

  • 2-Node Topology: Consists of the Traffic Generator (TG) and the Device under Test (DUT). Used for evaluating forwarding processing of the DUT, such as IP or SCION packet routing.

    2 Node Performance Setup

  • 3-Node Topology: Consists of the Traffic Generator (TG), the Device under Test (DUT), and a secondary Device under Test (DUT2). Used for evaluating the tunneling processing of the DUT, i.e., encapsulation and decapsulation, such as the IP in SCION packet tunneling. The DUT2 serves as the opposite side of the tunnel and is sufficiently size to not be a bottleneck of the test.

    3 Node Performance Setup

Each test uses multiple loss ratio search (MLRsearch) to determine the non-drop-rate (NDR) of the devices under test. MLRsearch is described in an IETF RFC draft, in short it’s an optimized method to determine the packet throughput rate of a device under test. The NDR is the maximum packet throughput rate at which no packets are lost. To determine the NDR we use the T-Rex traffic generator.

The performance evaluation is done automatically nightly. The results are stored and compared to previous results. If there was a regression in the performance, the team is notified and countermeasures are taken. This way performance regressions are prevented in new releases.

IP routing

To establish a baseline for a device, we evaluate the NDR of the IPv4 forwarding performance in the 2-node setup. This ensures that the network interface card (NIC) works as expected with the VPP software stack that we use. In this test case we expect to reach line rate of the NIC.

IP in SCION tunneling

This test case is used to determine the NDR of the DUT when IP packets are encapsulated and decapsulated into SCION packets using the IP-in-SCION tunneling component, with and without encryption. This test is performed in the 3-node setup.

SCION routing

This test case is used to determine the NDR for SCION routing performance of the DUT in the 2-node setup.

Flow reporting performance measurements

A GATE exports flow data to a cloud hosted flow exporter. The collected flow data is used to bill customers of the GATE infrastructure, thus its correctness is crucial.

We need to assure that the flow export pipeline can keep up with the traffic. Therefore, we are running tests that generate a high number of flows and assert that all the flows are correctly exported.

The basic setup for these test is the same as with SCION tunneling forwarding performance test. However, the flow exporter is enabled on DUT and a flow collector is run to collect the reports.

After the test is run, we check that the reported number of flows and number of packets match the expected values.

In real-world-like setup, the gateway reports the recorded flows every 30 minutes to the flow collector. For the test, the gateway reports every second to the flow collector, actually doing more work than in production.

Additionally, the flow cleanup period in the gateway is set such that the clean up never happens during the test. Once the test is finished, the cleanup is triggered and the time it takes to clean up the flows is measured and reported.

Currently, we test with the rate of 100000 packets/second and with different numbers of flows, up to 100000.

Test regression is reported when any flows or packets are missing in the report from the flow collector.