2.5. Practice with TRex

2.5.1. Download TREX

TREX should be installed in Tester, it could be downloaded from http://trex-tgn.cisco.com/trex/release/. We recommend to use the latest version v2.88.

wget http://trex-tgn.cisco.com/trex/release/v2.88.tar.gz

To read more about T-Rex stateless mode, read the T-Rex stateless support guide.

2.5.2. Configure TREX

2.5.2.1. Extract the downloaded TREX tarball

Extract the downloaded TREX file to /opt/trex/:

root@tester:/opt# mkdir trex
root@tester:/opt# cd trex
root@tester:/opt/trex# tar -zxvf v2.88.tar.gz
v2.88/
v2.88/_t-rex-64-debug
v2.88/t-rex-64-debug
...
v2.88/trex_client_v2.88.tar.gz

2.5.2.2. Generate Configure File

2.5.2.2.1. Identify the performance test ports in tester

execute the command like below to list the ports:

root@tester:/opt/trex/v2.88# ./dpdk_setup_ports.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if=ens160 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:0b:00.0 'Device 1592' if=ens192f0 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic
0000:0b:00.1 'Device 1592' if=ens192f1 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic
Other network devices
=====================
<none>

2.5.2.2.2. Generate Configure File

Assume that we use 0000:0b:00.0, 0000:0b:00.1 to connect to DUT port. And the ports topology as below:

trex.0 (0b:00.0/40:a6:b7:0b:55:78) <-------> DUT port 0 (05:00.0/b4:96:91:9f:63:68)
trex.1 (0b:00.1/40:a6:b7:0b:55:79) <-------> DUT port 1 (05:00.1/b4:96:91:9f:63:69)

We need to generate the trex config file as below:

root@tester:/opt/v2.88# ./dpdk_setup_ports.py -i
By default, IP based configuration file will be created. Do you want to use MAC based config? (y/N)y
+----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
| ID | NUMA |   PCI   |        MAC        |                    Name                    | Driver  | Linux IF |  Active  |
+====+======+=========+===================+============================================+=========+==========+==========+
| 0  | -1   | 03:00.0 | 00:0c:29:29:30:80 | VMXNET3 Ethernet Controller                | vmxnet3 | ens160   | *Active* |
+----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
| 1  | -1   | 0b:00.0 | 40:a6:b7:0b:55:78 | Device 1592                                | ice     | ens192f0 |          |
+----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
| 2  | -1   | 0b:00.1 | 40:a6:b7:0b:55:79 | Device 1592                                | ice     | ens192f1 |          |
+----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+
Please choose an even number of interfaces from the list above, either by ID, PCI or Linux IF
Stateful will use order of interfaces: Client1 Server1 Client2 Server2 etc. for flows.
Stateless can be in any order.
Enter list of interfaces separated by space (for example: 1 3) : 1 2
For interface 1, assuming loopback to its dual interface 2.
Destination MAC is 40:a6:b7:0b:55:79. Change it to MAC of DUT? (y/N).y
Please enter a new destination MAC of interface 1: b4:96:91:9f:63:68
For interface 2, assuming loopback to its dual interface 1.
Destination MAC is 40:a6:b7:0b:55:78. Change it to MAC of DUT? (y/N).y
Please enter a new destination MAC of interface 2: b4:96:91:9f:63:69
Print preview of generated config? (Y/n)Y
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['0b:00.0', '0b:00.1']
  port_info:
      - dest_mac: b4:96:91:9f:63:68
        src_mac:  40:a6:b7:0b:55:78
      - dest_mac: b4:96:91:9f:63:69
        src_mac:  40:a6:b7:0b:55:79

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3,4,5,6,7]

Save the config to file? (Y/n)Y
Default filename is /etc/trex_cfg.yaml
Press ENTER to confirm or enter new file:
File /etc/trex_cfg.yaml already exist, overwrite? (y/N)Y
Saved to /etc/trex_cfg.yaml.
root@tester:/opt/trex/v2.88#

We could not modify the DUT mac during the configuration and modify it in the generated file /etc/trex_cfg.yaml.

2.5.2.2.3. Modify Configure File

Make sure the DUT mac of the generated TREX file is correct, and add prefix and limit_memory is better:

root@tester:/opt/trex/v2.88# cat /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['0b:00.0', '0b:00.1']
  prefix: TREX
  limit_memory: 4096
  port_info:
      - dest_mac: b4:96:91:9f:63:68
        src_mac:  40:a6:b7:0b:55:78
      - dest_mac: b4:96:91:9f:63:69
        src_mac:  40:a6:b7:0b:55:79

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2,3,4,5,6,7]

root@tester:/opt/trex/v2.88#

2.5.2.3. Replace system scapy with TREX scapy

As the know trex issue – trex scapy lib will be conflict with scapy installed in OS, we need the replace the scapy in system with trex scapy.

  1. backup your scapy:

    cp -rf /usr/local/lib/python3.6/dist-packages/scapy /usr/local/lib/python3.6/dist-packages/scapy_backup
    
  2. unify scapy version with trex:

    cp  -rf /opt/trex/v2.88/external_libs/scapy-2.4.3/scapy/ /usr/local/lib/python3.6/dist-packages/scapy
    

2.5.3. Configure DTS

2.5.3.1. Configure execution.cfg

Add the performance test suite and set perf=true:

root@tester:dts# cat execution.cfg
[Execution1]
crbs=192.168.1.1
drivername=vfio-pci
build_type=meson
test_suites=
        tso,
targets=
    x86_64-native-linuxapp-gcc
parameters=nic_type=cfg:perf=true
root@tester:dts#

2.5.3.2. Configure conf/crbs.cfg

Set the pktgen_group=trex or pktgen_group=TREX, this item is case insensitive:

root@tester:dts# cat conf/crbs.cfg
[192.168.1.1]
dut_ip=192.168.1.1
dut_user=root
dut_passwd=dutpwd
os=linux
dut_arch=
tester_ip=192.168.1.2
tester_passwd=testerpwd
ixia_group=
pktgen_group=trex
channels=4
bypass_core0=True

2.5.3.3. Configure conf/ports.cfg

This configuration is just same with PF function test, so if you have completed some functional test, you have no need to modify the conf/ports.cfg, just like below:

root@tester:dts# cat conf/ports.cfg
[192.168.1.1]
ports =
    pci=0000:05:00.0,peer=0000:0b:00.0;
    pci=0000:05:00.1,peer=0000:0b:00.1;

In addition, it could be configured as below:

root@tester:dts# cat conf/ports.cfg
[192.168.1.1]
ports =
    pci=0000:05:00.0,peer=TREX:0;
    pci=0000:05:00.1,peer=TREX:1;

We recommend to use the first format configuration, as it has no need to do modification when we do functional test.

2.5.3.4. Configure conf/pktgen.cfg

Fill in the conf/pktgen.cfg with your TREX setting, for the first time, you should set start_trex=yes. Set the IP address of the device which you installed TREX into item ‘server=’, it should be the tester IP. The configuration should as below:

root@tester:dts# cat conf/pktgen.cfg
[TREX]
trex_root_path=/opt/trex/v2.88
trex_lib_path=/opt/trex/v2.88/automation/trex_control_plane/interactive
config_file=/etc/trex_cfg.yaml
server=192.168.1.1
pcap_file=/opt/trex/v2.88/stl/sample.pcap
core_num=4
#core_mask=0x3
ip_src=16.0.0.1
ip_dst=10.0.0.1
warmup=15
duration=-1
start_trex=yes

As the trex_lib_path may be different in different versions, you could find the correct path as blow command:

root@tester:/opt/trex/v2.88# find . -name trex_stl_lib
./automation/trex_control_plane/interactive/trex_stl_lib
root@tester:/opt/trex/v2.88#

2.5.3.5. Run DTS performance test with TREX

Now you can start DTS performance test with TREX:

root@tester:/home/zhaohy/dts# ./dts
                       dts:
DUT 192.168.1.1
                    tester: ssh root@192.168.1.1
...
pktgen: ssh root@192.168.1.1
pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg.yaml -c 4
                    pktgen: Starting Scapy server..... Scapy server is started
Trying to bind to igb_uio ...
/usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:0b:00.0 0000:0b:00.1
...
                 TestTSO: Test Case test_perf_TSO_2ports Begin
         dut.10.240.183.72:
                    tester:
                   TestTSO: Executing PMD using 1S/1C/2T
         dut.10.240.183.72: x86_64-native-linuxapp-gcc/app/dpdk-testpmd  -l 1,45 -n 4 -a 0000:05:00.0 -a 0000:05:00.1  --file-prefix=dpdk_31529_20210324143008   -- -i --rxd=512 --txd=512 --burst=32 --rxfreet=64 --mbcache=128 --portmask=0x3 --max-pkt-len=9000 --txpt=36 --txht=0 --txwt=0 --txfreet=32 --txrst=32
         dut.10.240.183.72: EAL: Detected 88 lcore(s)
         ...
pktgen: Rx Port 0 stats:
rx_port: 0,  rx_bps: 25354096640.000000, rx_pps: 1239130.250000
                    pktgen: throughput: pps_rx 5463897.750000, bps_rx 50961129472.000000
                    pktgen: traffic completed.
...
                   TestTSO:
+------------+---------------+------------+
| Frame Size | 1S/1C/2T Mpps | % linerate |
+============+===============+============+
| 128        | 5.371         | 4          |
+------------+---------------+------------+
| 2500       | 5.464         | 56         |
+------------+---------------+------------+
                       TestTSO: Test Case test_perf_TSO_2ports Result PASSED

2.5.4. FAQ

2.5.4.1. dpdk hugepage management conflict issue

trex use older dpdk version than we release cycle source code. When dpdk change the memory management merchanism, trex will meet the following issue.

Trex should run on an independent platform. DUT/Trex should run on two platforms:

  • one is used as TESTER and trex server, another one is used as DUT.(dts/pktgen)
  • one is used as trex server, another one is used as DUT/TESTER.(recommended scheme) This scheme can make sure that trex run on its full status capability.

When trex run with dts on the same platform, trex server sometimes boot up failed for hugepage error.

./t-rex-64  -i --stl -k 4

   Starting Scapy server..... Scapy server is started
   Trying to bind to igb_uio ...
   /usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:85:00.0 0000:8a:00.1
   The ports are bound/configured.
   Starting  TRex v2.41 please wait  ...
   EAL: Can only reserve 1766 pages from 4096 requested
   Current CONFIG_RTE_MAX_MEMSEG=256 is not enough
   Please either increase it or request less amount of memory.
   EAL: FATAL: Cannot init memory

   EAL: Cannot init memory

    You might need to run ./trex-cfg  once
   EAL: Error - exiting with code: 1
     Cause: Invalid EAL arguments

2.5.4.2. trex quit when using Niantic

when bind dut NNT port to igb_uio, peer port will get a link down status, then trex server using NNT nic will quit.

WATCHDOG: task 'master' has not responded for more than 2.00044 seconds - timeout is 2 seconds

*** traceback follows ***

1       0x55a7c779561a ./_t-rex-64(+0x12761a) [0x55a7c779561a]
2       0x7f23da4be1b0 /lib64/libpthread.so.0(+0x121b0) [0x7f23da4be1b0]
3       0x55a7c7942d40 rte_delay_us_block + 128
4       0x55a7c798d731 ixgbe_setup_mac_link_multispeed_fiber + 337
5       0x55a7c79a8f14 ./_t-rex-64(+0x33af14) [0x55a7c79a8f14]
6       0x55a7c7954c72 rte_eth_link_get_nowait + 114
7       0x55a7c776a988 DpdkTRexPortAttr::update_link_status_nowait() + 24
8       0x55a7c77856a6 CGlobalTRex::handle_slow_path() + 118
9       0x55a7c7785ad7 CGlobalTRex::run_in_master() + 759
10      0x55a7c7785e3c ./_t-rex-64(+0x117e3c) [0x55a7c7785e3c]
11      0x55a7c793efba rte_eal_mp_remote_launch + 346
12      0x55a7c7789e1e main_test(int, char**) + 1038
13      0x7f23d9417f2a __libc_start_main + 234
14      0x55a7c7719b9d ./_t-rex-64(+0xabb9d) [0x55a7c7719b9d]


*** addr2line information follows ***

??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0
??:0


./t-rex-64: line 80: 25870 Aborted                 (core dumped) ./_$(

2.5.4.3. other issues

  1. linux kernel version should not be too low.
  2. Trex only works with even number link peers.
  3. Trex only works with nics, which are using the same driver.
  4. Before boot up trex, please make sure the peer ports are on up status.
  5. If you have ran dpdk on the platform which you want to deploy trex-server, reboot the platform to make sure that trex-server can work fine.
  6. If using i40e driver, Trex v2.41 version need i40e nic firmware version newer than 5.02.
  7. trex will drop the received packet, which dst mac is the port mac address.