Practice with TRex ================== Download TREX ------------- TREX should be installed in Tester, it could be downloaded from http://trex-tgn.cisco.com/trex/release/. We recommend to use the latest version v2.88. .. code-block:: console wget http://trex-tgn.cisco.com/trex/release/v2.88.tar.gz To read more about T-Rex stateless mode, read the `T-Rex stateless support guide `__. Configure TREX -------------- Extract the downloaded TREX tarball ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Extract the downloaded TREX file to /opt/trex/:: root@tester:/opt# mkdir trex root@tester:/opt# cd trex root@tester:/opt/trex# tar -zxvf v2.88.tar.gz v2.88/ v2.88/_t-rex-64-debug v2.88/t-rex-64-debug ... v2.88/trex_client_v2.88.tar.gz Generate Configuration File ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Identify the performance test ports in tester ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ execute the command like below to list the ports:: root@tester:/opt/trex/v2.88# ./dpdk_setup_ports.py -s Network devices using DPDK-compatible driver ============================================ Network devices using kernel driver =================================== 0000:03:00.0 'VMXNET3 Ethernet Controller' if=ens160 drv=vmxnet3 unused=igb_uio,vfio-pci,uio_pci_generic *Active* 0000:0b:00.0 'Device 1592' if=ens192f0 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic 0000:0b:00.1 'Device 1592' if=ens192f1 drv=ice unused=igb_uio,vfio-pci,uio_pci_generic Other network devices ===================== Generate Configuration File ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Assume that we use 0000:0b:00.0, 0000:0b:00.1 to connect to DUT port. And the ports topology as below:: trex.0 (0b:00.0/40:a6:b7:0b:55:78) <-------> DUT port 0 (05:00.0/b4:96:91:9f:63:68) trex.1 (0b:00.1/40:a6:b7:0b:55:79) <-------> DUT port 1 (05:00.1/b4:96:91:9f:63:69) We need to generate the trex config file as below:: root@tester:/opt/v2.88# ./dpdk_setup_ports.py -i By default, IP based configuration file will be created. Do you want to use MAC based config? (y/N)y +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+ | ID | NUMA | PCI | MAC | Name | Driver | Linux IF | Active | +====+======+=========+===================+============================================+=========+==========+==========+ | 0 | -1 | 03:00.0 | 00:0c:29:29:30:80 | VMXNET3 Ethernet Controller | vmxnet3 | ens160 | *Active* | +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+ | 1 | -1 | 0b:00.0 | 40:a6:b7:0b:55:78 | Device 1592 | ice | ens192f0 | | +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+ | 2 | -1 | 0b:00.1 | 40:a6:b7:0b:55:79 | Device 1592 | ice | ens192f1 | | +----+------+---------+-------------------+--------------------------------------------+---------+----------+----------+ Please choose an even number of interfaces from the list above, either by ID, PCI or Linux IF Stateful will use order of interfaces: Client1 Server1 Client2 Server2 etc. for flows. Stateless can be in any order. Enter list of interfaces separated by space (for example: 1 3) : 1 2 For interface 1, assuming loopback to its dual interface 2. Destination MAC is 40:a6:b7:0b:55:79. Change it to MAC of DUT? (y/N).y Please enter a new destination MAC of interface 1: b4:96:91:9f:63:68 For interface 2, assuming loopback to its dual interface 1. Destination MAC is 40:a6:b7:0b:55:78. Change it to MAC of DUT? (y/N).y Please enter a new destination MAC of interface 2: b4:96:91:9f:63:69 Print preview of generated config? (Y/n)Y ### Config file generated by dpdk_setup_ports.py ### - version: 2 interfaces: ['0b:00.0', '0b:00.1'] port_info: - dest_mac: b4:96:91:9f:63:68 src_mac: 40:a6:b7:0b:55:78 - dest_mac: b4:96:91:9f:63:69 src_mac: 40:a6:b7:0b:55:79 platform: master_thread_id: 0 latency_thread_id: 1 dual_if: - socket: 0 threads: [2,3,4,5,6,7] Save the config to file? (Y/n)Y Default filename is /etc/trex_cfg.yaml Press ENTER to confirm or enter new file: File /etc/trex_cfg.yaml already exist, overwrite? (y/N)Y Saved to /etc/trex_cfg.yaml. root@tester:/opt/trex/v2.88# We could not modify the DUT mac during the configuration and modify it in the generated file /etc/trex_cfg.yaml. Modify Configuration File ^^^^^^^^^^^^^^^^^^^^^^^^^^ Make sure the DUT mac of the generated TREX file is correct, and add prefix and limit_memory is better:: root@tester:/opt/trex/v2.88# cat /etc/trex_cfg.yaml ### Config file generated by dpdk_setup_ports.py ### - version: 2 interfaces: ['0b:00.0', '0b:00.1'] prefix: TREX limit_memory: 4096 port_info: - dest_mac: b4:96:91:9f:63:68 src_mac: 40:a6:b7:0b:55:78 - dest_mac: b4:96:91:9f:63:69 src_mac: 40:a6:b7:0b:55:79 platform: master_thread_id: 0 latency_thread_id: 1 dual_if: - socket: 0 threads: [2,3,4,5,6,7] root@tester:/opt/trex/v2.88# Replace system scapy with TREX scapy ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There is a known issue whereby trex scapy lib will conflict with scapy installed in the OS. In order to resolve this, we need to replace the scapy in system with trex scapy. #. backup your scapy:: cp -rf /usr/local/lib/python3.6/dist-packages/scapy /usr/local/lib/python3.6/dist-packages/scapy_backup #. unify scapy version with trex:: cp -rf /opt/trex/v2.88/external_libs/scapy-2.4.3/scapy/ /usr/local/lib/python3.6/dist-packages/scapy Configure DTS -------------- Configure execution.cfg ~~~~~~~~~~~~~~~~~~~~~~~ Add the performance test suite and set perf=true:: root@tester:dts# cat execution.cfg [Execution1] crbs=192.168.1.1 drivername=vfio-pci build_type=meson test_suites= tso, targets= x86_64-native-linuxapp-gcc parameters=nic_type=cfg:perf=true root@tester:dts# Configure $DTS_CFG_FOLDER/crbs.cfg ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Set the pktgen_group=trex or pktgen_group=TREX, this item is case insensitive:: root@tester:dts# cat $DTS_CFG_FOLDER/crbs.cfg [192.168.1.1] dut_ip=192.168.1.1 dut_user=root dut_passwd=dutpwd os=linux dut_arch= tester_ip=192.168.1.2 tester_passwd=testerpwd ixia_group= pktgen_group=trex channels=4 bypass_core0=True Configure $DTS_CFG_FOLDER/ports.cfg ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This configuration is just same with PF function test, so if you have completed some functional test, you have no need to modify the $DTS_CFG_FOLDER/ports.cfg, just like below:: root@tester:dts# cat $DTS_CFG_FOLDER/ports.cfg [192.168.1.1] ports = pci=0000:05:00.0,peer=0000:0b:00.0; pci=0000:05:00.1,peer=0000:0b:00.1; In addition, it could be configured as below:: root@tester:dts# cat $DTS_CFG_FOLDER/ports.cfg [192.168.1.1] ports = pci=0000:05:00.0,peer=TREX:0; pci=0000:05:00.1,peer=TREX:1; We recommend to use the first format configuration, as it has no need to do modification when we do functional test. Configure $DTS_CFG_FOLDER/pktgen.cfg ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Fill in the $DTS_CFG_FOLDER/pktgen.cfg with your TREX setting, for the first time, you should set start_trex=yes. Set the IP address of the device which you installed TREX into item ‘server=’, it should be the tester IP. The configuration should as below:: root@tester:dts# cat $DTS_CFG_FOLDER/pktgen.cfg [TREX] trex_root_path=/opt/trex/v2.88 trex_lib_path=/opt/trex/v2.88/automation/trex_control_plane/interactive config_file=/etc/trex_cfg.yaml server=192.168.1.1 pcap_file=/opt/trex/v2.88/stl/sample.pcap core_num=4 #core_mask=0x3 ip_src=16.0.0.1 ip_dst=10.0.0.1 warmup=15 duration=-1 start_trex=yes As the trex_lib_path may be different in different versions, you can find the correct path with the below command:: root@tester:/opt/trex/v2.88# find . -name trex_stl_lib ./automation/trex_control_plane/interactive/trex_stl_lib root@tester:/opt/trex/v2.88# Run DTS performance test with TREX ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now you can start DTS performance test with TREX:: root@tester:/home/zhaohy/dts# ./dts dts: DUT 192.168.1.1 tester: ssh root@192.168.1.1 ... pktgen: ssh root@192.168.1.1 pktgen: cd /opt/trex/v2.88;./t-rex-64 -i --cfg /etc/trex_cfg.yaml -c 4 pktgen: Starting Scapy server..... Scapy server is started Trying to bind to igb_uio ... /usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:0b:00.0 0000:0b:00.1 ... TestTSO: Test Case test_perf_TSO_2ports Begin dut.10.240.183.72: tester: TestTSO: Executing PMD using 1S/1C/2T dut.10.240.183.72: x86_64-native-linuxapp-gcc/app/dpdk-testpmd -l 1,45 -n 4 -a 0000:05:00.0 -a 0000:05:00.1 --file-prefix=dpdk_31529_20210324143008 -- -i --rxd=512 --txd=512 --burst=32 --rxfreet=64 --mbcache=128 --portmask=0x3 --max-pkt-len=9000 --txpt=36 --txht=0 --txwt=0 --txfreet=32 --txrst=32 dut.10.240.183.72: EAL: Detected 88 lcore(s) ... pktgen: Rx Port 0 stats: rx_port: 0, rx_bps: 25354096640.000000, rx_pps: 1239130.250000 pktgen: throughput: pps_rx 5463897.750000, bps_rx 50961129472.000000 pktgen: traffic completed. ... TestTSO: +------------+---------------+------------+ | Frame Size | 1S/1C/2T Mpps | % linerate | +============+===============+============+ | 128 | 5.371 | 4 | +------------+---------------+------------+ | 2500 | 5.464 | 56 | +------------+---------------+------------+ TestTSO: Test Case test_perf_TSO_2ports Result PASSED FAQ --- dpdk hugepage management conflict issue ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ trex use older dpdk version than we release cycle source code. When dpdk change the memory management merchanism, trex will meet the following issue. Trex should run on an independent platform. DUT/Trex should run on two platforms: * one is used as TESTER and trex server, another one is used as DUT.(dts/pktgen) * one is used as trex server, another one is used as DUT/TESTER.(recommended scheme) This scheme can make sure that trex run on its full status capability. When trex run with dts on the same platform, trex server sometimes boot up failed for hugepage error. .. code-block:: console ./t-rex-64 -i --stl -k 4 Starting Scapy server..... Scapy server is started Trying to bind to igb_uio ... /usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:85:00.0 0000:8a:00.1 The ports are bound/configured. Starting TRex v2.41 please wait ... EAL: Can only reserve 1766 pages from 4096 requested Current CONFIG_RTE_MAX_MEMSEG=256 is not enough Please either increase it or request less amount of memory. EAL: FATAL: Cannot init memory EAL: Cannot init memory You might need to run ./trex-cfg once EAL: Error - exiting with code: 1 Cause: Invalid EAL arguments trex quit when using 82599 ~~~~~~~~~~~~~~~~~~~~~~~~~~ when bind dut 82599/500 Series port to igb_uio, peer port will get a link down status, then trex server using 82599/500 Series nic will quit. .. code-block:: console WATCHDOG: task 'master' has not responded for more than 2.00044 seconds - timeout is 2 seconds *** traceback follows *** 1 0x55a7c779561a ./_t-rex-64(+0x12761a) [0x55a7c779561a] 2 0x7f23da4be1b0 /lib64/libpthread.so.0(+0x121b0) [0x7f23da4be1b0] 3 0x55a7c7942d40 rte_delay_us_block + 128 4 0x55a7c798d731 ixgbe_setup_mac_link_multispeed_fiber + 337 5 0x55a7c79a8f14 ./_t-rex-64(+0x33af14) [0x55a7c79a8f14] 6 0x55a7c7954c72 rte_eth_link_get_nowait + 114 7 0x55a7c776a988 DpdkTRexPortAttr::update_link_status_nowait() + 24 8 0x55a7c77856a6 CGlobalTRex::handle_slow_path() + 118 9 0x55a7c7785ad7 CGlobalTRex::run_in_master() + 759 10 0x55a7c7785e3c ./_t-rex-64(+0x117e3c) [0x55a7c7785e3c] 11 0x55a7c793efba rte_eal_mp_remote_launch + 346 12 0x55a7c7789e1e main_test(int, char**) + 1038 13 0x7f23d9417f2a __libc_start_main + 234 14 0x55a7c7719b9d ./_t-rex-64(+0xabb9d) [0x55a7c7719b9d] *** addr2line information follows *** ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ??:0 ./t-rex-64: line 80: 25870 Aborted (core dumped) ./_$( Other issues / notes ~~~~~~~~~~~~~~~~~~~~~ #. linux kernel version should not be too low. #. Trex only works with even number link peers. #. Trex only works with nics, which are using the same driver. #. Before boot up trex, please make sure the peer ports are on up status. #. If you have ran dpdk on the platform which you want to deploy trex-server, reboot the platform to make sure that trex-server can work fine. #. If using i40e driver, Trex v2.41 version need i40e nic firmware version newer than 5.02. #. trex will drop the received packet, which dst mac is the port mac address.