Goal of the evaluation is to assess suitability of FreeBSD -CURRENT platform for 10GE networking. Our interest lies primarily in high-performance real-time multimedia applications, e.g. to use FreeBSD as platform for UDP packet reflectors for uncompressed HD video distribution, thus we're mostly interested in UDP tests with Jumbos enabled. However, in order to make the results more useful to others as well, I have included simple measurements of TCP as well.
The measurements were taken using two sort-of standard tools: iperf and netperf.
Word of caution on iperf: iperf is known to have some performance-limiting bugs, as discussed, e.g., here. See a comment on this in UDP/iperf section below.
Machines were connected through HP 6410cl switch with 10GE LR X2-SC modules and jumbos enabled in given VLAN. Each machine was connected using 2m LC-SC SM fiber. The UDP bandwidth measuremet was also implemented in a point to point (direct) way and showed the same performance, so the rest of measurements has been implemented through the switch.
hw.bce.msi_enable: 1 hw.pci.honor_msi_blacklist: 1 hw.pci.enable_msix: 1 hw.pci.enable_msi: 1
net.inet.tcp.sendspace=8388608 net.inet.tcp.recvspace=8388608 net.inet.udp.recvspace=8388608 net.inet.raw.recvspace=8388608 kern.ipc.maxsockbuf=10000000 kern.random.sys.harvest.ethernet=0 net.inet.ip.process_options=0 net.inet.tcp.log_debug=0 net.isr.direct=1
Id Refs Address Size Name 1 8 0xffffffff80100000 ab1e88 kernel 2 1 0xffffffff80bb2000 11680 if_mxge.ko 3 2 0xffffffff80bc4000 dab8 zlib.ko 4 1 0xffffffff80bd2000 d3a0 mxge_ethp_z8e.ko 5 1 0xffffffff80be0000 d240 mxge_eth_z8e.ko
mxge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4>
ether 00:60:dd:47:6b:f3
inet 172.30.80.10 netmask 0xffff0000 broadcast 172.30.255.255
inet 10.0.10.10 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet 10Gbase-LR (autoselect <full-duplex>)
status: active
For maximum sending/receving performance, the bandwidth specification for the sender was 9900M. This often resulted in overloading the receiver, generally inducing about 10% packet loss. The maximum lossless bandwidth was searched for but the value is limited by iperf capability to send precisely defined bandwidth (the actual bitrate may differ by several hundreds of Mbps for high bandwidth, e.g., requested 8800M, actually sent 9.33Gbps).
Sender: iperf -c 10.0.10.10 -u -l nnnn -i 1 -t 10 -b nnnnM -w 2M
Receiver: iperf -s -u -l 8900 -i 1 -w 8M
where nnnn values for sender were replaced by appropriate values for given measurement.
| Packet size [B] | sending [Gbps] | receiving [Gbps] | max. lossless [Gbps] | max. lossless [kpps] |
|---|---|---|---|---|
| 100 | 0.260 | 0.260 | 0.260 | 300 |
| 500 | 1.18 | 1.12 | 1.00 | 250 |
| 1000 | 2.34 | 2.21 | 2.00 | 250 |
| 1500 | 3.29 | 3.17 | 3.00 | 250 |
| 2000 | 4.19 | 4.10 | 4.00 | 250 |
| 3000 | 5.87 | 5.00 | 4.80 | 200 |
| 4000 | 7.21 | 6.00 | 6.40 | 200 |
| 5000 | 7.90 | 7.76 | 7.80 | 195 |
| 6000 | 8.96 | 8.81 | 8.00 | 166 |
| 7000 | 9.90 | 9.70 | 9.33 | 166 |
| 8500 | 9.90 | 9.90 | 9.90 | 145 |
| 8900 | 9.90 | 9.90 | 9.90 | 139 |
8500B frames
CPU utilization - sender: 13.3% user, 41.8% system, 44.9% idle
CPU utilization - receiver: 13.1% user, 44.1% system, 42.8% idle
100B frames
CPU utilization - sender: 14.7% user, 37.9% system, 47.4% idle
CPU utilization - receiver: 13.3% user, 53.5% system, 33.2% idle
The same measurement has been also perfomed from receiver to sender and shown the symetrical behovior, thus proving equivalence of the machines.
Testing of iperf modifications: Recently, two patches for iperf appeared in Linux kernel mailing list:
-ingo version shows inferior performance compared both the port-based version nad -drew version of iperf. Performance of -drew version is comparable to port-based version. Thus it follows -ingo version needs more attention to find out where the slow-down occurs. One clear thing is it happens both on sender and receiver, as when -drew version is used as a sender, -ingo version is only capable of receiving as much data as when sending from -ingo version. Interesting thing is that -drew version is slighly slower on sending and thus receiver always catches up.
| Packet size [B] | sending [Gbps] | receiving [Gbps] | max. lossless [Gbps] |
|---|---|---|---|
| 100 | 0.051 | 0.051 | 0.051 |
| 500 | 0.225 | 0.225 | 0.225 |
| 1500 | 0.771 | 0.771 | 0.771 |
| 8500 | 4.25 | 4.25 | 4.25 |
| Packet size [B] | sending [Gbps] | receiving [Gbps] | max. lossless [Gbps] |
|---|---|---|---|
| 100 | 0.239 | 0.239 | 0.2391 |
| 500 | 1.10 | 1.10 | 1.10 |
| 1500 | 3.08 | 3.08 | 3.08 |
| 8500 | 9.90 | 9.90 | 9.90 |
Sender: netperf -H 10.0.10.1 -n 4 -t UDP_STREAM -- -s 8M -S 8M -m nnnn -M nnnn
Receiver: netserver -n 4
where nnnn values for sender were replaced by appropriate values for given measurement.
| Packet size [B] | sent bandwidth [Gbps] | received bandwidth [Gbps] | received bandwidth [kpps] |
|---|---|---|---|
| 100 | 0.513 | 0.513 | 641 |
| 500 | 2.01 | 1.55 | 384 |
| 1000 | 3.84 | 3.03 | 378 |
| 1500 | 5.10 | 4.52 | 376 |
| 2000 | 6.13 | 5.76 | 360 |
| 3000 | 8.39 | 8.34 | 347 |
| 4000 | 9.53 | 9.37 | 292 |
| 5000 | 9.54 | 9.54 | 238 |
| 6000 | 9.59 | 9.59 | 200 |
| 7000 | 9.65 | 9.65 | 172 |
| 8500 | 9.66 | 9.65 | 142 |
| 8900 | 9.69 | 9.69 | 136 |
Sender: iperf -c 10.0.10.10 -u -l nnnn -i 1 -t 30 -w 8M
Receiver: iperf -s -u -l nnnn -i 1 -w 8M
where nnnn values for sender were replaced by appropriate values for given measurement.
Inflight had to be disabled (sysctl net.inet.tcp.inflight.enable=0) in order to achieve better stability over series of measurement. The test was run over a short link (RTT=0.045ms according to ping), so inflight was not appropriate. Interesting thing is that with inflight enabled, two successive runs of iperf with the same parameters fluctuated largely: e.g., for 7000B frames, between 6.8Gbps and 7.2Gbps. However, even with inflight disabled, the results fluctuated from occasionally. For smaller frame sizes (and thus smaller bitrates), the results became more consistent.
| Packet size [B] | achieved bandwidth [Gbps] |
|---|---|
| 100 | 0.319 |
| 500 | 1.35 |
| 1000 | 2.37 |
| 1500 | 3.13 |
| 2000 | 3.86 |
| 3000 | 4.78 |
| 4000 | 5.82 |
| 5000 | 6.28 |
| 6000 | 6.62 |
| 7000 | 6.83 |
| 8500 | 6.82 |
| 8900 | 7.02 |
Sender: netperf -H 10.0.10.1 -n 4 -t TCP_STREAM -- -s 8M -S 8M -m nnnn -M nnnn
Receiver: netserver -n 4
where nnnn values for sender were replaced by appropriate values for given measurement.
| Packet size [B] | achieved bandwidth [Gbps] |
|---|---|
| 100 | 0.642 |
| 500 | 2.27 |
| 1000 | 2.96 |
| 1500 | 3.49 |
| 2000 | 4.34 |
| 3000 | 4.46 |
| 4000 | 4.75 |
| 5000 | 4.96 |
| 6000 | 4.99 |
| 7000 | 6.24 |
| 8500 | 6.15 |
| 8900 | 5.56 |
To compare the performance with Linux on the same hardware, we have used:
[ 66.250715] myri10ge: Version 1.3.1-1.248 [ 66.336909] myri10ge 0000:06:00.0: handoff confirmed [ 66.450819] myri10ge 0000:06:00.0: handoff confirmed [ 66.545791] myri10ge 0000:06:00.0: MSI IRQ 503, tx bndry 4096, fw myri10ge_eth_z8e.dat, WC Enabled
FreeBSD (sender) to Linux (receiver), iperf:
Linux (sender) to FreeBSD (receiver), iperf:
Linux (sender) to Linux (receiver), iperf: