Fri, 09 Sep 2005
TCP segmentation offload
Last week LWN had an interesting article about TCP offload engines in the Linux kernel. This week they added a follow-up to that article written by a vendor of TOE-equipped hardware. I don't have a TOE-capable NIC myself, but in the comments section people discussed the whether a full TOE gives any significant advantage when compared to a simple TCP segmentation offload. Mingo said that any modern-enough NIC which supports TSO is also supported under Linux, including tg3 which we use in most of our servers.
I wondered whether my version of the tg3 hardware supports TSO, and if so, whether it is better than no TSO at all. I found that according to ethtool -k eth0 the TSO is switched off at Odysseus. So I tried to switch it on (altough I wondered why it is not switched on by default, provided that the hardware supports this feature).
I tried to measure the difference by downloading an ISO image of FC4 i386 CD1 (665434112 bytes) from two hosts connected to the same switch. I did 10 transfers of the same file with each settings, and took the average and maximum of the last five transfers only (to avoid any start-up temporary conditions). The client Alpha was dual Opteron 248 with Tyan S2882 board, and the client Beta was quad Opteron 848 on HP DL-585 board. The FTP server is dual Opteron 244 on Tyan S2882 board, and ProFTPD with sendfile(2) enabled.
| Client | TSO | Average speed | Max speed |
|---|---|---|---|
| Alpha | off | 108.7 MB/s | 110.5 MB/s |
| Alpha | on | 100.9 MB/s | 101.2 MB/s |
| Beta | off | 102.1 MB/s | 102.4 MB/s |
| Beta | on | 93.2 MB/s | 95.5 MB/s |
Surprisingly enough, the tests without TSO were faster than with it enabled. Looking at tcpdump it seems that the system with TSO enabled sends only a 15 KB-sized frames to the NIC instead of full 64 KB-sized ones:
18:45:38.993150 IP odysseus.ftp-data > alpha.33125: P 127424:143352(15928) ack 1 win 1460 <nop,nop,timestamp 2408404 698142942> 18:45:38.993203 IP odysseus.ftp-data > alpha.33125: P 143352:159280(15928) ack 1 win 1460 <nop,nop,timestamp 2408404 698142942>
So I wonder what is wrong with TSO on my hardware. Any clues? I think I should poke the netdev mailing list.