Transferring Large Amount of Data Over the Network: SCP, TAR; SSH, TAR; NC Compared

Scp is slow, that’s a known fact. Known and so annoying that someone tried to fix it by producing the hpn-ssh patch:

SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwidth network links.

Nonetheless, especially for small transfers, scp is straightforward and so that’s what I use. But transferring 100GB of data between 2 machines on the same LAN proved to be such a pain that I decided to opt for one of the alternatives, the 2 most common being tar over ssh and tar over netcat. The whole thing got me curious so I decided to do some testing/bechmarking.

This is no scientific test. There was background noise, OSes of the box were different, and more. But it’s good enough as a real life test between two boxes on the same LAN.

Test bed

Two boxes, referred to as hostA and hostB from now on, with the same specs:

vendor_id : AuthenticAMD
model name : AMD Sempron(tm) Processor 2800+
cpu MHz : 1600.010
MemTotal : 2009992 kB
SATA disks:  Timing cached reads: 1243.04 MB/sec
Timing buffered disk reads: 57.97 MB/sec
Network : VIA Technologies, Inc. VT6102 [Rhine-II] (rev 78)
Switch : Netgear 10/100 Mbs

Boxes were connected via a 10/100Mbs switch, living on the same LAN/subnet. Given the above setup it’s safe to assume that the network is the bottleneck, with its theoretical 12MB/s peak transfer rate.

Test cases and data set

I’ve created 2 directories, one containing 2000 100KB files , and the other 200 10MB files. All files I’ve been created using dd if=/dev/urandom of=file. These are the commands I’ve compared:

hostA: scp -r dir user@hostB:/tmp/
hostA: tar cf – dir | ssh user@hostB tar xf – -C /tmp/
hostA: tar cf – dir | nc -w1 hostB 6969 \
on hostB: nc -l -p 6969 | tar xf – -C /tmp/

I’ve also run a set of tests using ssh compression and tar gzip compression. To be noted that bzip2 compression is too CPU expensive to be generally worth it.

Results

This is a summary with totals for the entire dataset transfer with times in seconds

Conclusions

Scp is by far the slowest transfer method, 623% slower than the fastest case scenario. Contrary to the common conception that it’s ssh’s encryption layer to slow down the transfer, it is really scp being slow, as tar over ssh performs as good as over nc. The other 2 things to consider are the disastrous impact of ssh’s traffic compression (-C), which surprisingly slows down the transfer of roughly 42% in the case of scp and even 270% in the tar over ssh test, and the tar gzip compression, which results in transfers
being 87% faster over ssh and 134.38% over nc.