Linux ultra fast command line download accelerator

Usually I use wget, which is ultimate command line downloader. However, wget failed to accelerate my download speed. I was downloading 1.4GB file around 800KB/s download speed (this box is hooked to uplink port speed 10 Mbps ). However, remote server was restricting me.

Therefore, I decided to get rid of wget. I have downloaded axel – a light download accelerator for Linux command line.

How Axel works?

Axel does the same thing any other accelerator does: it opens more than one HTTP/FTP connection per download and each connection transfers its own, separate, part of the file. It may sound weird, but it works very well in practice. For example, some FTP sites limit the speed of each connection, therefore opening more than one connection at a time multiplies the allowable bandwidth. Be forewarned that some FTP operators don’t like it when you do this. It is better to open the additional connections on several servers and download from all of them simultaneously. Axel supports this feature too. You may either specify a list of mirrors to use or tell the program to search for mirrors.

This is a perfect tool over remote ssh session for downloading large file.

Install axel

If you are using Debian, type the following command to install axel:
# apt-get install axel

Or you can download axel from official website:
$ wget http://wilmer.gaast.net/downloads/axel-1.0b.tar.gz

Untar the axel:
$ tar -zxvf axel-1.0b.tar.gz

Configre and compile axel:
$ ./configure

Install axel:
# make install

On the other hand, just upload a newly build axel binary to remote Linux server using scp. Usually I do not install gcc c/c++ compilers collection on any of my production web/ftp/MySql servers for security reasons.

How do I use axel?

Just type the command as follows:
$ axel http://download.com/file.tar.gz

Limit speed
You can also specify a speed (bytes per sec) for axel so that it will not eat up all your bandwidth. For example following will try to keep the average speed around 5242880 (5120 Kilobyte per/sec):
$ axel -s 5242880 http://download.com/my.iso

Limit a number of connection
You can also specify number of connection you want to open. For example open 3 connections for downloading:
$ axel -n 3 -s 5242880 http://download.com/my.iso

But how fast is axel?

Here is sample test that demonstrates how fast is axel

$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

Output:

--12:10:31--  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
           => `linux-2.6.17.11.tar.bz2'
Resolving kernel.org... 204.152.191.5, 204.152.191.37
Connecting to kernel.org|204.152.191.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41,288,067 (39M) [application/x-bzip2]

100%[================================================================================>] 41,288,067     2.33M/s    ETA 00:00

12:10:48 (2.31 MB/s) - `linux-2.6.17.11.tar.bz2' saved [41288067/41288067]

$ axel http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

Output:

Initializing download: http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
File size: 41288067 bytes
Opening output file linux-2.6.17.11.tar.bz2.1
Starting download

[  0%]  .......... .......... .......... .......... ..........  [ 247.1KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 408.3KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 566.3KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 707.2KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 836.5KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 975.9KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1079.9KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1210.0KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1303.1KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1422.1KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1508.0KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1629.2KB/s]
..........
...
....
[ 99%]  .......... .......... .......... .......... ..........  [8710.2KB/s]
[ 99%]  .......... .......... .......... .......... ..........  [8680.7KB/s]
[100%]  .......... ..........

Downloaded 39.4 megabytes in 4 seconds. (8681.65 KB/s)

As you see axel downloaded same file in 4 seconds. Another great thing its binary size, I can put axel on boot disk and replace a wget.

prozilla – another good program with GUI frontend

One of the drawback of axel is you can not specify ftp username and password. You can use prozilla program, which also makes multiple connections and downloads a file in multiple parts simultaneously thus enhancing the download speed and dwnloading the file faster than a download with a single connection.

FTP passwords can be specified with the URL, or can be obtained automatically from ~/.netrc if it exists.

Install prozilla

# yum install prozilla

Or download prozilla from official web site.

To use prozilla just type the following command (command line version):
$ proz http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

(click to enlrage image)

Further readings

• Read axel and prozilla man pages
• Axel web site
• Prozilla web site