How to use lftp to accelerate ftp/https download speed on Linux/UNIX

lftp is a file transfer program. It allows sophisticated FTP, HTTP/HTTPS, and other connections. If the site URL is specified, then lftp will connect to that site otherwise a connection has to be established with the open command. It is an essential tool for all a Linux/Unix command line users. I have already written about Linux ultra fast command line download accelerator such as Axel and prozilla. lftp is another tool for the same job with more features. lftp can handle seven file access methods:

ADVERTISEMENTS

  1. ftp
  2. ftps
  3. http
  4. https
  5. hftp
  6. fish
  7. sftp
  8. file

So what is unique about lftp?

  • Every operation in lftp is reliable, that is any not fatal error is ignored, and the operation is repeated. So if downloading breaks, it will be restarted from the point automatically. Even if FTP server does not support REST command, lftp will try to retrieve the file from the very beginning until the file is transferred completely.
  • lftp has shell-like command syntax allowing you to launch several commands in parallel in the background.
  • lftp has a builtin mirror which can download or update a whole directory tree. There is also a reverse mirror (mirror -R) which uploads or updates a directory tree on the server. The mirror can also synchronize directories between two remote servers, using FXP if available.

How to use lftp as download accelerator

lftp has pget command. It allows you download files in parallel. The syntax is
lftp -e 'pget -n NUM -c url; exit'
For example, download http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.2.tar.bz2 file using pget in 5 parts:
$ cd /tmp
$ lftp -e 'pget -n 5 -c http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.2.tar.bz2'

Sample outputs:

45108964 bytes transferred in 57 seconds (775.3K/s)
lftp :~>quit

Where,

  1. pget – Download files in parallel
  2. -n 5 – Set maximum number of connections to 5
  3. -c – Continue broken transfer if lfile.lftp-pget-status exists in the current directory

How to use lftp to accelerate ftp/https download on Linux/Unix

Another try with added exit command:
$ lftp -e 'pget -n 10 -c https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.15.tar.xz; exit'



HTML 5 Video 01: lftp command in action

A note about parallel downloading

Please note that by using download accelerator you are going to put a load on remote host. Also note that lftp may not work with sites that do not support multi-source downloads or blocks such requests at firewall level.

NA command offers many other features. Refer to lftp man page for more information:
man lftp

🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallCentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNCentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
7 comments… add one
  • Terry Rankine Nov 8, 2007 @ 4:20

    Since we are all lazy people, put the quit in the -e command.


    lftp -e 'pget -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.2.tar.bz2;quit'

  • 🐧 nixCraft Nov 8, 2007 @ 13:09

    Terry,

    Heh nice, I just forgot to add quit.

  • rotyyu Jan 20, 2009 @ 3:44

    This tools is really crazy……..

  • John Schultz Aug 19, 2009 @ 17:57

    I was trying to use the lftp mirror command to copy a website from a Windows server to my Ubuntu desktop. The directories are copied just fine, but not the files in the directories. Am I missing something or is lftp not compatible with Windows servers?

    Any answers will be most appreciated.

    John

  • Martin C. Sep 10, 2012 @ 16:50

    Is there any way to use lftp to download two or more files at a time (parallel downloading)?

  • linuxicle Jun 18, 2013 @ 13:31

    It’s been a good help for geeker

  • Oleg Sep 10, 2014 @ 20:51

    Please let me know how to download lftp to UNIX?

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.