Linux wget: Your Ultimate Command Line Downloader

by on June 21, 2005 · 40 comments· LAST UPDATED September 12, 2010

in , ,

It is a common practice to manage UNIX/Linux/BSD servers remotely over the ssh session. You may need to download download the software or other files for installation. There are a few powerful graphical download manager exits for Linux and UNIX like operating systems:

  • d4x: Downloader for X is a Linux/Unix userfriendly program with nice X interface to download files from the Internet. It suppotrs both FTP and HTTP protocols, supports resuming
  • kget: KGet is a versatile and user-friendly download manager for KDE desktop system.
  • gwget / gwget2: Gwget is a download manager for the Gnome Desktop

However, when it comes to command line (shell prompt) wget the non-interactive downloader rules. It supports http, ftp, https protocols along with authentication facility, and tons of other options. Here are some tips to get most out of it:

Download a Single File Using wget

Type the following command:
$ wget http://www.cyberciti.biz/here/lsst.tar.gz
$ wget ftp://ftp.freebsd.org/pub/sys.tar.gz
$ wget -O output.file http://nixcraft.com/some/path/file.name.tar.gz
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.36-rc3.tar.bz2

Sample outputs:

Fig.01: wget example

Fig.01: wget example

How Do I Download Multiple Files Using wget?

Use the following syntax:
$ wget http://www.cyberciti.biz/download/lsst.tar.gz ftp://ftp.freebsd.org/pub/sys.tar.gz ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm
You can create a shell variable that holds all urls and use the 'BASH for loop' to download all files:

URLS=”http://www.cyberciti.biz/download/lsst.tar.gz \
ftp://ftp.freebsd.org/pub/sys.tar.gz \
ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm \
http://xyz.com/abc.iso"
for u in $URLS
do
 wget $u
done

How Do I Read URLs From a File?

You can put all urls in a text file and use the -i option to wget to download all files. First, create a text file:
$ vi /tmp/download.txt
Append a list of urls:

http://www.cyberciti.biz/download/lsst.tar.gz
ftp://ftp.freebsd.org/pub/sys.tar.gz
ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm
http://xyz.com/abc.iso

Type the wget command as follows:
$ wget -i /tmp/download.txt

Resume Downloads

You can also force wget to get a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of wget, or by another program:
$ wget -c http://www.cyberciti.biz/download/lsst.tar.gz
$ wget -c -i /tmp/download.txt

Please note that the -c option only works with FTP / HTTP servers that support the "range" header.

Force wget To Download All Files In Background

The -o option used to force wget to go into background immediately after startup. If no output file is specified via the -o option, output is redirected to wget-log file:
$ wget -cb -o /tmp/download.log -i /tmp/download.txt
OR
$ nohup wget -c -o /tmp/download.log -i /tmp/download.txt &
nohup runs the given COMMAND (in this example wget) with hangup signals ignored, so that the command can continue running in the background after you log out.

How Do I Limit the Download Speed?

You can limit the download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix. For example, --limit-rate=100k will limit the retrieval rate to 100KB/s. This is useful when, for whatever reason, you don't want Wget to consume the entire available bandwidth. This is useful when you want to download a large file file, such as an ISO image:
$ wget -c -o /tmp/susedvd.log --limit-rate=50k ftp://ftp.novell.com/pub/suse/dvd1.iso
Use m suffix for megabytes (--limit-rate=1m). The above command will limit the retrieval rate to 50KB/s. It is also possible to specify disk quota for automatic retrievals to avoid disk DoS attack. The following command will be aborted when the quota is (100MB+) exceeded.
$ wget -cb -o /tmp/download.log -i /tmp/download.txt --quota=100m
From the wget man page:

Please note that Wget implements the limiting by sleeping the appropriate amount of time after a network read that took less time than specified by the rate. Eventually this strategy causes the TCP transfer to slow down to approximately the specified rate. However, it may take some time for this balance to be achieved, so don't be surprised if limiting the rate doesn't work well with very small files.

Use wget With the Password Protected Sites

You can supply the http username/password on server as follows:
$ wget --http-user=vivek --http-password=Secrete http://cyberciti.biz/vivek/csits.tar.gz
Another way to specify username and password is in the URL itself.
$ wget 'http://username:password@cyberciti.biz/file.tar.gz
Either method reveals your password to anyone who bothers to run ps command:
$ ps aux
Sample outputs:

vivek     27370  2.3  0.4 216156 51100 ?        S    05:34   0:06 /usr/bin/php-cgi
vivek     27744  0.1  0.0  97444  1588 pts/2    T    05:38   0:00 wget http://test:test@www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.36-rc3.tar.bz2
vivek    27746  0.5  0.0  97420  1240 ?        Ss   05:38   0:00 wget -b http://test:test@www.kernel.org/pub/linux/kernel/v2.6/testing/linux-2.6.36-rc3.tar.bz2

To prevent the passwords from being seen, store them in .wgetrc or .netrc, and make sure to protect those files from other users with "chmod".

If the passwords are really important, do not leave them lying in those files either---edit the files and delete them after Wget has started the download.
G) Download all mp3 or pdf file from remote FTP server:
Generally you can use shell special character aka wildcards such as *, ?, [] to specify selection criteria for files. Same can be use with FTP servers while downloading files.
$ wget ftp://somedom.com/pub/downloads/*.pdf
$ wget ftp://somedom.com/pub/downloads/*.pdf
OR$ wget -g on ftp://somedom.com/pub/downloads/*.pdfH) Use aget when you need multithreaded http download:
aget fetches HTTP URLs in a manner similar to wget, but segments the retrieval into multiple parts to increase download speed. It can be many times as fast as wget in some circumstances( it is just like Flashget under MS Windows but with CLI):
$ aget -n=5 http://download.soft.com/soft1.tar.gzAbove command will download soft1.tar.gz in 5 segments.

Please note that wget command is available on Linux and UNIX/BSD like oses.

See man page of wget(1) for more advanced options.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 40 comments… read them below or add one }

1 JZA November 1, 2006 at 10:49 pm

I remember there was a way to download all .txt file from a URL however I haven’t been able to find it. You mention that wget http://../../*.txt will do it however it generate an error saying:
HTTP request sent, awaiting response… 404 Not Found

Reply

2 nixCraft November 1, 2006 at 10:56 pm

Yes. It can be done provided that remote ftp/web server support this feature. Due to abuse/security or to avoid server load most remote system disables this feature.

Try to use -i option as described above to fetch list of files from text file.
wget -i list.txt

Reply

3 dushyant January 3, 2007 at 11:32 am

if some site not giving ( showing its full URL )and also talking http passwd and user login than
how to get the data directory and how can i use these with wget command
plz mail me on my id i cant remember this site name .

Reply

4 mangesh July 13, 2007 at 7:42 am

download recessively, resume with passive

wget -c -r –passive-ftp -nH ftp://:@//ftp_dir/*.zip

Reply

5 akhil December 4, 2007 at 1:14 pm

wget is not working for HTTPS protocol..please tell me y?

Reply

6 Ramamoorthy November 21, 2013 at 7:48 am

It will work . if it shows certificate error , then use –no-check-certificate .
for example . wget –no-check-certificate https://example.com/example.zip
some https server or even http server will allow download only if you are logged in , in that case you need to log in in your browser , have to get the cookies and use that cookies in wget..

Reply

7 JoshuaTaylor.info May 9, 2008 at 4:46 am

Lol, these instructions are great for my dedicated server for spreading out my RS.com downloads.

Cheers!

Reply

8 Interesting November 20, 2008 at 2:54 pm

Very interesting thanks for the information on Wget my favorite way of uploading to my dedicated servers. Be it an over complicated and complex way but my favorite none the less

Reply

9 sara February 17, 2009 at 6:10 am

thanks alot for your help and usefull informations

Reply

10 David D. March 8, 2009 at 10:03 pm

Is there a way to specify the filename wget writes to?
Would the following do the trick?
“wget $options $url > customfilenameiwant.otherextension”,

Reply

11 nixCraft March 8, 2009 at 10:47 pm

Try following to save file.pdf as output.pdf
wget -O output.pdf http://example.com/file.pdf

Reply

12 Lakshmipathi.G April 29, 2009 at 6:42 am

Really useful info….exactly what i wanted …. thanks

Reply

13 mimo May 2, 2009 at 6:57 am

Nice hints!

What I am missing (and still searching for) is –> how to limit file size of a file to download? One of my current bash scripts is ‘spidering’ certain sites for certain archive files. IF encountered a 5GB *.zip file, It would happily download it, which I don’t want. So: what would be a good practise to limit downloads to, say, 2 MB?

Cheers

Reply

14 Neela.V September 15, 2009 at 6:31 am

Hi, i need to retrieve a file from an http site, which asks for username and password to access the site. How can i retrieve the file using wget? please guide me.

Reply

15 vivian September 18, 2009 at 4:46 am

Hello, i am not able to use wget in my ubuntu system.. whenever i try to download anything for example,
sudo wget http://www.cyberciti.biz/here/lsst.tar.gz
what i get is
–2009-09-18 10:15:15– http://www.cyberciti.biz/here/lsst.tar.gz
Resolving http://www.cyberciti.biz... 74.86.48.99
Connecting to http://www.cyberciti.biz|74.86.48.99|:80… ^C
The above is same for any website.
i am able to do sudo apt-get install and this is the evidence that i am connecting to the internet. But for wget case i am not able to. Is there any config settings that i need to do?

Reply

16 Palash October 15, 2009 at 8:28 am

Many many thanks for give important tips of wget.

Reply

17 Feseha February 13, 2010 at 3:50 am

Quite informative and great time saver!!!
Thanks

Reply

18 zazuge March 7, 2010 at 12:41 pm

multi mirror download of a single file trick
wget -c “url1″ -O image.iso
then use dd to fill zeros to the other chunks
dd if=/dev/zero of=image2.iso bs=1024 count=100000
dd if=/dev/zero of=image3.iso bs=1024 count=200000
dd if=/dev/zero of=image4.iso bs=1024 count=300000

then on multiple terminals do
wget -c “url2″ -O image2.iso
wget -c “url3″ -O image3.iso

now merge
dd if=image2.iso of=image.iso bs=1024 count=100000 skip=100000
dd if=image3.iso of=image.iso bs=1024 count=100000 skip=200000

^^

Reply

19 Mark May 14, 2010 at 3:37 am

Thanks! $ wget -c -i was very helpful. :)

Reply

20 bayyou August 13, 2010 at 5:14 am

$wget -cb -o
********** (10stars)

Reply

21 johnwerhun August 27, 2010 at 6:13 pm

How can I use wget to download Samba in Linux. I’ve tried several times but have been unsuccessful?

Reply

22 Trav September 14, 2010 at 9:35 am

Try MultiGet, it also splits the file up into threads to download faster.

Reply

23 James Locke December 13, 2010 at 2:59 pm

I am trying to download data from my server using wget.

I send the login details and store the cookie. I then rotate through n numbers to copy the data into a new file. The saved file is always blank i.e 0kb file size.

My website stores data on individual pages e.g.: (i have changed my actual site name to “mywebsite”)

‘http://admin.mywebsite.com/index.php/print_view/?html=true&order_id=50

I am trying to rotate through the numbers 50 to 1 and extract the data from each page.

The code I am using is below:

#!/usr/bin/perl

system (“wget –post-data ‘username=Test&password=Test&autologin=1′ –cookies=on –keep-session-cookies –save-cookies=cookie.txt http://admin.mywebsite.com/index.php/login“);

$x = 50;
while ($x <= 1) {
system ("wget –wait=400 –post-data 'html=true&order_id=50' –referer=http://admin.mywebsite.com/ –cookies=on –load-cookies=cookie.txt –keep-session-cookies –save-cookies=cookie.txt http://admin.mywebsite.com/index.php/print_view/&quot;);

system ("wget –post-data 'html=true&order_id=50' –referer=http://admin.mywebsite.com/ –cookies=on –load-cookies=cookie.txt –keep-session-cookies –save-cookies=cookie.txt http://admin.mywebsite.com/index.php/print_view?&quot;);
$x++;
}

How do I modify the code above so data is pulled correctly and the saved files are not blank? Thank you

Reply

24 Priya January 7, 2011 at 10:51 am

Sir,
I have used the command given as example but error is comming as given below.
wget http://www.cyberciti.biz/here/lsst.tar.gz
–21:55:38– http://www.cyberciti.biz/here/lsst.tar.gz
Resolving http://www.cyberciti.biz... 75.126.153.206, 2607:f0d0:1002:51::4
Connecting to http://www.cyberciti.biz|75.126.153.206|:80… failed: Network is unreachable.
Connecting to http://www.cyberciti.biz|2607:f0d0:1002:51::4|:80… failed: Network is unreachable.

Reply

25 zack February 9, 2011 at 4:40 am
wget --http-user=***** --http-passwd=********** -O test.txt -o LogFile.txt http://ds.data.jma.go.jp/gmd/ship/data/ShipData/dd/Z__C_RJTD_yyyymmddhhnn--_SHIP_ORIGINAL_AN.txt

Please help me. I want to download data from this website – http://ds.data.jma.go.jp/gmd/ship/download/howtoget.html using above wget command. The problem is how to download the whole one day data with minutes and hour is changing such as starting from 0000UTC-8am(Malaysia local time) 1/1/2010 until 1/1/2010-2350UTC. The data is available for every 10 minutes and time is in UTC format(hhmm).

variable
yyyy=year, mm=month, dd=day, hh=hour, nn=minute

so for this example, yyyy=2010, mm=01, dd=01, hh=hour, nn=minute, user=mmd, passwd=bmlofeb2011

I hope anyone can help me how to download the data.

Reply

26 Dinesh Jadhav February 15, 2011 at 7:59 am

I want to change the owner of the file with the wget command, what ever the filles wiil be downloaded needed to change the owner.

Reply

27 m3nt4t April 17, 2011 at 11:34 am

The easiest way to avoid changing owner/group of download files via wget is to use sudo (run process as other user) or set the download path to a mount point with specific uid/gid (e.g. uid=henry,gid=henry)

Reply

28 Ajay May 3, 2011 at 10:09 am

Hello,

how can i use the wget command if the following situation arises

1) when connected to a particular server the wget command will download the file. for this if we set a crontab then at the mentioned time the download will happen. but at time if there is a problem with the server and is not getting connected the wget command will overwrite the existing file with a dummy file there by loosing the contents
Is there any way to prevent this?? i.e. when not connected to the server the wget command should not create or overwrite.

Reply

29 Bipin May 20, 2011 at 8:23 pm

any Idea to put password from prompt with wget command.

Thanks,
Bipin Bahuguna

Reply

30 Krishna July 4, 2011 at 12:13 pm

When I use wget to download 5 files from server using a script it sends 1 GET request and waits for server to respond then sends the 2nd and so on. I want the GET to be sent simultaneously irrespective of the response from the server at the same moment. How to do this? Any insights? Thanks

Reply

31 Ajay July 4, 2011 at 1:46 pm

You can write a small shell script such that for 5 different files u write 5 wget command inside the shell script and finally run the shell script.

Reply

32 Ajay July 4, 2011 at 1:55 pm

URLS=”http://www.cyberciti.biz/download/lsst.tar.gz \
ftp://ftp.freebsd.org/pub/sys.tar.gz \
ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm \
http://xyz.com/abc.iso
for u in $URLS
do
wget $u
done

Reply

33 gbviswanadh September 18, 2011 at 11:46 am

how can extract file are in the format of 111.zip.bz2 and abc.zip.bz2 , so is this possible to bzip2 and unzip a these files in at time command,

Reply

34 mohit December 18, 2011 at 6:24 pm

how can i download a file over a https website using wget .

Reply

35 Baronsed July 1, 2012 at 12:36 pm

And what when it comes to get the list of links contained in a web page ?
$ lynx -dump -listonly file > html-list

^^

Reply

36 netlord July 10, 2012 at 3:05 pm

hi
and how can i (most easyli) get files from sourceforge?
this “automatic closiest proxy” selection is awful!

Reply

37 deepu August 25, 2012 at 8:03 pm

hi,
I have to download a .txt file on daily basis through our one of server, these all exercise i have done manualy. and save the file in to unix system. i want to create a automate script who can download and save file into unix directory.

Thanks in advance

Reply

38 mike September 18, 2012 at 10:57 am

I’ve been with windows my whole life until I started pc science in college I started playing around with linux its a whole new world!1

Reply

39 hitayezu June 9, 2013 at 12:43 pm

it great command thanks

Reply

40 Tony pezzella August 13, 2013 at 1:19 am

How do I upload a file using wget?

Reply

Leave a Comment

Tagged as: , , , , , , , , , , ,

Previous post:

Next post: