In this post we are going to review wget utility which retrieves files from World Wide Web (WWW) using widely used protocols like HTTP, HTTPS and FTP. Wget utility is freely available package and license is under GNU GPL License. This utility can be install any Unix-like Operating system including Windows and MAC OS. It’s a non-interactive command line tool. Main feature of Wget of it’s robustness. It’s designed in such way so that it works in slow or unstable network connections. Wget automatically start download where it was left off in case of network problem. Also downloads file recursively. It’ll keep trying until file has be retrieved completely.
First, check whether wget utility is already installed or not in your Linux box, using following command.
# rpm -qa wget
wget-1.12-1.4.el6.i686
Please install it using YUM command in case wget is not installed already or you can also download binary package at http://ftp.gnu.org/gnu/wget/.
# yum -y install wget
The -y option used here, is to prevent confirmation prompt before installing any package. For more YUM command examples and options read the article on 20 YUM Command Examples for Linux Package Management.
1. Single file download
The command will download single file and stores in a current directory. It also shows download progress, size, date and time while downloading.
# wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
--2012-10-02 11:28:30-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.5.3.tar.gz
100%[===================================================================================>] 446,966 60.0K/s in 7.4s
2012-10-02 11:28:38 (58.9 KB/s) - wget-1.5.3.tar.gz
2. Download file with different name
Using -O (uppercase) option, downloads file with different file name. Here we have given wget.zip file name as show below.
# wget -O wget.zip http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
--2012-10-02 11:55:54-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget.zip
100%[===================================================================================>] 446,966 60.0K/s in 7.5s
2012-10-02 11:56:02 (58.5 KB/s) - wget.zip
3. Download multiple file with http and ftp protocol
Here we see how to download multiple files using HTTP and FTP protocol with wget command at ones.
# wget http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig
--2012-10-02 12:11:16-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.5.3.tar.gz
100%[===================================================================================>] 446,966 56.7K/s in 7.6s
2012-10-02 12:11:29 (57.1 KB/s) - wget-1.5.3.tar.gz
--2012-10-02 12:11:29-- ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig
=> wget-1.10.1.tar.gz.sig
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.10.1.tar.gz.sig ... 65
==> PASV ... done. ==> RETR wget-1.10.1.tar.gz.sig ... done.
Length: 65 (unauthoritative)
100%[===================================================================================>] 65 --.-K/s in 0s
2012-10-02 12:11:33 (2.66 MB/s) - wget-1.10.1.tar.gz.sig
FINISHED --2012-10-02 12:11:33--
Downloaded: 2 files, 437K in 7.6s (57.1 KB/s)
4. Read URL’s from a file
You can store number of URL’s in text file and download them with -i option. Below we have created tmp.txt under wget directory where we put series of URL’s to download.
# wget -i /wget/tmp.txt
--2012-10-02 12:34:12-- http://ftp.gnu.org/gnu/wget/wget-1.5.3.tar.gz ftp://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz.sig
Resolving ftp.gnu.org... 208.118.235.20, 2001:4830:134:3::b
Connecting to ftp.gnu.org|208.118.235.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 446966 (436K) [application/x-gzip]
Saving to: wget-1.10.1.tar.gz.sig
100%[===================================================================================>] 446,966 35.0K/s in 10s
2012-10-02 12:34:23 (42.7 KB/s) - wget-1.10.1.tar.gz.sig
--2012-10-02 12:34:23-- http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1761607680 (1.6G) [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso
45%[========================================== ] 1,262,000 51.6K/s eta 8h 17m
5. Resume uncompleted download
In case of big file download, it may happen sometime to stop download in that case we can resume download the same file where it was left off with -c option. But when you start download file without specifying -c option wget will add .1 extension at the end of file, considering as a fresh download. So, it’s good practice to add -c switch when you download big files.
# wget -c http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
--2012-10-02 12:46:57-- http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1761607680 (1.6G), 1758132697 (1.6G) remaining [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso
51% [================================================= ] 3,877,262 47.0K/s eta 10h 27m ^
6. Download file with appended .1 in file name
When you start download without -c option wget add .1 at the end of file and start with fresh download. If .1 already exist .2 append at the end of file.
# wget http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
--2012-10-02 12:50:49-- http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
Resolving mirrors.hns.net.in... 111.91.91.34, 2401:4800::111:91:91:34
Connecting to mirrors.hns.net.in|111.91.91.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1761607680 (1.6G) [application/octet-stream]
Saving to: CentOS-6.3-x86_64-LiveDVD.iso.1
18% [================== ] 172,436 59.2K/s
See the example files with .1 extension appended at the end of the file.
# ls -l CentOS*
-rw-r--r--. 1 root root 3877262 Oct 2 12:47 CentOS-6.3-x86_64-LiveDVD.iso
-rw-r--r--. 1 root root 181004 Oct 2 12:50 CentOS-6.3-x86_64-LiveDVD.iso.1
7. Download files in background
With -b option you can send download in background immediately after download start and logs are written in /wget/log.txt file.
# wget -b /wget/log.txt ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
Continuing in background, pid 3550.
8. Restrict download speed limits
With Option –limit-rate=100k, the download speed limit is restricted to 100k and the logs will be created under /wget/log.txt as shown below.
# wget -c --limit-rate=100k /wget/log.txt ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
/wget/log.txt: Scheme missing.
--2012-10-02 13:16:21-- ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
=> debian-6.0.5-i386-DVD-1.iso
esolving ftp.iinet.net.au... 203.0.178.32
Connecting to ftp.iinet.net.au|203.0.178.32|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD (1) /debian/debian-cd/6.0.5/i386/iso-dvd ... done.
==> SIZE debian-6.0.5-i386-DVD-1.iso ... 4691312640
==> PASV ... done. ==> REST 2825236 ... done.
==> RETR debian-6.0.5-i386-DVD-1.iso ... done.
Length: 4688487404 (4.4G), 4685662168 (4.4G) remaining (unauthoritative)
0% [ ] 3,372,160 35.5K/s eta 28h 39m
9. Restricted FTP and HTTP downloads with username and password
With Options –http-user=username, –http-password=password & –ftp-user=username, –ftp-password=password, you can download password restricted HTTP or FTP sites as shown below.
# wget --http-user=narad --http-password=password http://mirrors.hns.net.in/centos/6.3/isos/x86_64/CentOS-6.3-x86_64-LiveDVD.iso
# wget --ftp-user=narad --ftp-password=password ftp://ftp.iinet.net.au/debian/debian-cd/6.0.5/i386/iso-dvd/debian-6.0.5-i386-DVD-1.iso
10. Find wget version and help
With Options –version and –help you can view version and help as needed.
# wget --version
# wget --help
In this article we have covered Linux wget command with options for daily administrative task. Do man wget if you wan to know more about it. Kindly share through our comment box or if we’ve missed out anything, do let us know.
0 Comments