≡ Menu

linux command line

Linux debugfs Hack: Undelete Files

Undeletion means restoring files which have been deleted from Linux ext3 file system using rm command. Deleted files can be recovered on ext3 file systems using the debugfs program. This quick tutorial describes how to recover a file that was recently deleted using nothing but standard Linux command line utilities.
[click to continue…]

News Round Up Feb-07, 2009

It's been over 10 days since I've blogged. I was busy with workload so I could not update the blog. I missed lots of news worthy stuff. Here is a quick news roundup:

a] A new custom version of Ubuntu aimed at netbooks and based on 8.04 Hardy Heron has been released by HP.

b] Open source Exchange server replacement only a year away.

c] Scripting SSH access and file transfers with Python can be frustrating - but the Paramiko module solves that in a powerful way.

d] Amazon offering 3 Full Version Games for a Limited period of time.

e] cyberciti.biz / nixcraft.com is now IPv6 enabled.

[click to continue…]

Cheat sheet act as a reference tool which provides cut and paste kind of commands to complete a specific task. I often recommend following set of best cheat sheets to students and IT professionals. It include Linux / UNIX command and shell scripting.
[click to continue…]

Mastering shell prompt can save tons of time. From the article:

The way you interface with a computer is changing constantly. Operating systems that once started as a command line-only interface have moved to a graphical front end. But moving away from what made the operating system great isn't always a step in the right direction. The IBM® AIX® operating system has kept to what's important: stability, functionality, robustness. And it has done it by keeping a strong command-line interface (CLI). If you never learned to use the CLI or need a refresher on its basics, read on.

After reading this article, you should now be able to use the Korn shell in ways you may not have known before. Mastering the command line can simplify your work and help you better understand how to make the shell and command line work for you rather than you working harder for it.

Linux Command Line List ( PDF Version )

This is easy to use Linux command line index. Linux commands divided into categories such as:
=> System information
=> Shutdown
=> Files and Directory
=> File search
=> Mounting a Filesystem
=> Disk Space
=> Users and Groups and others

Linux Commands Line list

Linux ultra fast command line download accelerator

Usually I use wget, which is ultimate command line downloader. However, wget failed to accelerate my download speed. I was downloading 1.4GB file around 800KB/s download speed (this box is hooked to uplink port speed 10 Mbps ). However, remote server was restricting me.

Therefore, I decided to get rid of wget. I have downloaded axel - a light download accelerator for Linux command line.

How Axel works?

Axel does the same thing any other accelerator does: it opens more than one HTTP/FTP connection per download and each connection transfers its own, separate, part of the file. It may sound weird, but it works very well in practice. For example, some FTP sites limit the speed of each connection, therefore opening more than one connection at a time multiplies the allowable bandwidth. Be forewarned that some FTP operators don't like it when you do this. It is better to open the additional connections on several servers and download from all of them simultaneously. Axel supports this feature too. You may either specify a list of mirrors to use or tell the program to search for mirrors.

This is a perfect tool over remote ssh session for downloading large file.

Install axel

If you are using Debian, type the following command to install axel:
# apt-get install axel

Or you can download axel from official website:
$ wget http://wilmer.gaast.net/downloads/axel-1.0b.tar.gz

Untar the axel:
$ tar -zxvf axel-1.0b.tar.gz

Configre and compile axel:
$ ./configure

Install axel:
# make install

On the other hand, just upload a newly build axel binary to remote Linux server using scp. Usually I do not install gcc c/c++ compilers collection on any of my production web/ftp/MySql servers for security reasons.

How do I use axel?

Just type the command as follows:
$ axel http://download.com/file.tar.gz

Limit speed
You can also specify a speed (bytes per sec) for axel so that it will not eat up all your bandwidth. For example following will try to keep the average speed around 5242880 (5120 Kilobyte per/sec):
$ axel -s 5242880 http://download.com/my.iso

Limit a number of connection
You can also specify number of connection you want to open. For example open 3 connections for downloading:
$ axel -n 3 -s 5242880 http://download.com/my.iso

But how fast is axel?

Here is sample test that demonstrates how fast is axel

$ wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

Output:

--12:10:31--  http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
           => `linux-2.6.17.11.tar.bz2'
Resolving kernel.org... 204.152.191.5, 204.152.191.37
Connecting to kernel.org|204.152.191.5|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41,288,067 (39M) [application/x-bzip2]
100%[================================================================================>] 41,288,067     2.33M/s    ETA 00:00
12:10:48 (2.31 MB/s) - `linux-2.6.17.11.tar.bz2' saved [41288067/41288067]

$ axel http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

Output:

Initializing download: http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2
File size: 41288067 bytes
Opening output file linux-2.6.17.11.tar.bz2.1
Starting download
[  0%]  .......... .......... .......... .......... ..........  [ 247.1KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 408.3KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 566.3KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 707.2KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 836.5KB/s]
[  0%]  .......... .......... .......... .......... ..........  [ 975.9KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1079.9KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1210.0KB/s]
[  0%]  .......... .......... .......... .......... ..........  [1303.1KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1422.1KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1508.0KB/s]
[  1%]  .......... .......... .......... .......... ..........  [1629.2KB/s]
..........
...
....
[ 99%]  .......... .......... .......... .......... ..........  [8710.2KB/s]
[ 99%]  .......... .......... .......... .......... ..........  [8680.7KB/s]
[100%]  .......... ..........
Downloaded 39.4 megabytes in 4 seconds. (8681.65 KB/s)

As you see axel downloaded same file in 4 seconds. Another great thing its binary size, I can put axel on boot disk and replace a wget.

prozilla - another good program with GUI frontend

One of the drawback of axel is you can not specify ftp username and password. You can use prozilla program, which also makes multiple connections and downloads a file in multiple parts simultaneously thus enhancing the download speed and dwnloading the file faster than a download with a single connection.

FTP passwords can be specified with the URL, or can be obtained automatically from ~/.netrc if it exists.

Install prozilla

# yum install prozilla

Or download prozilla from official web site.

To use prozilla just type the following command (command line version):
$ proz http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.11.tar.bz2

prozilla - another good program with GUI frontend
(click to enlrage image)

Further readings

I already wrote about Linux command line bittorrent client. However, I received few more queries regarding firewall issues. Basically you need to open ports using iptables.

Bittorrent client by default uses tcp 6881 to 6889 ports only. In order to work with Bittorrent client you need to open these ports on firewall. Remember, if you are behind a firewall (hardware or software) you need to enable port forwarding to internal systems.

Scenario # 1: Windows or Linux desktop behind router firewall

Internet ->     Hardware Router    -> Your Linux Desktop
          with port forwarding          Client
              enabled

You have router (ADSL/DSL/Cable modem+router) and you have already enabled port forwarding on router (open web browser > Open router web admin interface > Find port forwarding > Enable port forwarding for bittorent protocol). You also need to open port using following iptables rules on Linux desktop (open TCP port 6881 to 6999):

iptables -A INPUT -p tcp --destination-port 6881:6999 -j ACCEPT
iptables -A OUTPUT -p tcp --source-port 6881:6999 -j ACCEPT

Here is a complete sample firewall script:

#!/bin/sh
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
modprobe ip_conntrack
modprobe ip_conntrack_ftp
# Setting default filter policy
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
# Unlimited access to loop back
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
# Allow UDP, DNS and Passive FTP
iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
#allow bittorent incomming client request
iptables -A INPUT -p tcp --destination-port 6881:6999 -j ACCEPT
#Uncomment below to allow sshd incoming client request
#iptables -A INPUT -p tcp -dport 22 -j ACCEPT
# DROP everything and Log it
iptables -A INPUT -j LOG
iptables -A INPUT -j DROP

Scenario # 2

Internet -> Linux computer Router  ->  Your Linux Desktop
         with port forwarding      OR Windows XP client
         enabled using IPTABLES       IP:192.168.1.2
           IP:192.168.1.254

Here you are using a Linux as software firewall and iptables as your NAT (firewall) for internal network (192.168.1.2). You need to enable port forwarding to a internal Linux desktop (may be Windows XP desktop) for BitTorrent client system. Add following two line of code to your existing NAT firewall script.

iptables -t nat -A PREROUTING -p tcp --dport 6881:6889
-j DNAT --to-destination 192.168.1.2
iptables -A FORWARD -s 192.168.1.2 -p tcp --dport 6881:6889
-j ACCEPT

Related: Linux Command line BitTorrent client