Linux: Download all file from ftp server recursively

last updated in Categories FreeBSD, Gentoo Linux, Howto, Linux, Linux desktop, RedHat/Fedora Linux, Shell scripting, Solaris, Suse Linux, Sys admin, Tips, Ubuntu Linux, UNIX

You can use ncftpget command to download entire ftp directory and sub directories from remote ftp server. Let us say you would like to download /www-data directory and all subdirectories inside this one from ftp.nixcraft.net server. You need to use ncftpget command.

Install ncftp client

ncftp client software can be downloaded from http://www.ncftp.com/ncftp/ and works with FreeBSD, Solaris and all most all UNIX variant. You can also run command as follows to install ncftp:
$ sudo apt-get install ncftp

FTP get directory recursively

ncftpget is Internet file transfer program for scripts and advance usage. You need to use command as follows:
$ ncftpget –R –v –u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data
Where,

  • -R : Copy all subdirectories and files (recursive)
  • -v : Verbose i.e. display download activity and progess
  • -u “USERNAME” : FTP server username, if skipped ncftpget will try anonymous username
  • ftp.nixcraft.net : Ftp server name
  • /home/vivek/backup : Download everything to this directory
  • /www-data : Remote ftp directory you wish to copy

If you get an error which read as follows:

tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
Could not read directory listing data: Connection reset by peer

Then add –T option to ncftpget command:

$ ncftpget –T –R –v –u "ftpuser" ftp.nixcraft.net /home/vivek/backup /www-data

Where,

  • -T : Do not try to use TAR mode with Recursive mode

Linux create your own GnuPG private and public key

last updated in Categories Debian Linux, Gentoo Linux, GNU/Open source, Howto, Linux, RedHat/Fedora Linux, Security, Tips, Ubuntu Linux, UNIX

GNU gpg is encryption and signing tool.

The GNU Privacy Guard (GnuPG or GPG) is a free software replacement for the PGP suite of cryptographic software.

GnuPG encrypts messages using asymmetric keypairs individually generated by GnuPG users. The resulting public keys can be exchanged with other users in a variety of ways, such as Internet key servers. They must always be exchanged carefully to prevent identity spoofing by corrupting public key ↔ ‘owner’ identity correspondences. It is also possible to add a cryptographic digital signature to a message, so the message integrity and sender can be verified, if a particular correspondence relied upon has not been corrupted.

How do I create my own GnuPG private and public key

1) Login to your shell account

2) Use gpg command to create the keys
$ gpg --gen-key
Output:

gpg (GnuPG) 1.4.1; Copyright (C) 2005 Free Software Foundation, Inc.
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions. See the file COPYING for details.

gpg: directory `/home/vivek/.gnupg' created
gpg: new configuration file `/home/vivek/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/vivek/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/vivek/.gnupg/secring.gpg' created
gpg: keyring `/home/vivek/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) DSA and Elgamal (default)
   (2) DSA (sign only)
   (5) RSA (sign only)
Your selection? Press [Enter] Key
DSA keypair will have 1024 bits.
ELG-E keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) Press [Enter] Key
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) Press [Enter] Key
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) "

Real name: Vivek Gite
Email address: vivek@nixcraftcorp.com
Comment:[Enter] key
You selected this USER-ID:
    "Vivek Gite "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

Enter passphrase: [Enter password twice]
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++....+++++++++++++++..++++++++++..++++++++++...++++++++++++++++++++.+++++++++++++++++++++++++++++++++++.+++++..++++++++++++++++++++.++++++++++..+++++.+++++.+++++>+++++....+++++>.+++++...................................................................+++++^^^^^^^^^^^
gpg: /home/vivek/.gnupg/trustdb.gpg: trustdb created
gpg: key 8E19F126 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   1024D/8E19F126 2007-02-10
      Key fingerprint = A7AF E25D 3E8D 6946 37CC  8CCE 12C4 8DC1 8E19 F126
uid                  Vivek Gite 
sub   2048g/032824B9 2007-02-10

3) Now keys generated, you can list your own key using:
$ gpg -K
OR
$ gpg --list-keys
Output:

/home/vivek/.gnupg/pubring.gpg
------------------------------
pub   1024D/CA7A8402 2007-02-10
uid                  Vivek Gite 
sub   2048g/0A7B4F93 2007-02-10

Let us try to understand the line pub 1024D/CA7A8402 2007-02-10:

  • pub : Public key
  • 1024D : The number of bits in the key
  • CA7A8402 : The key ID
  • 2007-02-10 : The date of key creation
  • Vivek Gite : The user real name
  • <vivek@nixcraftcorp.com> : The email id

Most important is the key ID i.e. CA7A8402. Make sure you use powerful passphrase to protect keys and not the easy one.

4) To list secret key, type the command:
$ gpg --list-secret-keys
Output:

/home/vivek/.gnupg/secring.gpg
------------------------------
sec   1024D/CA7A8402 2007-02-10
uid                  Vivek Gite 
ssb   2048g/0A7B4F93 2007-02-10

Bash Shell: Find Out Linux / FreeBSD / UNIX System Load Average

last updated in Categories Linux, Monitoring, Sys admin, UNIX

Yes, I know we can use the uptime command to find out the system load average. The uptime command displays the current time, the length of time the system has been up, the number of users, and the load average of the system over the last 1, 5, and 15 minutes. However, if you try to use the uptime command in script, you know how difficult it is to get correct load average. As the time since the last, reboot moves from minutes, to hours, and an even day after system rebooted. Just type the uptime command:

Shell scripting (BASH) : How to accept password in script

last updated in Categories Linux, Shell scripting, Sys admin, Tips, UNIX

You can use read command with -s option, which causes input coming from a terminal to not be echoed. This is useful to verify password or something else.

One line is read from the keyboard or from file descriptor FD if the -u option is supplied, and the first word is assigned to the first NAME, the second word to the second NAME, and so on, with leftover words assigned to the last NAME. Only the characters found in $IFS are recognized as word delimiters. If no NAMEs are supplied, the line read is stored in the REPLY variable. If the -r option is given, this signifies `raw’ input, and backslash escaping is disabled. The -d option causes read to continue until the first character of DELIM is read, rather than newline. If the -p
option is supplied, the string PROMPT is output without a trailing newline before attempting to read. If -a is supplied, the words read are assigned to sequential indices of ARRAY, starting at zero. If -e is supplied and the shell is interactive, readline is used to obtain the line. If -n is
supplied with a non-zero NCHARS argument, read returns after NCHARS characters have been read. The -s option causes input coming from a terminal to not be echoed.

Following script, demonstrate this concept:

1) Download/view the passwordscript.bash script

2) Save the changes and execute script:
$ chmod +x passwordscript.bash
$ ./passwordscript.bash

Linux > More on USER ID, Password, and Group management

last updated in Categories FAQ, Howto, Linux, UNIX

In order login into Linux system (over ssh or other services ) you need a username and password.
Username and password stored in /etc/passwd and /etc/shadow file respectively. When you supplies password, it encrypts and compare with password stored in /etc/shadow, which is also in, encrypted format (it was stored when you or system administrator registers/updates it). If both are equal, you are in. Once logged in, you become the number to Linux kernel. You can obtain your user id and other information using id command:

$ id
uid=1002(vivek) gid=1002(vivek) groups=1002(vivek), 0(wheel)

Where,
=> Username = vivek
=> User numeric id (uid) = 1002

Numbers are uses to represent users and groups in Linux kernel because:
1) Simplified user and group management
2) Security management easy
3) Your UID applied to all files you create

It is always good idea to use the UID more than 1000 for all users for security reason.

Zero UID

The UID number 0 is special and used by the root user. The zero (0) UID enjoys the unrestricted/unlimited access to Linux system. Note that 0 UID assigned to name root; if you wish you can change this (poorly written program may fail) and assign different name.

Similarly, you have group id (GID). It is use by Linux to refer group names. Single user can be member of multiple groups. This result into very good flexibility for access the system and the sharing files. Many UNIX system uses wheel group as power user group. Like the UID value, zero GID value zero enjoys the unrestricted/unlimited access to Linux system.

Some time Linux and other UNIX like (FreeBSD, Solaris etc) uses EUID, RUID, and SUID concept.

The Effective User ID (EUID)

It is use to determine what level of access the current process has. When EUID is zero then the process has unrestricted/unlimited access. Following commands can be used to print Effective User ID under Linux:
$ whoami
$ id -un

The Real User ID (RUID):

It is use to identify who you actually are. Once it is setup by system (usually login program) it cannot be change till your session terminates. You cannot change your RUID. Only root (or person having zero UID) can change the RUID. Use the command id as follows to obtain Real user ID:
$ id –ru

The Saved User ID (SUID):

When new process / executable file such as passwd, started the effective user id that is in force at the time is copied to the saved user id. Because of this feature, you are able to update your own password stored in /etc/shadow file. Off course, executable file must have set-user-id bit on in order to setuid (system call). Before process ending itself it switches back to SUID.

In short,

  • RUID : Identify the real user, normal user cannot change it.
  • EUID : Decides access level, normal user can change it.
  • SUID : Saves the EUID, normal user cannot change it.
  • Real Group ID : Identify the real group
  • Effective Group ID and Supplementary group ID : Decides access level

Note that access level means kernel can determine whether you have access to devices, files etc.

Solaris tar command to backup data on tape device

last updated in Categories Howto, Solaris, Sys admin, Tips, UNIX

Tar name come from Tape ARchiver. It is both a file format and the name of the program used to handle such file. Tar archive files have names ending in “.tar”. If an archive is compressed, the compression program adds its own suffix as usual, resulting in filename endings like “.tar.Z”, “.tar.gz”, and “.tar.bz2”. Tar doesn’t require any particular filename suffix in order to recognize a file as an archive. Tar was originally created for backups on magnetic tape, but it can be used to create tar files anywhere on a filesystem. Archives that have been created with tar are commonly referred to as tarballs.

Create a new set of backup

To create a Tar file, use tar command as follows:
# tar cvf /dev/rmt/X file1 file2 dir1 dir2 file2 …
Where

  • c – Create a new files on tape/archive
  • v – verbose i.e. show list of files while backing up
  • f – tape device name or file

For example, backup /export/home/vivek/sprj directory to tape device /dev/rmt/0, enter
# tar cvf /dev/rmt/0 /export/home/vivek/sprj/
Remember c option should only use to create new set of backup.

Appending or backing up more files to same tape using tar

tar provides r option for appending files to tape. For example to backup /data2/tprj/alpha1 files to same tape i.e. appending files to a first tape device:
# tar rvf /dev/rmt/0 /data2/tprj/alpha1/*
Where

  • r – append files to the end of an archive/tape

List files on a tape using tar command

To display file listing of a first tape use tar as follows:
# tar tvf /dev/rmt/0
To listing the Contents of a Stored Directory (for example wwwroot directory):
# tar tvf /dev/rmt/0 wwwroot
Where

  • t – list the contents of an archive/tape

Retrieve / restore tape backup taken with tar

1) Use tar command as follows to retrieve tape drive backup to current directory:
(a) Change directory where you would like to restore files:
# cd /path/to/restore
# pwd

(b) Now, do a restore from tape:
# tar xvf /dev/rmt/0

To specify target directory use –C option

Restore everything to /data2 directory:
# tar xvf /dev/rmt/0 –C /data2
To retrieve directory or file use tar as follows:
# tar xvf /dev/rmt/0 tprj
Note that Solaris tar command is little different from GNU tar, if you wish to use gnu tar with Solaris use command gtar. Gnu tar accepts same command line options plus bunch of additional options 🙂

See Sun Solaris tar man page and tapes ~ creates /dev entries for tape drives attached to the system.

Linux > Command line BitTorrent client

last updated in Categories Linux, RedHat/Fedora Linux, Shell scripting, Suse Linux, Tip of the day, Tips, Ubuntu Linux, UNIX

BitTorrent is the name of a peer-to-peer (P2P) file distribution protocol, and of a free software implementation of that protocol.

The BitTorrent client is a computer program developed by BitTorrent, Inc. used to download and upload files via the BitTorrent protocol. You can download python based command line BitTorrent client download for Linux/BSD/Mac OS X/UNIX link oses. This is quite useful, if you would like to download file remotely over SSH session. bittorrent.com offers bittorrent client in .deb (Debian Linux) or rpm (RedHat/Fedora Core/CentOS Linux) file formats.

To use the bittorrent in command line one you need to use command bittorrent-console or bittorrent-curses (see below for examples).

Step # 1: Find Python version

Make sure you download file according to your python version. Type following command to find out python version:
$ python -V
Output:

Python 2.4.2

Step # 2: Download bittorrent client

Use wget to download bittorrent client

Download bittorrent client for Debian Linux:

$ wget http://download.bittorrent.com/dl/bittorrent_5.0.3_python2.4.deb

Download bittorrent clientFor Red Hat / Fedora Core / Cent OS Linux

$ wget http://download.bittorrent.com/dl/BitTorrent-5.0.3-1-Python2.4.noarch.rpm

Step # 3: Linux Install BitTorrent client

First login as a root user (use su or sudo command). If you are using Debian Linux, use dpkg command install bittorrent client
# dpkg -i bittorrent_5.0.3_python2.4.deb
If you are using Red Hat / Fedora Core /Cent OS Linux, use rpm command install bittorrent client:
# rpm -ivh BitTorrent-5.0.3-1-Python2.4.noarch.rpm

Step # 4: Start using or downloading files

Use bittorrent client as follows:
$ /usr/bin/bittorrent-curses 'http://www.some.org/DesktopBSD.iso.torrent'
OR
$ bittorrent-curses 'http://www.some.org/DesktopBSD.iso.torrent'
OR
$ /usr/bin/bittorrent-curses '/path/to/file.torrent'
You can try out pure console based client bittorrent-console instead of curses based client:
$ bittorrent-console '/path/to/file.torrent'
$ /usr/bin/bittorrent-console 'http://www.some.org/DesktopBSD.iso.torrent'

Tips about using and Troubleshooting BitTorrent client

(A) If you get [Errno 2] No such file or directory, you need to use wget or lynx command to download .torrent file to local hard drive first. For example
$ bittorrent-curses 'http://www.mininova.org/get/some_file[222].torrent'
First download download .torrent file using any one of the following method:
$ lynx 'http://www.mininova.org/get/some_file[222].torrent'
When prompted save .torrent file to a hard-disk. Another option is use wget to download .torrent file:
$ wget 'http://www.mininova.org/get/some_file[222].torrent'
Now start the downloading as follows:
$ bittorrent-curses 'some_file[222].torrent'

(B) If you get error ‘This seems to be an old Python version which does not support detecting the filesystem encoding. Assuming ‘ascii’, then upgrade your python version to 2.4.xx:

Debian user upgrade python version using apt-get command:
# apt-get update
# apt-get install python2.4

Red Hat Linux user try out:
# up2date python2.4
Fedora Linux user try out:
# yum install python2.4

Remember if you have both versions (python v2.3 and v2.4) installed, you need to run bittorent client as follows:
$ python2.4 /usr/bin/bittorrent-curses '/path/to/file.torrent'
Command line BitTorrent client should work with other UNIX like operating systems such as FreeBSD/OpenBSD/Solaris etc (as long as you have Python it should work).

The current client enables a range of features including multiple parallel downloads. It also intermediates peering between itself, source file servers (read as trackers) and other clients, thereby yielding great distribution efficiencies. The client also enables users to create and share torrent files. See help files for more information.

See also: