Search Linux / UNIX log files smartly for an alert or warning error

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Howto, RedHat/Fedora Linux, Suse Linux, Sys admin, Troubleshooting, UNIX last updated September 26, 2007

So how do you find an alert or warning words in a log file over text based session? Simply use old good grep command. Usually I recommend searching following words
=> fail
=> denied
=> segfault
=> segmentation
=> rejected
=> oops
=> warn

Find an alert or warning words from log files

You need to use grep command:
grep {search-word} /path/to/log/file

Find out all segfault error from /var/log/messages file, enter the following command as privileged user:
# grep -i segfault /var/log/messages
Output:

Sep 23 12:20:09 node10 kernel: mutt[8896]: segfault at 0000000000000010 rip 0000000000439d5e rsp 00007fff36a30040 error 6
Sep 24 12:20:10 node10 kernel: mutt[20107]: segfault at 0000000000000010 rip 0000000000439d5e rsp 00007fffd99dbac0 error 6
Sep 25 12:20:09 node10 kernel: mutt[19734]: segfault at 0000000000000010 rip 0000000000439d5e rsp 00007fff5d807290 error 6

Look like node10’s mutt command generated segfault error while sending daily reports attachment via email.

GUI Tools

System Log Viewer is a graphical, menu-driven viewer that you can use to view and monitor your system logs. System Log Viewer comes with a few functions that can help you manage your logs, including a calendar, log monitor and log statistics display.

Redhat / CentOS tool

Redhat (RHEL) Linux offers gui tool called Log Viewer. Type the redhat-logviewer command at a shell prompt or use GUI menus to start the same. You can set filter words (alter words) by clicking on Edit > Preferences menu > Alter tab > Add button

Debian / Ubuntu tool

Debian / Ubuntu Linux also offers GUI tool to view and search log files by setting filters. Click on Applications menu > Choose System Tools > Admin > System Log.
Debian / Ubuntu Linux also offers GUI tool to view and search log files by setting filters

Quick tip: Capture Linux network packets to a file

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Howto, Linux, Networking, OpenBSD, RedHat/Fedora Linux, Security, Suse Linux, Tips, Ubuntu Linux, UNIX last updated September 24, 2007

tcpdump command dump traffic on a network in real time. It prints out a description of the contents of packets on a network interface.

How do I capture network packets to a file?

By default traffic is dumped on a screen. To capture these packets to a file, enter the following command as the root user:
# tcpdump -i eth0 -w traffic.eth0

How do I read packets from a file?

The -w flag causes it to save the packet data to a file called traffic.eth0 for later analysis, and/or with the -r flag, which causes it to read from a saved packet file rather than to read packets from a network interface:
# tcpdump -r traffic.eth0

Force BIND DNS Server to take full advantage of Dual Core Multiple Intel / AMD Cpu

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Howto, Linux, RedHat/Fedora Linux, Sys admin, Tips, Troubleshooting, Tuning, Ubuntu Linux, UNIX last updated September 7, 2007

One of my client runs dedicated NS1 and NS2 to host more than 3000+ domains. Recently they upgraded their servers to latest Dual Core Dual AMD server with CentOS 5.0 and BIND server.

By default BIND / named will try to determine the number of CPUs present and create one thread per CPU. If it is unable to determine the number of CPUs, a single worker thread will be created. However due to some reason the BIND server failed to automatically utilize all of the system’s available CPUs. So how do you force DNS Server to take advantage of multiple CPUs under CentOS Linux?

After a little investigation, named man page pointed out me in right direction ~ -n #CPU option, which creates #cpus worker threads to take advantage of multiple CPUs.

Force BIND DNS Server to take advantage of multiple CPUs

In order to enable multiple CPU open /etc/sysconfig/named file under CentOS / RHEL / Fedora Linux:
# vi /etc/sysconfig/named
To force bind to take advantage of 4 CPUs, add / modify as follows:
OPTIONS="-n 4"
Save and close the file. Restart named service:
# /etc/init.d/named restart

A note about Debian / Ubuntu Linux user

If you are a Debian / Ubuntu Linux modify /etc/defaults/bind9 file:
$ sudo vi /etc/defaults/bind9
Append config line:
OPTIONS="-n 4"
Please note append -n 4 to the end of other options (if any), for example:
OPTIONS="-4 -6 -n 4"
Save and close the file. Restart BIND server:
$ sudo /etc/init.d/bind9 restart

More more information read named man page.

Download: Linux distribution for PPC / PowerPC architecture

Posted on in Categories Ask nixCraft, Debian Linux, Download of the day, Gentoo Linux, GNU/Open source, Linux desktop, Linux distribution, RedHat/Fedora Linux, Ubuntu Linux last updated August 27, 2007

Camira asks:

Where can I find the Linux distribution for the ppc architecture?

I guess the best distro for the PowerPC based PC is Yellow Dog Linux (YDL). Just like other distro it is a free software and open-source Linux distribution for Power Architecture hardware. It is based on Fedora Linux. According to wikipedia:

PowerPC is a RISC microprocessor architecture created by the 1991 Apple–IBM–Motorola alliance, known as AIM. Originally intended for personal computers, PowerPC CPUs have since become popular embedded and high-performance processors as well. PowerPC was the cornerstone of AIM’s PReP and Common Hardware Reference Platform initiatives in the 1990s, but the architecture found the most success in the personal computer market in Apple’s Macintosh lines from 1994 to 2006 (before Apple’s transition to Intel).

PowerPC based system includes Sony PlayStation PS3, IBM Power and pSeries, Mercury Cell, Apple Macintosh G3, G4 & G5 computers, including iBooks, PowerBooks and Apple PowerPC etc.

Download PowerPC Linux distributions

Please note that you can also try out FreeBSD, OpenBSD and NetBSD for the PPC architecture based system.

Howto Use SSH To Run Command On A Remote Machine

Posted on in Categories Automation, CentOS, Debian Linux, FreeBSD, Gentoo Linux, High performance computing, Howto, Linux, Shell scripting, Sys admin, Tuning, Ubuntu Linux, UNIX last updated August 17, 2007

This article examined a simple, but powerful, method to run commands on a remote machine using combination of ssh and a shell script:

Use Secure Shell (SSH) to run commands on remote UNIX systems and, with some simple scripts, put together a system that enables you to manage many systems simultaneously from one machine without having to log in directly to the machines themselves. Also examine the basics of a distributed management system and some scripts and solutions using the technique.

I have already covered how to execute commands on multiple Linux or UNIX servers via a shell script. The disadvantage of shell script is commands do not run in parallel on all servers. However, several tools exist to automate this procedure in parallel. With the help of tool called tentakel (highly recommended) , you run distributed command execution. Also, you can execute commands on multiple Linux or UNIX servers using special tools such as multixterm from expect project.

=> Distributed administration using SSH

Copy hard disk or partition image to another system using a network and netcat (nc)

Posted on in Categories Backup, CentOS, Data recovery, Debian Linux, File system, FreeBSD, Gentoo Linux, Howto, RedHat/Fedora Linux, Suse Linux, Sys admin, Tips, Ubuntu Linux last updated August 12, 2007

netcat utility (nc command) considered as TCP/IP swiss army knife. It reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable “back-end” tool that can be used directly or easily driven by other programs and scripts. At the same time, it is a feature-rich network debugging and exploration tool, since it can create almost any kind of connection you would need and has several interesting built-in capabilities.

I also install the netcat package for administering a network and you’d like to use its debugging and network exploration capabilities.

One my favorite usage is to migrating data between two server hard drives using netcat over a network. It is very easy to copy complete drive image from one server to another.

You can also use ssh for the same purpose, but encryption adds its own overheads. This is tried and trusted method (hat tip to karl) .

Make sure you have backup of all important data.

Install netcat

It is possible that nc may not be installed by default under Redhat / CentOS / Debian Linux.

Insall nc under Redhat / CentOS / Fedora Linux

Use yum command as follows:
# yum install nc
Output:

Loading "installonlyn" plugin
Loading "rhnplugin" plugin
Setting up Install Process
Setting up repositories
rhel-x86_64-server-vt-5   100% |=========================| 1.2 kB    00:00
rhel-x86_64-server-5      100% |=========================| 1.2 kB    00:00
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for nc to pack into transaction set.
nc-1.84-10.fc6.x86_64.rpm 100% |=========================| 6.9 kB    00:00
---> Package nc.x86_64 0:1.84-10.fc6 set to be updated
--> Running transaction check

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 nc                      x86_64     1.84-10.fc6      rhel-x86_64-server-5   56 k

Transaction Summary
=============================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 56 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): nc-1.84-10.fc6.x86 100% |=========================|  56 kB    00:00
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: nc                           ######################### [1/1]

Installed: nc.x86_64 0:1.84-10.fc6
Complete!

Debian / Ubuntu Linux netcat installation

Simply use apt-get command:
$ sudo apt-get install netcat

WARNING! These examples may result into data loss, ensure there are good backups before doing this, as using command wrong way can be dangerous.

How do I use netcat to copy hard disk image?

Our sample setup

-----------------------
HostA // 192.168.1.1
------------------------
           sda
        NETWORK
           sdb
------------------------
HostB // 192.168.1.2
-------------------------

Your task is copy HostA /dev/sda to HostB’s /dev/sdb using netcat command. First login as root user

Command to type on hostB (receiving end ~ write image mode)

You need to open port on hostB using netcat, enter :
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb
Where,

  • -p 2222 : Specifies the source port nc should use, subject to privilege restrictions and availability. Make sure port 2222 is not used by another process.
  • -l : Used to specify that nc should listen for an incoming connection rather than initiate a connection to a remote host.
  • bzip2 -d : Compresses image using the Burrows-Wheeler block sorting text compression algorithm, and Huffman coding. This will speed up network transfer ( -d : force decompression mode)
  • dd of=/dev/sda : /dev/sda is your hard disk. You can also specify partition such as /dev/sda1

Command to type on hostA (send data over a network ~ read image mode)

Now all you have to do is start copying image. Again login as root and enter:
# bzip2 -c /dev/sda | netcat hostA 2222
OR use IP address:
# bzip2 -c /dev/sda | netcat 192.168.1.1 2222

This process takes its own time.

A note about latest netcat version 1.84-10 and above

If you are using latest nc / netcat version above syntax will generate an error. It is an error to use -l option in conjunction with the -p, -s, or -z options. Additionally, any timeouts specified with the -w option are ignored. So use nc command as follows.

On hostA, enter:
# nc -l 2222 > /dev/sdb
On hostB, enter:
# nc hostA 2222< /dev/sda
OR
# nc 192.168.1.1 2222< /dev/sda

Using a second machine (hostB), connect to the listening nc process at 2222 (hostA), feeding it the file (/dev/sda)which is to be transferred. You can use bzip2 as follows.
On hostA, enter:
# nc -l 2222 | bzip2 -d > /dev/sdb
On hostB, enter:
# bzip2 -c /dev/sda | nc 192.168.1.1 2222

Further readings

How do I improve performance?

As suggested by anonymous user:

You should definitely use bs=16M or something like that. Otherwise, the copy will take forever. Copying a 300 GB hard drive over a 1 Gbps cross-over cable took about 1 1/2 hours or so using bs=16M Without this option, the same thing would have taken about 7 hours.

In short use command as follows:
# netcat -p 2222 -l |bzip2 -d | dd of=/dev/sdb bs=16M

Updated for accuracy.

lftp to accelerate ftp / http download speed under Linux and UNIX

Posted on in Categories CentOS, Debian Linux, FreeBSD, Gentoo Linux last updated August 10, 2007

lftp is a file transfer program that allows sophisticated ftp, http and other connections to other hosts. If site is specified then lftp will connect to that site otherwise a connection has to be established with the open command. This is an essential tool for all a Linux admin. I’ve already written about Linux ultra fast command line download accelerator such as Axel and prozilla. lftp is another tool for same job with more features.

lftp can handle seven file access methods :
=> ftp
=> ftps
=> http
=> https
=> hftp
=> fish
=> sftp
=> file

So what’s special about lftp?

  • Every operation in lftp is reliable, that is any not fatal error is ignored and the operation is repeated. So if downloading breaks, it will be restarted from the
    point automatically. Even if ftp server does not support REST command, lftp will try to retrieve the file from the very beginning until the file is transferred completely.
  • lftp has shell-like command syntax allowing you to launch several commands in parallel in background
  • lftp has builtin mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server. Mirror can also synchronize directories between two remote servers, using FXP if available.

Use lftp as download accelerator

lftp has pget command. It allows you download files in parallel. lftp pget command syntax:
lftp -e ‘pget -n 5 ftp://path/to/file

For example, download http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.2.tar.bz2 file using pget in 5 parts:
$ cd /tmp
$ lftp -e 'pget -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.2.tar.bz2'

Output:

45108964 bytes transferred in 57 seconds (775.3K/s)
lftp :~>quit

A note about parallel downloading

Please note that by using download accelerator you are going to put a load on remote host. Also note that lftp may not work with sites that do not support multi-source downloads or blocks such requests at firewall level.

lftp command offers many other features. Refer to lftp man page for more information:
man lftp