Automation Linux/Unix tips from nixCraft

How To Install Puppet System Management Tool To Automate Tasks Across The Network

last updated August 12, 2008 in Automation, Howto, Linux, package management, Tips, UNIX

Puppet is an open source configuration management tool. It is written in Ruby. This software is considered as an alternative to Cfengine configuration management system. I’ve used Cfengine for pushing configurations, applying patches and so on.

Flickrfs: Access / Backup Flickr Photos From Linux

last updated April 20, 2008 in Automation, Backup, Gnome, Howto, Linux, Linux desktop

Flickrfs is a virtual filesystem which mounts your Flickr account on a Linux machine, allowing you to browse through your photos as if they were on a locally connected drive.

Howto Use SSH To Run Command On A Remote Machine

last updated August 17, 2007 in Automation, CentOS, Debian Linux, FreeBSD, Gentoo Linux, High performance computing, Howto, Linux, Shell scripting, Sys admin, Tuning, Ubuntu Linux, UNIX

This article examined a simple, but powerful, method to run commands on a remote machine using combination of ssh and a shell script:

Use Secure Shell (SSH) to run commands on remote UNIX systems and, with some simple scripts, put together a system that enables you to manage many systems simultaneously from one machine without having to log in directly to the machines themselves. Also examine the basics of a distributed management system and some scripts and solutions using the technique.

I have already covered how to execute commands on multiple Linux or UNIX servers via a shell script. The disadvantage of shell script is commands do not run in parallel on all servers. However, several tools exist to automate this procedure in parallel. With the help of tool called tentakel (highly recommended) , you run distributed command execution. Also, you can execute commands on multiple Linux or UNIX servers using special tools such as multixterm from expect project.

=> Distributed administration using SSH

Linux: How To Create Multiple Users Accounts in Batch

last updated August 21, 2006 in Automation, CentOS, Debian Linux, FAQ, Howto, Linux, RedHat/Fedora Linux, Shell scripting, UNIX

The useradd command is used to create a new user or update default new user information from the command line. However, sometimes you need to create a large number of users or import usernames from a text file. You can use newusers command, which update and create new users in batch.

Increase productivity with FTP autologin and macros

last updated February 28, 2006 in Automation, Backup, CentOS, Debian Linux, Howto, Linux, programming, RedHat/Fedora Linux, Ubuntu Linux

You may use many macros under office packages. However, your ftp client also supports macros. You can use ~/.netrc – user configuration file. The .netrc file contains login and initialization information used by the auto-login process and stores macros information. It resides in the user’s home directory.

Turn on FTP client auto login

You need to add username and password to file ~/.netrc. Open config file using a text editor such as vi:
$ vi ~/.netrc
Append or add following lines to it:
machine ftp.myserver.com login USERNAME password PASSWORD
Save file and exit to shell prompt. Make sure, only owner can read the file:
$ chmod 0600 ~/.netrc
To connect type command:
$ ftp ftp.myserver.com

FTP Macros

Now let us say every time you connected to ftp server you would like to switch to binary mode and turn off prompt as well as go to directory /pub/data/backup/rdbms/dump/. You can create a macro to automate all these three steps:

i) Open ~/.netrc ftp configuration file:
$ vi ~/.netrc
ii) Define a macro
You need to use the following syntax:

macdef macro-name1
command1
command2

macdef macro-name2
command1
command2

Please note that each macro definition ends with a null line (consecutive new line characters in a file or carriage returns from the terminal). There is a limit of 16 macros and 4096 total characters in all defined macros. Macros remain defined until a close command is executed.

Append following text to .netrc file:

macdef FOO
bin
prom
cd  /pub/data/backup/rdbms/dump/
ls

Save and close the file. Now connect to ftp server:
$ ftp ftp.myserver.com
Output:

Connected to ftp.myserver.com
220 ftp.myserver.com NcFTPd Server (licensed copy) ready.
Remote system type is UNIX.
Using binary mode to transfer files.

To execute a macro FOO type the command:

ftp> $ FOO

Output:

bin
200 Type okay.
prom
Interactive mode off.
cd /pub/data/backup/rdbms/dump/
250 "/pub/data/backup/rdbms/dump/" is new cwd.
ftp> ls

Execute commands on multiple hosts using expect tool

last updated January 16, 2006 in Automation, CentOS, Debian Linux, Download of the day, File system, FreeBSD, Gentoo Linux, GNU/Open source, High performance computing, Howto, Linux, Linux login control, Monitoring, Networking, OpenBSD, OS X, RedHat/Fedora Linux, Shell scripting, Sys admin, Tips, Tuning

This is Part III in a series on Execute Commands on Multiple Linux or UNIX Servers Simultaneously. The full series is Part I, Part II, and Part III.

In third and final part of this series I will cover expect tool. Expect is a program that talks to other interactive programs according to a script. Expect is useful for running any program which requires interaction between the program and the user. For example

Set or change password via script
Ftp/ssh authentication
Provide answers to command via script
If you have multiple servers then you can execute commands on multiple hosts simultaneously.

Install expect

Expect comes with special pre installed script called multixterm expect. If you are using Debian Linux then use apt-get as follows :

# apt-get install expect
# apt-get install expectk

If you are using Red hat Linux then use up2date command as follows:

# up2date -i expect

Fedora core (RHEL 5) / CentOS Linux user can use yum:

# yum install expect expectk

You can use ports to install expect under FreeBSD or use following command:

# pkg_add -v -r expect

Please note you can download expect from offical web site.

General syntax of multixterm:

multixterm -xc “command arg1 arg2” server1 server2…
Multixterm creates multiple xterms that can be driven together or separately so that you can execute or run on multiple hosts/servers simultaneously. The following command line starts up two xterms using ssh to the hosts 192.168.1.16 and 192.168.1.11:

$ multixterm -xc "ssh root@%n" 192.168.1.11 192.168.1.16

Where,

-xc : The optional -xc argument indicates a command to be run in each named xterm (see -xn). With no -xc argument, the command is the current shell. This means you type command once and it get executed on both ssh hosts/servers.
ssh root@%n : ssh is a command to connect remote server with root user. This name will also be substituted for any %n in the command argument.

Debian Linux stores multixterm at /usr/share/doc/expect directory. You need to type commands in multixterm window itself (stdin window). Click on stdin window and type uptime or w command:
For example when I typed the uptime and w command it looked like as follows:

Tentakel to execute commands on multiple Linux or UNIX Servers

last updated December 27, 2005 in Automation, CentOS, Debian Linux, Download of the day, FreeBSD, Gentoo Linux, Howto, Linux, Monitoring, Networking, OpenBSD, RedHat/Fedora Linux, Sys admin, Tips, Tuning, Ubuntu Linux, UNIX

This is Part II in a series on Execute Commands on Multiple Linux or UNIX Servers Simultaneously. The full series is Part I, Part II, and Part III. Many times, you want to execute a command not only on one server, but also on several servers. For example, find out

Version of kernel
Version of Apache web server
Update static html or images files on all web servers via rsync
Find out user information, server information, memory usage etc
Security/patch checking

tentakel

I have already covered how to execute commands on multiple Linux or UNIX servers via shell script. The disadvantage of script is commands do not run in parallel on all servers. However, several tools exist to automate this procedure in parallel. With the help of tool called tentakel, you run distributed command execution. It is a program for executing the same command on many hosts in parallel using ssh (it supports other methods too). Main advantage is you can create several sets of servers according requirements. For example webserver group, mail server group, home servers group etc. The command is executed in parallel on all servers in this group (time saving). By default, every result is printed to stdout (screen). The output format can be defined for each group.

How it works?

Consider the following sample setup:

admin workstation   Group                  Hosts
|----------------> www-servers        host1, host2,host3
|----------------> homeservers        192.168.1.12,192.168.1.15
IP: 192.168.1.1

You need to install tentakel on admin workstation (192.168.1.1). We have two group servers, first is group of web server with three host and another is homeservers with two hosts.

The requirements on the remote hosts (groups) need a running sshd server on the remote side. You need to setup ssh-key based login between admin workstation and all group servers/hosts to take full advantage of this tentakel distributed command execution method.

System requirement

Tentakel requires a working Python installation. It is known to work with Python 2.3. Python 2.2 and Python 2.1 are not supported. If you are using old version of python then please upgrade it.

Configuration

Let us see howto install and configure tentakel.

Step # 1 : Download tentakel

Visit sourceforge home page to download tentakel or download RPM files from tentakel home page.

Step # 2: Install tentakel

Untar source code, enter:

# tar -zxvf tentakel-2.2.tgz

You should be root user for the install step. To install it type

# make
# make install

Step # 3 Configure groups

For demonstration purpose we will use following setup:

   admin pc                    Group           hosts
Running Debian Linux       homeservers     192.168.1.12 192.168.1.15
User: jadmin

Copy sample tentakel configuration file tentakel.conf.example to /etc directory

# cp tentakel.conf.example /etc/ tentakel.conf

Modify /etc/tentakel.conf according to above setup, at the end your file should look like as follows:

# first section: global parameters
set ssh_path="/usr/bin/ssh"
set method="ssh"  # ssh method
set user="jadmin"   # ssh username for remote servers
#set format="%d %o\n" # output format see man page
#set maxparallel="3"  # run at most 3 commands in parallel

# our home servers with two hosts
group homeservers ()
+192.168.1.12 +192.168.1.15

# localhost
group local ()
+127.0.0.1

Save the file and exit to shell prompt. Where,
group homeservers () : Group name
+192.168.1.12 +192.168.1.15 : Host inclusion. name is included and can be an ip address or a hostname.

Step # 4 Configure SSH password less login

Configure ssh-key based login to avoid password prompt between admin workstation and group servers for jadmin user.

Step # 5 Test tentakel

$ tentakel -g homeservers

interactive mode
tentakel(homeservers)>

Where,
-g groupname: Select the group groupname The group must be defined in the configuration file (here it is homeservers). If not specified tentakel implicitly assumes the default group.

At tentakel(homeservers)> prompt type command uname and uptime command as follows:

exec "uname -mrs"
exec "uptime"

Few more examples
Find who is logged on all homeservers and what they are doing (type at shell prompt)

$ tentakel -g homeservers "w"

Executes the uptime command on all hosts defined in group homeservers:

$ tentakel -g homeservers uptime

As you can see, tentakel is very powerful and easy to use tool. It also supports the concept of plugins. A plugin is a single Python module and must appear in the $HOME/.tentakel/plugins/ directory. Main advantage of plugin is customization according to your need. For example, entire web server or mysql server farm can be controlled according our requirements.
However, tentakel is not the only utility for this kind of work. There are programs that do similar things or have to do with tentakel in some way. The complete list can be found online here. tentakel should work on almost all variant of UNIX/BSD or Linux distributions.

Time is a precious commodity, especially if you’re a system administrator. No other job pulls people in so many directions at once. Users interrupt you constantly with requests, preventing you from getting anything done and putting lots of pressure on you. What do you do? The answer is time management. Read our book review of Time Management for System Administrators. Continue reading Execute commands on multiple hosts using expect tool Part III of this series.

Reference:

Read tentakel man page for tentakel configuration options
tentakel home page

Update: Damon confirmed that it works on Windows too with little modification.