≡ Menu

sysadmin

Power Off Server Once In a While?

From my mailbag:

I turn off the PC at home or the office once in a while. Now, I've server at colocation center. Do we need to run server 24/7? I do reboot the server once a month. Is it advisable to completely power off a server once in a while instead of 24/7 running?

[click to continue…]

How To Patch Running Linux Kernel Source Tree

Yesterday, I wrote about a serious Linux kernel bug and fix. However, few readers like to know about patching running Linux kernel. Patching production kernel is a risky business. Following procedure will help you to fix the problem.

Step # 1: Make sure your product is affected

First find out if your product is affected by reported exploit. For example, vmsplice() but only affects RHEL 5.x but RHEL 4.x,3.x, and 2.1.x are not affected at all. You can always obtain this information by visiting vendors bug reporting system called bugzilla. Also make sure bug affects your architectures. For example, a bug may only affect 64 bit or 32 bit platform.

Step # 2: Apply patch

You better apply and test patch in a test environment. Please note that some vendors such as Redhat and Suse modifies or backports kernel. So it is good idea to apply patch to their kernel source code tree. Otherwise you can always grab and apply patch to latest kernel version.

Step # 3: How do I apply kernel patch?

WARNING! These instructions require having the skills of a sysadmin. Personally, I avoid recompiling any kernel unless absolutely necessary. Most our production boxes (over 1400+) are powered by mix of RHEL 4 and 5. Wrong kernel option can disable hardware or may not boot system at all. If you don't understand the internal kernel dependencies don't try this on a production box.

Change directory to your kernel source code:
# cd linux-2.6.xx.yy
Download and save patch file as fix.vmsplice.exploit.patch:
# cat fix.vmsplice.exploit.patch
Output:

--- a/fs/splice.c
+++ b/fs/splice.c
@@ -1234,7 +1234,7 @@ static int get_iovec_page_array(const struct iovec __user *iov,
                if (unlikely(!len))
                        break;
                error = -EFAULT;
-               if (unlikely(!base))
+               if (!access_ok(VERIFY_READ, base, len))
                        break;
                /*

Now apply patch using patch command, enter:
# patch < fix.vmsplice.exploit.patch -p1
Now recompile and install Linux kernel.

I hope this quick and dirty guide will save someones time. On a related note Erek has unofficial patched RPMs for CentOS / RHEL distros.

When you work in tech support department and deal with inexperienced clients debugging problems turns into a nightmare. As a sysadmin, you won't become too paranoid if less experienced people have root-access. As a consultant, you won’t feel isolated if you don’t have remote access to your systems. As a support engineer, you won’t become frustrated if a customer has fiddled around with some important config file and you have to find which. As a performance tuner, you can capture the state of the system configuration in between performance tests/benchmarks.

Luckily, some nifty tools can create a system's hardware and software configuration snapshot. This kind of information is valuable asset while troubleshooting problems.

dconf (System config collector) is one of such tool. It allows to take your system configuration with you on the road, compare identical systems (like nodes in a cluster) to troubleshoot HW or SW problems, indeed a lifesaver.

Dconf is also useful in projects where you have to manage changes as a team. Dconf can run periodically and send out system changes to a list of email addresses so that they can be revised and discussed in group.

You can customize your dconf configuration for specific needs, like making a profile of your web server’s hardware or copy specific software configuration files to send out or compare with other systems.

As a sysadmin, you will not become too paranoid if less experienced people have root-access. As a consultant, you will not feel isolated if you do not have remote access to your systems. As a support engineer, you will not become frustrated if a customer has fiddled around with some important config file and you have to find which. As a performance tuner, you can capture the state of the system configuration in between performance tests/benchmarks.

Install dconf

If you are using Debian / Ubuntu Linux then type the command:
# apt-get install dconf
You can download Dconf for RedHat or Suse Linux here

Create a system's hardware and software configuration snapshot

Once installed you can simply create a snapshot using dconf command:
# dconf
It will write snapshot in /var/log/dconf/ directory. To view current snapshot info, enter:
# zcat /var/log/dconf/dconf-$HOSTNAME-latest.log.gz
To check the latest changes against the previous snapshot:
# zdiff -u /var/log/dconf/dconf-$HOSTNAME-previous.log.gz /var/log/dconf/dconf-$HOSTNAME-latest.log.gz

See also: