My 10 Linux and UNIX Command Line Mistakes

      Anyone who has never made a mistake has never tried anything new. — Albert Einstein.
Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX sysadmin. This page lists my top ten Linux or Unix command line mistakes.

A list of my 10 UNIX command line mistakes


They say, “Failure is the key to success; each mistake teaches us something.” I hope you will learn something from my 10 Linux or Unix command line mistakes as well as the comments posted below by my readers.

userdel Command

The file /etc/deluser.conf was configured to remove the home directory (it was done by previous sys admin and it was my first day at work) and mail spool of the user to be removed. I just wanted to remove the user account and I end up deleting everything (note -r was activated via deluser.conf):
userdel foo

Rebooted Solaris Box

On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-name

Destroyed named.conf

I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.conf

Destroyed Working Backups with Tar and Rsync (personal backups)

I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk
tar -zcvf project.tar.gz functions

I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I have switched to rsnapshot)
rsync -av -delete /dest /src
Again, I had no backup.

Deleted Apache DocumentRoot

I had sym links for my web server docroot (/home/httpd/http was symlinked to /www). I forgot about symlink issue. To save disk space, I ran rm -rf on http directory. Luckily, I had full working backup set.

Accidentally Changed Hostname and Triggered False Alarm

Accidentally changed the current hostname (I wanted to see current hostname settings) for one of our cluster node. Within minutes I received an alert message on both mobile and email.
hostname foo.example.com

Public Network Interface Shutdown

I wanted to shutdown VPN interface eth0, but ended up shutting down eth1 while I was logged in via SSH:
ifconfig eth1 down

Firewall Lockdown

I made changes to sshd_config and changed the ssh port number from 22 to 1022, but failed to update firewall rules. After a quick kernel upgrade, I had rebooted the box. I had to call remote data center tech to reset firewall settings. (now I use firewall reset script to avoid lockdowns).

Typing UNIX Commands on Wrong Box

I wanted to shutdown my local Fedora desktop system, but I issued halt on remote server (I was logged into remote box via SSH):
halt
service httpd stop

Wrong CNAME DNS Entry

Created a wrong DNS CNAME entry in example.com zone file. The end result – a few visitors went to /dev/null:
echo 'foo 86400 IN CNAME lb0.example.com' >> example.com && rndc reload

Failed To Update Postfix RBL Configuration

In 2006 ORDB went out of operation. But, I failed to update my Postfix RBL settings. One day ORDB was re-activated and it was returning every IP address queried as being on its blacklist. The end result was a disaster.

Conclusion

All men make mistakes, but only wise men learn from their mistakes — Winston Churchill.
From all those mistakes I have learn that:

  1. You must keep a good set of backups. Test your backups regularly too.
  2. The clear choice for preserving all data of UNIX file systems is dump, which is only tool that guaranties recovery under all conditions. (see Torture-testing Backup and Archive Programs paper).
  3. Never use rsync with single backup directory. Create a snapshots using rsync or rsnapshots.
  4. Use CVS/git to store configuration files.
  5. Wait and read command line twice before hitting the dam [Enter] key.
  6. Use your well tested perl / shell scripts and open source configuration management software such as puppet, Ansible, Cfengine or Chef to configure all servers. This also applies to day today jobs such as creating the users and more.

Mistakes are the inevitable, so have you made any mistakes that have caused some sort of downtime? Please add them into the comments section below.

🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
692 comments… add one
  • Httqm Aug 3, 2017 @ 8:55

    working on a pre-production box and having to repeatedly move files around, so I ran :
    mv $d/* .
    whereas I meant :
    mv $D/* .
    $D was like “/a/very/long/path/with/projectcode_XYZ_123-456.090/YYYYMMDD/version_01.02.00/config/sOmEsTuFf”
    Which was interpreted as :
    mv /* .

    But I’ve been able to fix it without rebooting and without restoring a backup (so very few people noticed it 😉
    Full story + solution : http://doc.callmematthi.eu/Linux.html#accidentMoveSlashStar

  • BeeJay Aug 2, 2017 @ 23:37

    I was once in a Data Center where each rack containing Sun servers had a green-screen terminal connected via a serial console server to each Sun server in that rack. Every terminal had a keyboard with a layout that had the “Esc” key in the top left location… except for 1 terminal which the top left key was “Break”. I was doing work on the console of the production internet proxy server which was on the terminal that had the break key in the top left corner. Now I’ve been using the vi editor to edit files for years, and my natural habit was to hit the “esc” key after typing to exit “insert” mode. While on this particular terminal editing a file, I “automatically” hit the “esc” key, unfortunately it wasn’t “esc’ but “break” and for those old Sun servers, pressing the “Break” key on the console would stop the kernel running and drop you to the “ok” prompt from the boot prom. Usually you can just type “go” and the kernel will then continue running from where it left off, so I typed “go” but instead of just resuming, the system did a kernel panic. Needless to say, after this incident, all sun servers had their key switch turned to the “lock” position which prevents the break key from stopping the kernel.

  • Solbu May 31, 2017 @ 11:59

    My biggest blunder was way back when MS-DOS still was huge, all thou W95 had come along.

    I needed to copy something onto a floppy, but needed to make sure it was empty, so I checked to see if it was something on it that I needed before deleting the contents on the floppy.

    c:\> dir a:

    I didn’t need the content on the floppy, so I deleted it using this command:

    c:\> del *.*

    My computer was not usable for about a week after that, as I didn’t have the install disks at home.

    Fortunately it wasn’t any directories on the floppy, or I would have used deltree. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.