My 10 Linux and UNIX Command Line Mistakes

      Anyone who has never made a mistake has never tried anything new. — Albert Einstein.
Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX sysadmin. This page lists my top ten Linux or Unix command line mistakes.

A list of my 10 UNIX command line mistakes


They say, “Failure is the key to success; each mistake teaches us something.” I hope you will learn something from my 10 Linux or Unix command line mistakes as well as the comments posted below by my readers.

userdel Command

The file /etc/deluser.conf was configured to remove the home directory (it was done by previous sys admin and it was my first day at work) and mail spool of the user to be removed. I just wanted to remove the user account and I end up deleting everything (note -r was activated via deluser.conf):
userdel foo

Rebooted Solaris Box

On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-name

Destroyed named.conf

I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.conf

Destroyed Working Backups with Tar and Rsync (personal backups)

I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk
tar -zcvf project.tar.gz functions

I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I have switched to rsnapshot)
rsync -av -delete /dest /src
Again, I had no backup.

Deleted Apache DocumentRoot

I had sym links for my web server docroot (/home/httpd/http was symlinked to /www). I forgot about symlink issue. To save disk space, I ran rm -rf on http directory. Luckily, I had full working backup set.

Accidentally Changed Hostname and Triggered False Alarm

Accidentally changed the current hostname (I wanted to see current hostname settings) for one of our cluster node. Within minutes I received an alert message on both mobile and email.
hostname foo.example.com

Public Network Interface Shutdown

I wanted to shutdown VPN interface eth0, but ended up shutting down eth1 while I was logged in via SSH:
ifconfig eth1 down

Firewall Lockdown

I made changes to sshd_config and changed the ssh port number from 22 to 1022, but failed to update firewall rules. After a quick kernel upgrade, I had rebooted the box. I had to call remote data center tech to reset firewall settings. (now I use firewall reset script to avoid lockdowns).

Typing UNIX Commands on Wrong Box

I wanted to shutdown my local Fedora desktop system, but I issued halt on remote server (I was logged into remote box via SSH):
halt
service httpd stop

Wrong CNAME DNS Entry

Created a wrong DNS CNAME entry in example.com zone file. The end result – a few visitors went to /dev/null:
echo 'foo 86400 IN CNAME lb0.example.com' >> example.com && rndc reload

Failed To Update Postfix RBL Configuration

In 2006 ORDB went out of operation. But, I failed to update my Postfix RBL settings. One day ORDB was re-activated and it was returning every IP address queried as being on its blacklist. The end result was a disaster.

Conclusion

All men make mistakes, but only wise men learn from their mistakes — Winston Churchill.
From all those mistakes I have learn that:

  1. You must keep a good set of backups. Test your backups regularly too.
  2. The clear choice for preserving all data of UNIX file systems is dump, which is only tool that guaranties recovery under all conditions. (see Torture-testing Backup and Archive Programs paper).
  3. Never use rsync with single backup directory. Create a snapshots using rsync or rsnapshots.
  4. Use CVS/git to store configuration files.
  5. Wait and read command line twice before hitting the dam [Enter] key.
  6. Use your well tested perl / shell scripts and open source configuration management software such as puppet, Ansible, Cfengine or Chef to configure all servers. This also applies to day today jobs such as creating the users and more.

Mistakes are the inevitable, so have you made any mistakes that have caused some sort of downtime? Please add them into the comments section below.

🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source & DevOps topics via:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
692 comments… add one
  • Frank May 19, 2017 @ 11:30

    A customer with a huge mail box called and asked me to delete all emails older than one year. Ran find /var/vmail/domain/mailbox -mtime -365 -exec rm -f {} \; *facepalm*

  • Rajneesh Gadge Feb 15, 2017 @ 14:32

    Very nice article. Its really a very good idea to learn a lesson from other’s mistakes.

    This one happened with me quiet a few times, when I’m connected to multiple ssh session of different hosts, I end up running a script or a bash command on a wrong host.

    To overcome this, I use multiple monitors and align sessions on different screens by cascading them vertically in the same order so I can recognise on which to do it. 🙂

  • Chris Smith Feb 5, 2017 @ 11:00

    I created a file named ~ once and tried to delete it with

    rm -rf ~

    I added the -rf out of habit, so after a few seconds of running and me wondering why it was taking so long to delete one file, I hit Ctrl-C, then realised what I’d done.

  • r001 Jan 23, 2017 @ 0:04

    cd /tmp/
    chmod 755 -R .* – ‘.*’ matches ‘..’, so I changed the permissions on the entire filesystem. 🙂

  • Semi Jan 16, 2017 @ 10:43

    My best command ever:
    nohup find / -name “*” -exec rm -f {} ;

    And logout right after that.

    Also my timing was so perfect. Day before Christmas eve. I was going on holiday and I was almost late from bus when I issued that command… After a second I did realized what did I wrote (It should had been with ~), I managed to get in and end it before everything was wiped. I have no memory how much data was deleted, but if I think that it really did not had time to do real damage.

  • Beverly Jan 13, 2017 @ 0:31

    “Uname -a “on a solaris box running financial Oracle databases.

  • Matt Ryan Nov 18, 2016 @ 15:30

    As root, I wanted to remove all dot-files in a certain home directory. Some of them were dot-directories so I made it recursive: rm -rf .*

    I failed to anticipate that .* matches .., which references the directory above. So, the rm command proceeded to recursively remove every home directory on the machine.

    A safer way of doing this would have been: rm -rf .??*

  • Raymond Henick Nov 15, 2016 @ 12:15

    one mistake made at a major financial corp I worked at in the past was a junior admin cd into a directory to issue rm -rf only to realize he needed root. sudo -s su – drops you NOT on the directory you were in if you cd into it before the sudo. yeah, bad day.

  • Doychin Nov 14, 2016 @ 19:44

    I used to have FreeBSD server with apache, mysql, php, bind, mail server, fully working IPv6 and a lot of other things. Spent 3 years to write down myself a web panel like cPanel. Had more than 60 users with their stuff on it. One night I go out, had some drinks, go back home, login as root and wanted to “rm -rf /test-script” but I forgot to press the damn Tab. So everything ended with “rm -rf /”. All was gone. No backups, no nothing…

  • Bleepers Nov 8, 2016 @ 15:37

    Do something like..

    vim file.py l chmod +x file.py
    Instead of
    vim file.py ; chmod +x file.py

  • just another admin Aug 5, 2016 @ 18:11

    made many. funniest was:

    last | reboot

  • david Jul 31, 2016 @ 3:08

    I have done a few terrible things over the years. The bad part is I do not remember them all well enough to recite details. But I think I have nevertheless learned something: slow down, pay attention.

    One thing I remember was with the mysql command line, making a catastrophic typo and trashing thousands of rows. This was on the master of a master/save configuration — so slave replication was not my friend in this scenario. Stupid, slavish slave!

  • Len Jul 6, 2016 @ 19:02

    Following a security audit, I was asked to remove an account (on a test Solaris database server) that had been created by the auditors as part of the audit. Instead of running userdel, I just used the (Solaris 8?) CDE user account tool, checked the “remove home directory” box and clicked OK. After it ran for minutes and minutes and minutes and minutes, I killed it and tried to ‘ls’… lots of library errors and nothing worked. It was then I remembered that the auditor account was ID 0 (root equivalent) and Solaris by default makes root’s home directory the root filesystem. Ick. The DBA was gracious and said her test server really needed rebuilding anyway. 🙂 It still stung, though…

  • Lee Jun 30, 2016 @ 16:41

    Good clean fun!

    I don’t think that UNIX still allows this to happen thankfully. This was back in the mid-90s when I first started working on HP-UX and needed a perl script to kill off lots of idle / disconnected user processes. I found that killing off the parent process was the best way to handle it. Somehow the code to find the parent found a disconnected process that had been adopted by init (ie, PPID=1) and it then dutifully killed process 1. I literally had just enough time to see the screen print out what I had done via the script, exclaim something to express my alarm such as “Oh s**t!”, then watch the console for our production server go into a halt sequence. So i don’t think it can happen these days, but don’t try it on a prod box if you decide to test that. These days I don’t experiment on prod systems though!

    I’ve also learned the hard way to make a command I’m building from copying and pasting options to be non-executable: go to the beginning of the command line and type in “echo ” or “# ” to force a command to be just printed instead of executed That way if the pasted text has an embedded newline it won’t trigger execution of the actual command, when I’m ready I just make the command executable and bang it.

  • David Jun 13, 2016 @ 22:21

    I just did a hostname -n. Today meant to do a uname -n in my mind,

  • Geoffrey Jun 3, 2016 @ 12:31

    I was working with a Porteus Linux system running from a USB drive. The way Porteus works is kind of funny. A live filesystem is created at boot and optionally everything can be loaded into RAM. I forgot that I was in the directory that the system boots from (on the flashdrive) and ran rm -rvf *. Luckily my user data was saved in a different location. Keeping mind that I’m only twelve, though.

  • Tom May 1, 2016 @ 21:35

    My painful one was:
    rm -rf * instead of rm -rf *.o
    Lost a lot of c code that day.

  • Marcin Apr 12, 2016 @ 8:54

    Copied something to Documents folder for backup
    sudo cp /path/to/something/ /home/user/Documents/

    Then cleared the original folder with the same command with cp being rm this time…
    sudo rm -rf /path/to/something/ /home/user/Documents/

    Poof! All documents gone. Fortunately UbuntuOne was still alive, and I got most of the data back.

  • Euhill Apr 3, 2016 @ 6:05

    That all pales in comparison to what I once did when I was new to Linux. As root I typed rm *.*. Bye bye OS. Ended up having to reinstall. Never did that again.

  • Anon Mar 23, 2016 @ 3:12

    As root:
    # kill -9 -1

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.