# My 10 UNIX Command Line Mistakes

by on June 21, 2009 · 563 comments· Last updated June 23, 2009

Anyone who has never made a mistake has never tried anything new. -- Albert Einstein.

Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX admin.

## userdel Command

The file /etc/deluser.conf was configured to remove the home directory (it was done by previous sys admin and it was my first day at work) and mail spool of the user to be removed. I just wanted to remove the user account and I end up deleting everything (note -r was activated via deluser.conf):
userdel foo

## Rebooted Solaris Box

On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-name

## Destroyed named.conf

I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.conf

## Destroyed Working Backups with Tar and Rsync (personal backups)

I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk tar -zcvf project.tar.gz functions
I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I’ve switched to rsnapshot)
rsync -av -delete /dest /src

## Deleted Apache DocumentRoot

I had sym links for my web server docroot (/home/httpd/http was symlinked to /www). I forgot about symlink issue. To save disk space, I ran rm -rf on http directory. Luckily, I had full working backup set.

## Accidentally Changed Hostname and Triggered False Alarm

Accidentally changed the current hostname (I wanted to see current hostname settings) for one of our cluster node. Within minutes I received an alert message on both mobile and email.
hostname foo.example.com

## Public Network Interface Shutdown

I wanted to shutdown VPN interface eth0, but ended up shutting down eth1 while I was logged in via SSH:
ifconfig eth1 down

## Firewall Lockdown

I made changes to sshd_config and changed the ssh port number from 22 to 1022, but failed to update firewall rules. After a quick kernel upgrade, I had rebooted the box. I had to call remote data center tech to reset firewall settings. (now I use firewall reset script to avoid lockdowns).

## Typing UNIX Commands on Wrong Box

I wanted to shutdown my local Fedora desktop system, but I issued halt on remote server (I was logged into remote box via SSH):
halt service httpd stop

## Wrong CNAME DNS Entry

Created a wrong DNS CNAME entry in example.com zone file. The end result - a few visitors went to /dev/null:
echo 'foo 86400 IN CNAME lb0.example.com' >> example.com && rndc reload

## Failed To Update Postfix RBL Configuration

In 2006 ORDB went out of operation. But, I failed to update my Postfix RBL settings. One day ORDB was re-activated and it was returning every IP address queried as being on its blacklist. The end result was a disaster.

## Conclusion

All men make mistakes, but only wise men learn from their mistakes -- Winston Churchill.

From all those mistakes I’ve learnt that:

1. Backup = ( Full + Removable tapes (or media) + Offline + Offsite + Tested )
2. The clear choice for preserving all data of UNIX file systems is dump, which is only tool that guaranties recovery under all conditions. (see Torture-testing Backup and Archive Programs paper).
3. Never use rsync with single backup directory. Create a snapshots using rsync or rsnapshots.
4. Use CVS to store configuration files.
5. Wait and read command line again before hitting the dam [Enter] key.
6. Use your well tested perl / shell scripts and open source configuration management software such as puppet, Cfengine or Chef to configure all servers. This also applies to day today jobs such as creating the users and so on.

Mistakes are the inevitable, so did you made any mistakes that have caused some sort of downtime? Please add them into the comments below.

You should follow me on twitter here or grab rss feed to keep track of new changes.

#### Featured Articles:

1 Jon June 21, 2009 at 2:42 am

My all time favorite mistake was a simple extra space:
 cd /usr/lib ls /tmp/foo/bar
I typed
rm -rf /tmp/foo/bar/ *
rm -rf /tmp/foo/bar/*
The system doesn’t run very will without all of it’s libraries……

2 Vinicius August 21, 2010 at 5:42 pm

I Did something similar on a remote server
I was going to type ‘chmod -R 755 ./’ but i throw ‘chmod -R 755 /’ |:

3 James January 6, 2011 at 11:01 am

Yes – I think I’ve made almost every possible linux mistake over the years – when I was a young sys admin I did exactly what you did – and put a space in the middle of a rm -rf /stuff/to/delete/ *
I think now that the best thing is to use virtual machines, and backup those VM’s locally and remotely. It’s easy to restart a VM & roll back changes if needed.

4 Michael Shigorin January 9, 2011 at 5:03 pm

* VM is no silver bullet (and definitely no substitute for /dev/head and /proc/care);
* zsh can warn on this type of error, and its rm has additional -s option to handle “buried symlink” case;
* I’ve got a habit of hitting (at least when zsh/bash3 is handy) and examining the static list instead of removing by pattern.

5 Simon January 9, 2011 at 10:54 pm

Michael, I don’t mean to be judgemental or start a discussion, but the idea behind this comments section (at least to my understanding) is to share experience and non-obvious mistakes in order to keep others from making them, not to discuss general ideas on how to do things like backup, etc. So please share more experience rather than correcting mistakes others have made.
Cheers, Simon

6 Michael Shigorin January 10, 2011 at 6:27 pm

Yeah, see below — sorta slow mood with a “feel that” attitude… dropping/hiding extra comments by me might benefit the page.

Thanks Simon!

7 robert wlaschin May 1, 2012 at 9:57 pm

Hm… I was trying to format a USB flash

dd if=big_null_file of=/dev/sdb

unfortunately /dev/sdb was my local secondary drive, sdc was the usb … shucks.

I discovered this after I rebooted.

8 Alex February 8, 2013 at 2:04 pm

Did the same thing, Found out after 20 minutes or so, by then most of the important data was gone…

9 Ken Kam February 27, 2011 at 3:44 pm

I did this as well just the other day. I thought it took a while to delete a folder, until I realised there was a space in between =/.

10 Jeff April 21, 2011 at 10:46 pm

I did something similar on my first day as a junior admin. As root, I copied my buddy’s dot files (.profile, etc.) from his home directory to mine because he had some cool customizations. He also had some scripts in a directory called .scripts/ that he wanted me to copy. I gave myself ownership of the dot files and the contents of the .scripts directory with this command:

cd ~jeff; chown -R jeff .*

It was only later that I realized that “.*” matched “.” and “..”, so my userid owned the entire machine… which happened to be our production Oracle database.

That was 15 years ago and we’ve both changed jobs a few times, but that friend reminds me of that mistake every time I see him.

11 Alex July 8, 2011 at 9:40 am

Or yeah :) This mistake is definitely in TOP10. I’ve done it too.

12 Sam Watkins July 12, 2011 at 3:30 pm

For most of these errors above that occured in the workplace, perhaps the biggest mistake was that a senior admin or manager allowed some junior who does not know the difference between / and \ to type at a # root prompt on a valuable production server. I would not so much blame the junior, but I would suggest that the (ir)responsible senior should be fired! If my 3 year old son strangles the cat on my watch, I’m responsible!

13 Tom April 8, 2012 at 5:43 pm

Recent issue.
We’ve switched all servers PSU1 to backup power PSU2 since all servers have redundant power units to replace main UPS with higher model. However, SAN switches do not have redundant PSU. So, We’re observed how LUN paths are switched from VMWare vSphere Client, and they were up after switching power one by one.
However, storage for main Oracle DB box didn’t come back because of Windows driver failure..
Lesson learned: ALWAYS check are all LUNs back on-line, for Windows and Linux separately.

14 nels June 21, 2009 at 7:32 am

rm -rf /tmp
I didn’t think it would hose me too much, but on a F10 box it was important……

15 Ed May 28, 2011 at 3:30 pm

how about rm -rf * whilst in /etc?

Thought I was a subdirectory of my home, forgot the cd I’d issued a few minutes before. Fortunately an ancient error.

16 Crash July 19, 2011 at 3:39 pm

Been there, done that. Pain in the butt!

17 Jose ruiz August 22, 2011 at 9:36 pm

The rm -rf is the most common for Linux beginners that ends up dooming people. I ran into a problem where I needed to make sure the date on the servers was at least 30 seconds apart from each server in a oracle database network. I forgot to put a “.” at the end to represent the seconds so the next day my servers had a date where the year was 2048, even now my co-workers still call me lightspeed.

18 Cody January 16, 2012 at 9:15 pm

Sounds like my mom’s office (telecommunication). They’re relentless. That’s a pretty funny story (and thing to call you; sure it’s at your expense but it’s not really that mean, and its more clever).

I think my favourite story is something that your (Jose’s) story reminds me of. My mom fixes databases and other problems. One time there was a database issue she was debugging. She was talking to the person on the phone. The person kept reading the error he was getting to my mom (over phone). Finally, my mom realized that he had actually printed out the file right – the file is what contained an error – not the printer itself! In other words, he printed out a file that contained the error and he thought that it was the printer having an issue. But hey, he learned from it and we all make mistakes at times. Those who say you should just fire them don’t realize that if that were the case they’d have no employees (and potentially law suits – yes, I’m serious. Some firings over things that may seem harmless can lead to lawsuits, whether its not a legit/unfounded case or not).

19 firmit June 21, 2009 at 9:00 am

Some classics here :)

20 georgesdev June 21, 2009 at 9:15 am

never type anything such as:
rm -rf /usr/tmp/whatever
maybe you are going to type enter by mistake before the end of the line. You would then for example erase all your disk starting on /.

if you want to use -rf option, add it at the end on the line:
rm /usr/tmp/whatever -rf

21 Leon July 29, 2010 at 2:43 pm

That was a good one!! :)

22 Denis November 23, 2010 at 9:27 am

I think it is a good practice to use parameter i whithin the -rf:
rm -rfi /usr/tmp/whatever
-i will ask you do you sure to delete all that stuff.

23 John February 25, 2011 at 11:11 am

I worked with a guy who always used “rm -rf” to delete anything. And he always logged in as root. Another worker set the stage for him by creating a file called “~” in a visible location (that would be a filed entered as “\~”, as not to expand to the user’s home directory. User one then dealt with that file with “rm -rf ~”. This was when the root home directory was / and not something like /root. You got it.

24 S February 28, 2011 at 5:32 am

ouch !!

25 Cody March 22, 2011 at 1:33 pm

(Note to mod: put this in wrong place initially; sorry about that. here is the correct place).

This reminds me of when I told a friend a way to auto-log out on login (many ways but this would be more obscure). He then told someone who was “annoying” him to try it on his shell. End result was this person was furious. Quite so. And although I don’t find it so funny now (keyword not as – I still think it’s amusing), I found it hilarious then (hey, was young and obnoxious as can be!).

The command, for what its worth :

echo “PS1=kill -9 0” >> ~/.bash_profile

Yes, that’s setting the prompt to run the command : kill -9 0 upon sourcing of ~/.bash_profile which means kill that shell. Bad idea!

I don’t even remember what inspired me to think of that command as this was years and years ago. However, it does bring up an important point :

Word of the wise : if you do not know what a command does, don’t run it! Amazing how many fail that one…

26 mike March 29, 2011 at 11:35 am

I altered /boot/grub/rm wrong .iso
reboot

27 mike March 29, 2011 at 11:47 am

and if you look at that comment I even managed to mess up
/boot/rm wrong.iso
not /boot/grub……..

28 Peter Odding January 7, 2012 at 6:40 pm

I once read a nasty trick that’s also fun in a very sadistic kind of way:

echo ‘echo sleep 1 >> ~/.profile’ >> /home/unhappy-user/.profile

The idea is that every time the user logs in it will take a second longer than the previous time… This stacks up quickly and gets reallllly annoying :-)

29 Daniel Hoherd May 4, 2012 at 4:58 pm

Another good test is to first do “echo rm -rf /dir/whatever/*” to see the expansion of the glob and what will be deleted. I especially do this when writing loops, then just pipe to bash when I know I’ve got it right.

30 3ToKoJ June 21, 2009 at 9:26 am

public network interface shutdown … done
typing unix command on wrong box … done
Delete apache DocumentRoot … done
Firewall lockdone … done with a NAT rule redirecting the configuration interface of the firewall to another box, serial connection saved me

I can add, being trapped by aptitude keeping tracks of previously planned — but not executed — actions, like “remove slapd from the master directory server”

31 UnixEagle June 21, 2009 at 11:03 am

Rebooted the wrong box
While adding alias to main network interface I ended up changing the main IP address, the system froze right away and I had to call for a reboot
Instead of appending text to Apache config file, I overwritten it’s contents
Firewall lockdown while changing the ssh port
Wrongfully run a script contained recursive chmod and chown as root on / caused me a downtime of about 12 hours and a complete re-install

Some mistakes are really silly, and when they happen, you don’t believe your self you did that, but every mistake, regardless of it’s silliness, should be a learned lesson.
If you did a trivial mistake, you should not just overlook it, you have to think of the reasons that made you did it, like: you didn’t have much sleep or your mind was confused about personal life or …..etc.

I like Einstein’s quote, you really have to do mistakes to learn.

32 smaramba June 21, 2009 at 11:31 am

typing unix command on wrong box and firewall lockdown are all time classics: been there, done that.
but for me the absolute worst, on linux, was checking a mounted filesystem on a production server…

fsck /dev/sda2

the root filesystem was rendered unreadable. system down. dead. users really pissed off.
fortunately there was a full backup and the machine rebooted within an hour.

33 Don May 10, 2011 at 4:14 pm

I know this thread is a couple of years old but …

Using lpr from the command line, forgetting that I was logged in to a remote machine in another state. My print job contained sensitive information which was now on a printer several hundred miles away! Fortunately, a friend intercepted the message and emailed me while I was trying to figure out what was wrong with my printer :-)

34 od June 21, 2009 at 12:50 pm

“Typing UNIX Commands on Wrong Box”

Yea, I did that one too. Wanted to shut down my own vm but I issued init 0 on a remote server which I accessed via ssh. And oh yes, it was the production server.

35 Adi June 21, 2009 at 10:24 pm

tar -czvf /path/to/file file_archive.tgz
tar -czvf file_archive.tgz /path/to/file
I ended up destroying that file and had no backup as this command was intended to provide the first backup – it was on the DHCP Linux production server and the file wad dhcpd.conf!

36 ikke July 15, 2010 at 3:33 pm

same with
tar -czvf ${pattern}* instead of tar -czvf${target} ${pattern}* PLUS no backup EQUALS :( Reply 37 Mel August 5, 2010 at 4:54 pm yeah i did this too, luckily only to the maillog but still! lots of (*&$(£*$&£(!!! Reply 38 Addy August 2, 2011 at 10:43 am me too, but with the basic cp cp destination source instead of cp source destination Reply 39 sims June 22, 2009 at 2:23 am Funny thing, I don’t remember typing typing in the wrong console. I think that’s because I usually have the hostname right there. Fortunately, I don’t do the same things over and over again very much. Which means I don’t remember command syntax for all but most used commands. Locking myself out while configuring the firewall – done – more than once. It wasn’t really a CLI mistake though. Just being a n00b. georgesdev, good one. I usually: ls -a /path/to/files to double check the contents then up arrowkey homekey hit del a few times and type rm. I always get nervous with rm sitting at the prompt. I’ll have to remember that -rf at the end of the line. I always make mistakes making links. I can never remember the syntax. :/ Here’s to less CLI mistakes… (beer) Reply 40 Grant D. Vallance June 22, 2009 at 7:56 am A couple of days ago I typed and executed (as root): rm -rf /* on my home development server. Not good. Thankfully, the server at the time had nothing important on it, which is why I had no backups … I am still not sure *why* I did when I have read about all the warnings about using this command. (A dyslexic moment with the syntax?) Ah well, a good lesson learned. At least it was not the disaster it could of been. I shall be *very* paranoid about this command in the future. Reply 41 wtf dude April 8, 2013 at 6:01 pm wtf dude, this command just doesn’t make ANY sense … why did u exeute it … Reply 42 Joren June 22, 2009 at 9:30 am I wanted to remove the subfolder etc from the /usr/local/matlab/ directory. So I accidentally added the ‘/’ symbol in a force of habit when going to the /etc folder and I typed from the /usr/local/matlab directory:  sudo rm /etc instead of  sudo rm etc Without the entire /etc folder the computer didn't work anymore (which was to be expected ofcourse) and I ended up reinstalling my computer. Reply 43 Michael Shigorin January 9, 2011 at 5:20 pm A collague got a habit of explicitly spelling relative paths, even if he’s not to execute off-$PATH executable — like:
cp ./a ./b
rm ./c

(btw it’s not worth getting used to cp/mv/rm -i — if you like it, put it in explicitly as well)

44 Ramaswamy June 22, 2009 at 10:47 am

Deleted the files
I used place some files in /tmp/rama and some conf files at /home//httpd/conf file
I used to swap between these two directories by “cd -”
Executed the command rm -fr ./*
supposed to remove the files at /tmp/rama/*, but ended up by removing the file at /home//httpd/conf/*, with out any backup

conclusion: check the directory where the rm command removing files of which directory

45 Robsteranium June 22, 2009 at 11:05 am

Aza Rashkin explains how habituation can lead to stupid errors – confirming “yes I’m sure/ overwrite file etc” automatically without realising it. Perhaps rm and the > command need an undo/ built-in backup…

46 Ulver June 22, 2009 at 2:06 pm

rm -rf /dev/{null,zero} $HOME/dev/ instead of cp -rf /dev/{null,zero}$HOME/dev/{null,zero}

…live cd and mknod savemes

echo > somefileWithManyCodeLines.pl

echo >> somefileWithManyCodeLines.pl

the x-term history saves me !! (preveusly i dump the content of the file using cat)

so doesn’t care what term uses…set the history at the max.

and…. my prefered (many years ago happens me ….could be an excelent way to …really delete & hide information)

dd if=/dev/urandom of=/dev/withManyData count=1024 bs=1024

dd if=/dev/zero of=/dev/withManyData count=1 bs=512

the last one i callit “dd unleashed” ….

47 Greywolf February 9, 2012 at 9:18 pm

On a SunOS/Solaris box, or anything with dynamically linked critical programs [cp, cat, tar, sh], if /dev/zero or /usr/lib/crt0.so vanishes, you’re screwed. /dev/zero is especially insidious, because, in their infinite wisdom, they decided to make mknod a dynamically linked executable, so you can’t even mknod it.

48 Ulver June 22, 2009 at 2:08 pm

i forget it….i the last command i haven’t way to rollback only reinstalling

49 Andrea Ratto June 22, 2009 at 5:36 pm

I did a rm -r /usr on my two-years-old gentoo installation. Ctrl-C after one second, most of /usr/bin removed. Running ubuntu since then.

50 RudyD June 22, 2009 at 11:32 pm

Sure there were.
First thing to take an ocassional backup before any modifications.
And be careful with tnf testing of scripts with redirections to important commands with absolute path to the binary. In case. somehow I have found the mail binary in a same version server.

Other nice thing to remember. Do wait for password prompt. Especially when someone around you.
Sometimes there are a few seconds to wait and not to type clean readable characters to the console.

51 Nate June 23, 2009 at 12:16 am

My worst mistake was when I started using Ubuntu and changing abruptly (but willingly) from Windows to Linux. I accidentally deleted the entire filesystem with a command. No backups but it was a clean install.

52 Yonitg June 23, 2009 at 8:06 am

Great post !
I did my share of system mishaps,
killing servers in production, etc.
the most emberassing one was sending 70K users the wrong message.
or beter yet, telling the CEO we have a major crysis, gathering up many people to solve it, and finding that it is nothing at all while all the management is standing in my cube.

53 Serge June 23, 2009 at 3:45 pm

or chown all the hidden file in a website directory by ” chown user.group .* ” end up changing owner on all the websites .. (one directory up)

one should use :
chown user.group .??*

good job ;-)

54 rafael March 2, 2012 at 7:45 pm

I did it too, but to /home/user/.. At the end, I’ve changed all ownership of /home subdirectories

:ó(

55 Derrick June 23, 2009 at 4:10 pm

While adding a user, I attempted to specify the home directory. The directory did not exist yet, so I used the “m” option to create the directory as the user was being created.

But, I left off the name of the home directory that I wanted to create when I typed:

# useradd -s /bin/false -d /export/home -m new_user

I designated the home directory of the new user as /export/home instead of /export/home/new_user.

I ended up wiping out all the other users’ home directories on that server that were also located in /export/home!

I broke out in a cold sweat.

Thank goodness I was able to replace those directories from a backup.

*Whew*

56 Mat Enders April 17, 2011 at 10:33 am

I did something very similar. I was creating all of the user accounts on the new samba domain controller that was going into production the next day. Everything was done and configured except creating the users. when creating the last user instead specifying her home directory as /home/staff/9807mr I designated it as /home/staff there by putting only way to fix was reinstall had all of the configuration files backed up but was still up all night recreating all the user accounts.

57 Solaris June 23, 2009 at 8:37 pm

Firewall lock out: done.
Command on wrong server: done.

And the worst: update and upgrade while some important applications were running, of
course on a production server.. as someone mentioned the system doesn’t run very well
without all of its original libraries :)

58 niskotink0 June 24, 2009 at 11:14 am

hi.. )
My biggest error, with IPFW on FreeBSD system..
i’m turn on it in kernel:
options IPFIREWALL
and forget ..
options IPFIREWALL_DEFAULT_TO_ACCEPT
reboot.. and ..ups )))

59 Valqk June 26, 2009 at 3:51 pm

This is how I protect myself from making wrong halt:

in short I use molly-guard

60 Valqk June 26, 2009 at 4:03 pm

Rebooted/halted wrong severs – done. (posted link how to protect from this on linux)
stop wrong interface, firewall lockup – done.
some fun examples.
wanted to delete all hidden files in users home.
rm -fr .* :-D guess if that match . ..? :)
and the nasties thing I did recently is to run reiserfschk on a lvm device! ROLF! Thanks got it was on a testing setup server… neither reiser checked ok neither lvm worked after that. :)

61 ddn June 26, 2009 at 10:18 pm

Have done /sbin/init 1 instead of /sbin/init q multiple times.
after siwtching to MS Natural Keyboard.

62 Alex July 8, 2011 at 10:10 am

I’ve got this “init 1″ too :) It was core router of our home network.

63 Vivek Gite June 27, 2009 at 2:47 am

@everyone:

Thanks for sharing your experience with us.

64 Peko June 30, 2009 at 8:46 am

I invented a new one today.

Just assuming that a [-v] option stands for –verbose

Yep, most of the time. But not on a [pkill] command.
[pkill -v myprocess] will kill _any_ process you can kill — except those whose name contains “myprocess”. Ooooops. :-!
(I just wanted pkill to display “verbose” information when killing processes)

Yes, I know. Pretty dumb thing. Lesson learned ?

” Read The Fantastic Manual — First” ;-)

65 Maroo July 1, 2009 at 4:46 pm

I issued the following command on a BackOffice Trading Box in an attempt to clean out a user’s directory. But issued it in the /local. The command ended up taking out the Application mounted SAN directory and the /local directory.

find . -name “foo.log*” -exec ls -l {} \;|cut -f2 -d “/”|while read NAME; do gzip -c $NAME >$NAME.gz; rm -r $NAME; done Took out the server for an entire day. Reply 66 Jai Prakash July 3, 2009 at 1:43 pm Mistake 1: My Friend tried to see last reboot time and mistakenly executed command “last | reboot” instead of “last | grep reboot” It made a outage on Production DB server. Mistake 2: Another guy, wants to see the FQDN on solaris box and executed “hostname -f” It changed the hostname name to “-f” and clients faced lot of connectivity issues due to this mistake. [ hostname -f is used in Linux to see FQDN name but it solaris its usage is different ] Reply 67 Name July 4, 2009 at 5:20 pm Worse thing i’ve done so far, It accidentally dropped a MySQL database containing 13k accounts for a gameserver :D Luckily i had backups but took a little while to restore, Reply 68 Scott September 20, 2011 at 8:04 pm I’ve done something very similar - mysql> update tablename set field=value; OOPS! Should have been: mysql> update tablename set field=value where anotherfield=anothervalue; Reply 69 Vince Stevenson July 6, 2009 at 6:23 pm I was dragged into a meeting one day and forgot to secure my Solaris session. A colleague and former friend did this: alias ll=’/usr/sbin/shutdown -g5 -i5 “Bye bye Vince”‘ He must have thought that I was logged into my personal host machine, not the company’s cashcow server. What happens when it all goes wrong. Secure your session… Rgds Vince Reply 70 Bjarne Rasmussen July 7, 2009 at 7:56 pm well, tried many times, the crontab fast typing failure… crontab -r instead of -e e for edit r for remove.. now i always use -l for list before editing… Reply 71 Eob October 6, 2011 at 9:08 pm Yes, done that one. R and E are rather close to each other on the keyboard. In fact my name is really Rob! Reply 72 Ian July 8, 2009 at 4:15 am Made a script that automatically removes all files from a directory. Now, rather than making it logically (this was early on) I did it stupidly. cd /tmp/files rm ./* Of course, eventually someone removed /tmp/files.. Reply 73 shlomi July 12, 2009 at 9:21 am Hi On My RHEL 5 sever I create /tmp mount point to my storage and tmpwatch script that run under cron.daily removes files which have not been accessed 12 hours !!! Reply 74 Ville July 14, 2009 at 12:17 am I run a periodic (daily) script on a BSD system to clean out a temp directory for joe (the editor). Anything older than a day gets wiped out. For some historical reason the temp directory sits in /usr/joe-cache rather than in, for instance, /usr/local/joe-cache or /var/joe-cache or /tmp/joe-cache. The first version of the line in the script that does the deleting looked like this:  find /usr/joe-cache/.* -maxdepth 1 -mtime +1 -exec rm {} \;  Good thing the only files in /usr were two symlinks that were neither mission critical nor difficult to recreate as the above also matches “/usr/edit-cache/..” In the above the rather extraneous (joe doesn’t save backup files in sub-directories) “-maxdepth 1″ saved the entire /usr from being wiped out! The revised version:  find -E /usr/joe-cache/ -regex '/usr/joe-cache/\.?.+$' -maxdepth 1 -mtime +1 -exec rm {} \;
.. which matches files beginning with a dot within “/usr/joe-cache”, but won’t match “/usr/joe-cache/..”

Lesson learned: always test find statements with “-print” before adding “-exec rm {} \;”.

75 brownis July 18, 2009 at 6:47 am

sudo chmod 777 * -R
sudo chown user * -R
it is good but not if u r not located at the ~/

i had to reinstall linux, because of the permission mistake !

76 Shrikant July 22, 2009 at 2:42 pm

mv /* /tmp (or some where else).

77 Shantanu Oak July 24, 2009 at 6:31 am

I remember loosing the .tar files, and now I know why :)
I have never lost important files while using rm -rf
But I do remember copying / creating files at the root when I forgot to add the starting /

78 Shantanu Oak July 24, 2009 at 7:12 am

One more mistake that I do remember is copying a directory to another server but without using the recursive option. That copied the files found at the root but the files stored in the sub-folders were not copied.

79 M.S. Babaei August 1, 2009 at 3:39 am

once upon a time mkfs is killing me on ext3 partition I want
mkfs.ext3 /dev/sda1
I did this
mkfs.ext3 /dev/sdb1

I never forget what I lost??

80 Braindead August 1, 2009 at 5:57 am

I’ve most of these too :D

My favorite mistake was sitting around waiting for 250 or so gigs of stuff to copy to an nfs share, only to find I forgot to mount the remote share and it all just copied into the directory in /mnt instead. Good thing I had a huge root partition…

81 veeresh August 7, 2009 at 6:03 am

i have information abouttttt unix commands

82 Simon B August 7, 2009 at 2:47 pm

Whilst a colleague was away from their keyboard I entered :

 rm -rf *
… but did not press enter on the last line (as a joke). I expected them to come back and see it as a joke and rofl….back space… The unthinkable happened, the screen went to sleep and they banged the enter key to wake it up a couple of times. We lost 3 days worth of business and some new clients. estimated cost $50,000+ Reply 83 Steve March 18, 2011 at 7:02 am That seems a bit like cutting someone’s brake lines as a joke. The least you could have done is cd to their music directory or something. :P Reply 84 Susheel Jalali. September 5, 2011 at 7:38 am Fear of running an unintended command or unexpected behaviour, has always inculcated in me a habit to this day: I never wake up the screen with [ENTER], but always with the most harmless [SHIFT] key. Reply 85 RudyD November 2, 2011 at 10:52 pm Hi! Something similar here. I use the [Num Lock] or the [Ctrl] twice for this reason first… A was wandering that is there any harm to these versions than that the double [Ctrl] is mapped to the console switching on some KVMs? plus one: There were a few good times routinely push [Ctrl+Alt+Del] commands on a Virtual host console with plenty of Windows servers and a few linux ones. You can bet on this. Very good trick to draw attention… Reply 86 Julian November 30, 2011 at 8:23 pm During my first job in aix while saving a file with vim, it happens that sometimes you press another key after pressing w so the file gets saved with the new name. Usually i simply delete these files and nothing more happens. But this is a task i have automated in my mind (rm -rf file) i don’t know how my fingers reached the star key but once it happened that i saved the file as *. Imagine what happened after i finished working in the script and went back to shell to remove the file and my automated ‘rm -rf file’ stuff came to my mind …. my whole user directory deleted …. Reply 87 ginzero August 17, 2009 at 5:10 am tar cvf /dev/sda1 blah blah… Reply 88 Kevin August 25, 2009 at 10:50 am tar cvf my_dir/* dir.tar and your write your archive in the first file of the directory … Reply 89 ST September 17, 2009 at 10:14 am I’ve done the wrong server thing. SSH’d into the mailserver to archive some old messages and clear up space. Mistake #1: I didn’t logoff when I was done, but simply minimized the terminal and kept working Mistake#2: At the end of the day I opened what I thought was a local terminal and typed: /sbin/shutdown -h now thinking I was bringing down my laptop. The angry phone calls started less than a minute later. Thankfully, I just had to run to the server room and press power. I never thought about using CVS to backup config files. After doing some really dumb things to files in /etc (deleting, stupid edits, etc), I started creating a directory to hold original config files, and renaming those files things like httpd.conf.orig or httpd.conf.091709 As always, the best way to learn this operating system is to break it…however unintentionally. Reply 90 amanda January 7, 2011 at 5:58 pm Ohhh, I did this once in an LPIC certification class. I had my laptop running Ubuntu, but we all had an account on SUSE box the instructor wanted us to go through the class on, so I was logged into that as well via ssh. Two identical-looking terminal windows up… you can guess what I did. The worst part was that we had been working for nearly an hour and some people hadn’t saved their files… Reply 91 Wolf Halton September 21, 2009 at 3:16 pm Attempting to update a Fedora box over the wire from Fedora8 to Fedora9 I updated the repositories to the Fedora9 repos, and ran # yum -y upgrade I have now tested this on a couple of boxes and without exception the upgrades failed with many loose older-version packages and dozens of missing dependencies, as well as some fun circular dependencies which cannot be resolved. By the time it is done, eth0 is disabled and a reboot will not get to the kernel-choice stage. Oddly, this kind of update works great in Ubuntu. Reply 92 Ruben September 24, 2009 at 8:23 pm while cleaning the backup hdd late the night, a ‘/’ can change everything… “rm -fr /home” instead of “rm -fr home/” It was a sleepless night, but thanks to Carlo Wood and his ext3grep I rescued about 95% of data ;-) Reply 93 foo September 25, 2009 at 9:36 pm # svn add foo —-> Added 5 extra files that were not to be commited, so I decided to undo the change,delete the files and add to svn again….. # svn rm foo –force and it deleted the foo directory from disk :(…lost all my code just before the dead line :( Reply 94 Michael Shigorin January 9, 2011 at 6:06 pm > …lost all my code just before the dead line :( Learned from borked git rebase –interactive early on in a personal project repo: tar workdir up (or copy it, be it rsync, scp, cp, or mc, whatever), only then continue with whatever you’ve a tiniest bit of doubt with. Reply 95 Peter Odding January 7, 2012 at 9:47 pm I do the same thing when I’m afraid of breaking a source code repository. Depending on context, sometimes “cp -al important-files important-backup” is enough and it’s a lot faster than creating a tarball (that command creates a tree of hard links to the files in the original tree, so if you edit a file the change is visible in both directories, but if you accidentally delete some files from the original directory you can still reach them from the backup directory). Reply 96 foo September 25, 2009 at 9:41 pm wanted to kill all the instances of a service on HP-UX (pkill like util not available)… # ps -aef | grep -v foo | awk {print’$2′} | xargs kill -9

Typed “grep -v” instead of “grep -i” and u can guess what happened :(

97 LinAdmin September 29, 2009 at 2:38 pm

Typing rm -Rf /var/*  in the wrong box. Recovered in few minutes by doing  scp root@healty_box:/var .  – the ssh session on the just broken box was still open . This saved my life :-P

98 Deltaray October 3, 2009 at 4:37 am

Like Peko above, I too once ran pkill with the -v option and ended up killing everything else. This was on a very important enterprise production machine and I reminded myself the hard lesson of making sure you check man pages before trying some new option.

I understand where pkill gets its -v functionality from (pgrep and thus from grep), but honestly I don’t see what use of -v would be for pkill. When do you really need to say something like kill all processes except this one? Seems reckless. Maybe 1 in a million times you’d use it properly, but probably most of the time people just get burned by it. I wrote to the author of pkill about this but never heard anything back. Oh well.

99 Guntram October 5, 2009 at 7:51 pm

This is why i never use pkill; always use something like “ps ….| grep …” and, when it’s ok, type a ” | awk ‘{print $2}’ | xargs kill” behind it. But, as a normal user, something like “pkill -v bash” might make perfect sense if you’re sitting at the console (so you can’t just switch to a different window or something) and have a background program rapidly filling your screen. Worst thing that ever happened to me: Our oracle database runs some rdbms jobs at midnight to clean out very old rows from various tables, along the line of “delete from XXXX where last_access < sysdate-3650". One sunday i installed ntp to all machines, made a start script that does an ntpdate first, then runs ntpd. Tested it:$ date 010100002030; /etc/init.d/ntpd start; date
Worked great, current time was ok.
$date 010100002030; reboot After the machine was back up i noticed i had forgotten the /etc/rc*.d symlinks. But i never thought of the database until a lot of people were very angry monday morning. Fortunately, there's an automated backup every saturday. Reply 100 sqn October 7, 2009 at 6:05 pm tried to lockout a folder by removing it’s attributes (chmod 000) as a beginner and wanted to impress myself, did: # cd /folder # chmod 000 .. -R used two points instead of one, and of course the system used the upper folder witch is / for modifying attributes ended up getting out of my home and go the the server to reset the permissions back to normal. I got lucky because i just did a dd to move the system from one HDD to another and I haven’t deleted the old one yet :) And of course the classical configuring the wrong box, firewall lockout :) Reply 101 dev October 15, 2009 at 10:15 am while I was working on many ssh window: rm -rf * I intended to remove all files under a site, after changing the current working directory, then replacing with the stable one wrong window, wrong server, and I did it on production server xx(( just aware the mistakes 1.5 after typing [ENTER] no backup. maybe luckily, the site was keep running smooth.. it seems that the deleted files were such images, or media contents 1-2 secs incidental removal in heavy machine gave me loss approx. 20 MB Reply 102 LMatt October 17, 2009 at 3:36 pm In a hurry to get a db back up for a user, I had to parse through nearly a several terabyte .tar.gz for the correct SQL dumpfile. So, being the good sysadmin, I locate it within an hour, and in my hurry to get db up for the client who was on the phone the entire time: mysql > dbdump.sql Fortunately I didn’t sit and wait all that long before checking to make sure that the database size was increasing, and the client was on hold when I realized my error. mysql > dbdump.sql — SHOULD be — mysql < dbdump.sql I had just sent stdout of the mysql CLI interface to a file named dbdump.sql. I had to re-retrieve the damn sqldump file and start over! BAH! FOILED AGAIN! Reply 103 Mr Z October 18, 2009 at 5:13 am After 10+ years I’ve made a lot of mistakes. Early on I got myself in the habit of testing commands before using them. For instance: ls ~usr/tar/foo/bar then rm -f ~usr/tar/foo/bar – make sure you know what you will delete When working with SSH, always make sure what system you are on. Modifying system prompts generally eliminates all confusion there. It’s all just creating a habit of doing things safely… at least for me. Reply 104 Michael Shigorin January 9, 2011 at 6:14 pm > ls ~usr/tar/foo/bar then rm -f ~usr/tar/foo/bar Alt-. (or less repeatable, Esc .) might help as it’s “last argument history” at least in zsh/bash/ksh: ls -lh somethingreallylong rm -f [PAUSE] > When working with SSH, always make sure what system you are on. My favourite ~/.screenrc part: caption always “%{+b rk}%H%{gk} |%c %{yk}%d.%m.%Y | %72=Load: %l %{wk}” Reply 105 Michael Shigorin January 9, 2011 at 6:23 pm erm, the example should read (<brackets> didn’t make it): ls -lh somethingreallylong rm -f [Alt-.] [PAUSE] [Enter] Reply 106 chris October 22, 2009 at 11:15 pm cd /var/opt/sysadmin/etc rm -f /etc note the /etc. It was supposed to be rm -rf etc Reply 107 Jonix October 23, 2009 at 11:18 am The deadline were coming too close to comfort, I’d worked for too looong hours for months. We were developing a website, and I was in charge of developing the CGI scripts which generated a lot of temporary files, so on pure routine i worked in “/var/www/web/” and entered “rm temp/*” which i misspelled at some point as “rm tmp/ *”. I kind of wondered, in my overtired brain, what took so long for the delete to finish, it should only be 20 small files that is should delete. The very next morning the paying client was to fly in and pay us a visit, and get a demonstration of the project. P.S Thanks to Subversion and opened files in Emacs buffers I managed to get almost all files back, and I had rewritten the missing files before the morning. Reply 108 Cougar October 29, 2009 at 3:00 pm rm * in one of my project directory (no backup). I planned to do rm *~ to delete backup files but used international keyboard where space was required after ~ (dead key for letters like õ).. Reply 109 BattleHardened October 30, 2009 at 1:33 am Some of my more choice moments: postsuper -d ALL (instead of -r ALL, adjacent keys – 80k spooled mails gone). No recovery possible – ramfs :/ Had a .pl script to delete mails in .Spam directories older than X days, didn’t put in enough error checking, some helpdesk guy provisioned a domain with a leading space in it and script rm’d (rm -rf /mailstore/ domain.com/.Spam/*) the whole mailstore. (250k users – 500GB used) – Hooray for 1 day old backup chown -R named:named /var/named when there was a proc filesystem under /var/named/proc. Every running process on system got chown.. /bin/bash, /usr/sbin/sshd and so on. Took hours of manual find’s to fix. .. and pretty much all the ones everyone else listed :) You break it, you fix it. Reply 110 PowerPeeCee November 2, 2009 at 1:01 am As an Ubuntu user for a while, Y’all are giving me nightmares, I will make extra discs and keep them handy. Eek! I am sure that I will break it somehow rather spectacularly at some point. Reply 111 mahelious November 2, 2009 at 10:44 pm second day on the job i rebooted apache on the live web server, forgetting to first check the cert password. i was finally able to find it in an obscure doc file after about 30 minutes. the resulting firestorm of angry clients would have made Nero proud. I was very, very surprised to find out I still had a job after that debacle. lesson learned: keep your passwords secure, but handy Reply 112 Shantanu Oak November 3, 2009 at 11:20 am scp overwrites an existing file if exists on the destination server. I just used the following command and soon realised that it has replaced the “somefile” of that server!! scp somefile root@192.168.0.1:/root/ Reply 113 thatguy November 4, 2009 at 3:37 pm Hmm, most of these mistakes I have done – but my personal favourite. # cd /usr/local/bin # ls -l -> that displayed some binaries that I didn’t need / want. # cd .. # rm -Rf /bin – Yeah, you guessed it – smoked the bin folder ! The system wasn’t happy after that. This is what happens when you are root and do something without reading the command before hitting [enter] late at night. First and last time … Reply 114 Gurudatt November 6, 2009 at 12:05 am chmod 777 / never try this, if u do so even root will not be able to login Reply 115 Alex July 8, 2011 at 10:35 am A few days ago to give myself root’s pemissions I asked a collegue to do “sudo chmod 640 /etc/sudoers” on his ubuntu box. Result – sudo not working completely, and root’s password was unknown :/ Booting from LiveCD saved our day. But I consider this sudo’s behaviour in ubuntu rather stupid. Reply 116 Michael Shigorin July 8, 2011 at 11:57 am I’ve done (just as quite a few other folks) chmod -R .* back then, and still consider not reading manuals, not experimenting small scale beforehand, and blaming a tool instead of myself when I’m at fault rather stupid. (however dumbed down ubuntu might already be, suggesting breaking a controversial* but still security related tool even further isn’t gonna win someone IQ awards eh?) * security pros argue that PermitRootLogin without-password and several r_* accounts at UID 0 are better at avoiding password leak/privilege escalation: http://www.openwall.com/lists/owl-users/2004/10/20/6 Reply 117 Cody January 16, 2012 at 9:31 pm As Michael pointed out, the stupid thing is blaming a program for functionality when the person is at fault (please note I”m not trying or not even am calling you stupid, see next point). There’s a reason permissions are the way they are. Something may seem stupid, but does that make it so ? For instance : You know why you need to be root to chown a file even if it belongs to you ? Does breach of security mean anything to you ? Because as I recall, that would (could) be the end result. And yes this is very much related to permission issues. Reply 118 richard November 9, 2009 at 6:59 pm so in recovering a binary backup of a large mysql database, produced by copying and tarballing ‘/var/lib/mysql’, I untarred it in tmp, and did the recovery without incident. (at 2am in the morning, when it went down). Feeling rather pleased with myself for suck a quick and successful recovery, I went to deltete the ‘var’ directory in ‘/tmp’ . I wanted to type: rm -rf var/ instead I typed : rm -rf /var unfortunatley I didnt spot it for a while, and not until after did I realize that my on-site backups were stored in /var/backups … IT was a truly miserable few days that followed while I pieced together the box from SVN and various other sources … Reply 119 Henry November 10, 2009 at 6:00 pm Nice post and familiar with the classic mistakes. My all time classic: - rm -rf /foo/bar/ * [space between / and *] Be carefull with clamscan’s: –detect-pua=yes –detect-structured=yes –remove=no –move=DIRECTORY I chose to scan / instead of /home/user and I ended with a screwed apt, libs, and missing files from allover the place :D I luckily had –log=/home/user/scan.log and not console output, so I could restore the moved files one by one next time I use –copy instead of move and never start with / these 2 happened at home, while working I’ve learned a long time ago (SCO Unix times) to backup files before rm :D Reply 120 Derek November 12, 2009 at 10:26 pm Heh, These were great. I have many above.. my first was reboot ….Connection reset by peer. Unfortunately, I thought I was rebooting my desktop. Luckily, the performance test server I was on hadn’t been running tests(normally they can take 24-72 hours to run).. symlinks… ack! I was cleaning up space and thought weird.. I don’t remember having a bunch of databases in this location.. rm -f * unfortunately, it was a symlink to my /db slice, that DID have my databases, friday afternoon fun. I did a similar with being in the wrong directory… deleted all my mysql binaries. This was also after we had acquired a company and the same happened on one of their servers months before.. we never realized that, and the server had an issue one dady… so we rebooted. Mysql had been running in memory for months, and upon reboot there was no more mysql. Took us a while to figure that out because no one had thought that the mysql binaries were GONE! Luckily I wasn’t the one who had deleted the binaries, just got to witness the aftermath. Reply 121 Ahmad Abubakr November 13, 2009 at 2:23 pm My favourite :)  sudo chmod 777 /  Reply 122 Paulraj May 20, 2011 at 6:19 pm @ Ahmad, As I am a beginner using Ubuntu, I too faced the same problem of this chmod command. should be careful when using the commands like this. @ Vivek, thanks for this nice post. Bookmarked it ! Reply 123 jason November 18, 2009 at 4:19 pm The best ones are when you f*ck up and take down the production server and are then asked to investigate what happened and report on it to management…. Reply 124 Mr Z November 19, 2009 at 3:02 pm 125 John November 20, 2009 at 2:29 am Clearing up space used by no-longer-needed archive files: # du -sh /home/myuser/oldserver/var 32G /home/myuser/oldserver/var # cd /home/myuser/oldserver # rm -rf /var The box ran for 6 months after doing this, by the way, until I had to shut it down to upgrade the RAM…although of course all the mail, Web content, and cron jobs were gone. *sigh* Reply 126 Erick Mendes November 24, 2009 at 7:55 pm Yesterday I’ve locked my self outside of a switch I was setting up. lol I was setting up a VLAN on it and my PC was directly connected to it thru one of the ports I messed up. Had to get thru serial to undo vlan config. Oh, the funny thing is that some hours later my boss just made the same mistake lol Reply 127 John Kennedy November 25, 2009 at 2:09 pm Remotely logged into a (Solaris) box at 3am. Made some changes that required a reboot. Being too lazy to even try and remember the difference between Solaris and Linux shutdown commands I decided to use init. I typed init 0…No one at work to hit the power switch for me so I had to make the 30 minute drive into work. This one I chalked up to being a noob…I was on an XTerminal which was connected to a Solaris machine. I wanted to reboot the terminal due to display problems…Instead of just powering off the terminal I typed reboot on the commandline. I was logged in as root… Reply 128 bram November 27, 2009 at 8:45 pm on a remote freebsd box: [root@localhost ~]# pkg_delete bash The next time i tried to log in, it kept on telling me access denied… hmmmm… ow sh#t (since my default shell in /etc/passwd was still pointing to a non-existent /usr/local/bin/bash, i would never be able to log in) Reply 129 Li Tai Fang November 29, 2009 at 8:02 am On a number of occasions, I typed “rm” when I wanted to type “mv,” i.e., I wanted to rename a file, but instead I deleted it. Reply 130 vmware November 30, 2009 at 4:59 am last | reboot instead last | grep reboot Reply 131 um yea June 14, 2011 at 7:39 pm I have done that before. Reply 132 ColtonCat December 2, 2009 at 4:21 am I have a habit of renaming config files I work on to the same file with a “~” at the end for a backup, so that I can roll back if I make a mistake, and then once all is well I just do a rm *~. Trouble happened to me when I accidentally typed rm * ~ and as Murphy would have it a production asterisk telephony server. Reply 133 bye bye box December 2, 2009 at 7:54 pm Slicked the wrong box in a production data center at my old job. In all fairness it was labeled wrong on the box and kvm ID. Now I’ve learned to check hostname before decom’ing anything. Reply 134 Murphy's Red December 2, 2009 at 9:11 pm Running out of diskspace while updating a kernel on FreeBSD. Not fully inserting a memory module on my home machine which shortcircuited my motherboard. On several occasions i had to use a rdesktop session to windows machine and use putty to connect to a machine (yep.. i know it sounds weird ;-) ) Anyway.. text copied in windows is stored differently than text copied in the shell. Why changing a root passwd on a box, (password copied using putty) i just control v-ed it and logged off. I had to go to the datacenter to boot into single user mode to acces the box again. Using the same crappy setup, i copied some text in windows, accidently hit control-v in the putty screen of the box i was logged into as root, the first word was halt, the last character an enter. Configuring nat on the wrong interface while connected through ssh Adding a new interface on a machine, filled in the details of a home network in kudzu which changed the default gateway to 192.168.1.1 on the main interface. Only checking the output of ifconfig but not the traffic or gateway and dns settings. fsck -y on filesystem without unmounting it Reply 135 Michael Shigorin January 9, 2011 at 7:55 pm > Why changing a root passwd on a box …do check that you can still access it while having a root shell open. This applies to sudo reconfiguration, groups, uids/gids, upgrades(!), and to some extent to network interface configuration and firewalls. I’d usually reconfigure iptables like this — under screen(1) of course: [apply changes]; sleep 30; [rollback changes] where “apply” might be iptables -A/-I (then rollback might be iptables -D or -F), or “service iptables restart” (with “service iptables stop” to let me back in). Sure the particular solution depends on existence of e.g. NAT rules to still access the system but that’s rather nasty a habit of itself. If I press Enter after considering the changes being done and suddenly the screen stops to respond, then I’ll wait half a minute and hopefully get the console back to reconsider. Reply 136 ehrichweiss December 3, 2009 at 6:55 pm I’ve definitely rebooted the wrong box, locked myself out with firewall rules, rm -rf’ed a huge portion of my system. I had my infant son bang on the keyboard for my SGI Indigo2 and somehow hit the right key combo to undo a couple of symlinks I had created for /usr(I had to delete them a couple of times in the process of creating them) AND cleared the terminal/history so I had no idea what was going on when I started getting errors. I had created the symlink a week prior so it took me a while to figure out what I had to do to get the system operational again. My best and most recent FUBAR was when I was backing up my system(I have horrible, HORRIBLE luck with backups to the point I don’t bother doing them any more for the most part); I was using mondorescue and backing the files up to an NTFS partition I had mounted under /mondo and had done a backup that wouldn’t restore anything because of an apostrophe or single quote in one of the file names was backing up, so I had to remove the files causing the problem which wasn’t really a biggie and did the backup, then formatted the drive as I had been planning………..only to discover that I hadn’t remounted the NTFS partition under /mondo as I had thought and all 30+ GB of data was gone. I attempted recovery several times but it was just gone. Reply 137 fly December 4, 2009 at 3:55 pm my personal favorite, a script somehow created few dozens file in /etc dit … all named ??somestrings so i promplty did rm -rf ??* … (at the point when i hit [enter] i remebered that ? is a wildchar … Too late :)) luckily that was my home box … but reinstall was imminent :) Reply 138 bips December 6, 2009 at 9:56 am il m’est arrivé de farie :  crontab -r au lieu de :  crontab -e ce qui a eu pour effet de vider la liste crontab… Reply 139 bips December 6, 2009 at 9:59 am also i’ve done  shutdown -n  (i thaught -n meant “now”) which had for consequence to reboot the server without networking… Reply 140 Deltaray December 6, 2009 at 4:51 pm bips: What does shutdown -n do? Its not in the shutdown man page. Reply 141 miss December 14, 2009 at 8:42 am crontab -e vs crontab -r is the best :) Reply 142 marty December 18, 2009 at 12:21 am the extra space before a * is one I’ve done before only the root cause was tab completion. #rm /some/directory/FilesToBeDele[TAB]* Thinking there were multiple files that began with FilesToBeDele. Instead, there was only one and pressing tab put in the extra space. Luckily I was in my home dir, and there was a file with write only permission so rm paused to ask if I was sure. I ^C and wiped my brow. Of course the [TAB] is totally unneccesary in this instance, but my pinky is faster than my brain. Reply 143 Janne K. October 13, 2011 at 2:18 pm Tab completion is *so* handy, I love it. Back in the days, my zsh didn’t ask too many questions. # rm -rf /etc/(something)[TAB][CR] Note that ‘#’. Well somehow, the (something) part there got lost and my fingers, of course, were faster than my optical nerves and brain. Lady luck was smiling on me that day, this happened to my own workstation. Try running without /etc, it’s quite hilarious. (Never, ever rm -rf anything beginning with ‘/’. cd, cd, cd, cd, cd…) Reply 144 Sam Watkins October 14, 2011 at 6:50 am with GNU rm, I like to put the -rf at the end, like this: rm /etc/foo/bar -rf so I can be careful! Better still to use mv, i.e. mv -i /etc/foo/bar/ ~/trash/ or a ‘move to trash’ script aliased to ‘rm’. Then you can empty it later if you really did want to destroy it, like those desktop users do. Reply 145 fastgsx December 19, 2009 at 2:54 am Worst mistake ever, ran fsck -y on an encrypted volume, trashed the whole partition. You can only check the filesystem after its unlocked and mounted as a logical volume. Reply 146 CB December 23, 2009 at 10:40 pm Worst. Sysadmin. Ever. …nah, just kidding. If anything, you’ve proved your worth by helping newbies like me who don’t know what they are doing. I’m taking your lessons to heart. Thanks for taking the time to put them out there. Reply 147 Evilnick December 29, 2009 at 4:51 pm heheh. yeah, i think we have all been there. I always change the default prompt on server boxes, preferably to some flashing, garishly coloured text just to remind me i am not at home any more. Reply 148 Solaris Admin January 4, 2010 at 3:57 am Solars 10 – Hot drive replacement – Lab system prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s – /dev/rdsk/c1t1d0s2 …… however c1t1d0 was my good disk… not the one I had just replaced :( Reply 149 Danny January 6, 2010 at 7:34 pm  chmod -R www-data:www-data /etc/  Later on….  reboot SSH never came back up. Wouldn't start with those permissions Reply 150 LongShot January 8, 2010 at 8:58 am My own favorite was “chown -R userx:userx ../*” when I meant ./* in /opt/somdir – it recursed nicely, I cursed not so nicely. In my defence, I was trying to make sure I got the . files. It took many hours to straighten that mess out. Another favorite was on day one of a new job. The local alpha-geek was hotdogging and he ran a script that pushed a new user into /etc/password on all production servers. But the script had no error checking and he ended up zeroing out /etc/passwd on every single one (30+ HPUX). It was like watching a slow-motion trainwreck. I felt much less intimidated after that ;) In terms of that sinking feeling, I was telnet’d in to multiple production servers at multiple call centers (pre SSH – yes, I’m that old). One sever started circling the drain (known database ipc problems) and the only solution was a quick reboot before it locked up. I grabbed a window and ran shutdown. Of course it was the wrong window so I took down 250+ people at a remote site and let the sever lock up at my own site for another 250+ people. The remote site was bad enough, but after the hard power-off at my site I had to repair around 20 large ISAM database files which took about two hours. Now I try to use a different background color for each server I connect to. Reply 151 i-ghost January 9, 2010 at 5:55 pm You can always modify /etc/bash.bashrc and add: alias rm"=rm -i" Mine reads: alias rm=" echo -------; echo Think before you delete...; echo Use yes on stdin or -f if you must bypass the prompt; echo You have been warned!; echo -------; rm -i"  It’s not the best fix, and there are better ones out there, but it works fine for home desktops. Reply 152 mh166 January 11, 2010 at 9:41 am # svn rm foo –force and it deleted the foo directory from disk :(…lost all my code just before the dead line :( Oh boy … I did the very same thing leading to everlasting loss of dozens of TEX-files … Another one happened to me when I was going to create a cronjob that deletes all files older than X days. So I was at the shell in the correct folder and tested it:  find -mtime +23 -print -exec rm -f \{\} \;  Worked like charm … Therefore I was putting it just like that into the crontab and went to sleep. On the next day I got a 8+ Meg Logwatch-Mail … thousands of lines telling me some libs weren’t found. The bad thing: I didn’t give a starting directory for find, since for testing I did cd to it … Therefore the cron started deleting from where it was: right away at / …. Lucky me I had a second box that was almost identically … *pheew* But of course, the classics have also been done: wrong box, firewall lockout, halt instead of reboot, deleting the just-restored files from the production-directory instead of the temporary location under /tmp … there was lots of fun. And i bet there will be much more waiting for me to happen. ;) Regards, mh166 Reply 153 Not Available January 13, 2010 at 6:31 pm Copying many gigs of stuff with dd command. If you mix the source and destination, you very precisely make of copy of the blank disk over whatever it was you thought you were copying. Reply 154 KING SABRI January 14, 2010 at 2:25 pm I was Make Fatal Mistake also yum -y remove *python* it’s make damage to most of programs :( thank you Vivek Reply 155 sidhabo January 15, 2010 at 8:07 am Doing a lot of copy/paste from installation manuals always risky. Like copy with this not too unusual prompt: hostname :> /bin/sh If you mistakenly copy not only /bin/sh you end up wiping out whatever executable you where trying to run ( in this case /bin/sh ) by doing # :> /bin/sh Reply 156 Luke Seelenbinder January 15, 2010 at 8:28 pm Worst mistake: rm -rf /* I meant this: rm -rf ./* Yes… production server. Yes… I had to totally reinstall the system. Yes… I didn’t have a current backup. And you guys that you did stupid stuff. :-) Reply 157 Justin Anthony January 18, 2010 at 2:27 am i dont consider myself a noob but i did the most noobish thing in the world today ls -a > ls Nothing wrong with it, except i was in the /bin directory testing some new shell scripts I couldnt figure out why my ls command would return a blank on every folder. Reply 158 vicky January 20, 2010 at 3:12 pm It was really great of u all to write these mistakes.I have just started on, and would always remember to check and check again, the commands I use !! :) Reply 159 Sam Watkins January 20, 2010 at 7:59 pm One mistake I made was to run “slay” as an unprivileged user. This damn program by default (mean mode) will kill all your own processes in that case. It shouldn’t be shipped like that in a serious distro like Debian or Ubuntu, but they ignored my complaint. Reply 160 Yogesh January 21, 2010 at 12:23 pm I used rm -rf * instead of rm -rf xxx.* That too in tomcat server home directory. In result needed install tomcat server one more time. :) Reply 161 matt January 22, 2010 at 4:48 am instead of sudo sh -c ‘cat /etc/passwd > /etc/passwd.core’ i did: sudo sh -c ‘cat > /etc/passwd /etc/passwd.core’ needless to say this made the system useless. i’m not sure why i was even doing this when i could have done sudo cp /etc/passwd /etc/passwd.core. it think it’s because i sometimes do sudo sh -c ‘head -25 /etc/passwd > /etc/passwd.core’. the lucky part was that this server was not in production yet. Reply 162 heinzharald January 26, 2010 at 12:05 am Nice. Thank you for openly sharing. For all the “rm” errors – I’ve learned the hard way and have replaced “rm” via alias in bash to “rm -i” that way I get a wakeup call as soon as I want to delete big-time. I have to type “/bin/rm” to bypass it. Reply 163 Eddie January 26, 2010 at 1:08 am One time I typed g++ -o program.cpp program.cpp instead of g++ -o program program.cpp! It was a final proyect program, so I had to write it all over again in a couple of hours. One of my coworkers did: apt-get remove perl At first sight it does not seem to dangerous, but it removed many programs, including the X environment and even the apt applications, according to him. I used that command on my old computer when I got a new one, just to see what happened. :D Reply 164 Zachary January 26, 2010 at 4:24 am The amount of advertising on this site is disgusting. Reply 165 Paul January 27, 2010 at 12:26 pm # cat /dev/nul >/etc/motd came out for some reason as cat /dev/nul >/etc/passwd GOK why. A senior moment near going home time. Couldn’t find any old passwd files around so had to invent one with root in so I could log in at the console and extract the original from the backup tape. ps showed the users as numbers. No one noticed but my boss went into a sweat when I told him the next day :) SCO box. Last millennium! Reply 166 heinzharald January 27, 2010 at 3:24 pm @Zachary What ads? I use Adblock Plus for Firefox – not one f**kin ad left :-) Reply 167 DirtySaint January 27, 2010 at 7:04 pm hah! Hah!…majority seem to hv turtured “rm” to death. Surprisingly “rm” is still around… anyways my personal goof-up, about twenty years back, when there was no way of knowing where u r in the filesystem – except using “pwd”…no bash, no helpful PS1 configured to show ur current location, etc. (Actually after this episode I created a small script to show the current location). So here I was on a Xenix 286 system (apparently a unix “developed”/supplied by Microsoft), in single user mode and thinking I was in “/tmp”, issued “rm -r *” (during those days /tmp was just a simple directory)… Well! Instead of “/tmp” I was in “/” …rest u can imagine…also in those days “rm” was much more powerful…I don’t recall whether “-f” switch was yet invented or for that matter “-i”. Or maybe I was unaware!…”rm” was pretty raw and did what it was asked to do – no questions asked! But from that day I treat “rm” as my extra-marital wife…be real careful… Reply 168 Ben January 28, 2010 at 12:30 pm I had a Mail-System with a IMAP-Maildir-Structure. For any reason a single Maildir was created under root. And the Name of the User had an german mutated vowel in its name – so the name was not /.foo but /.foaer& or something. (Maildirs had a point in front of the name to know its a maildir) I copied the hole directory to the home-directory and wanted to delete the directory under root – and typed rm -rf . foaer& (you see this little space….). Now I had to do another work on the same machine – and i had to download a package with 300MB – i took wget and waited till it had been downloaded. But when half of the package was ready there comes a line behind the progress bar says: already deleted. Now I becomed scared – because of the “&” as last letter in the name of the directory – and then in the command – rm -rf erased the hole disk in the background…. And my last backup was about 2 weeks old – so, the hole company lost 2 weeks of mails – and i had a 24 hours job to completly setup the mail-server again – and this happend sunday night – my company startet at 06:00 o´clock in the morning to work. What a shitty night… But, after this there was a budget for a new Mail-Backup-System^^ So always be careful with mutated vowels and rm -rf! Reply 169 glee January 28, 2010 at 3:36 pm route -f xxx I was trying to add a backup route to a primary sybase server and didn’t include the “hop count”. Since it didn’t work, I tried to “force” it and ended up deleting all routes from the routing table. :/ Reply 170 Phyrstorm January 29, 2010 at 1:53 am Done just about all of the above. The worst I have done is dropped the public interface on a production server at a conference center in the middle of DefCon 17 at the Riviera. Luckily there was IT on site to get the system rebooted within 5 minutes. Reply 171 r4pP157 January 29, 2010 at 5:28 pm On a production Oracle RAC server, while connected to the console of one of the nodes through the system controller I typed: ~. Which sent the server to OBP, halting the OS instead of typing: ~~# To exit console and go back to system controller. Luckily that one time, the other node was up. Reply 172 CuriousJoe January 31, 2010 at 11:41 pm I had a usb drive with many folders of photos of my home in a main folder called “home”. I accidentally copied these as root and the folder was thus owned by root. Later on another computer went to this directory noticed the permission problem, switched to root, copied the photos and mysteriously typed: rm -rf /home in that situation that one “/” costs me a whole days work. Daft mistake, a well, live and learn. Reply 173 Wolf Halton September 5, 2010 at 5:12 am I had a similar experience. I had an old machine with several users. I wanted to add another harddrive make the /home partition there and move everything across. The problem came when I wanted to delete the old home directory to get more space back on the original disk. rm -rf /home deleted the new home directory contents, including all the old home directory’s contents. at the end I had 100% of nothing in the most important user’s directory. No back-up and no excuses at all. This happened 6 months ago, and I could have told me (if I had bothered asking) that this would happen. Reply 174 GuyK February 2, 2010 at 12:15 pm Is there a smart way to chmod an entire path, like mkdir -p ? As I typed , I forgot to remove an usual -R : cd /any/deep/path/to/allow/ chmod -R a+rx ../../../../.. oups. all the server was chmoded …. luckily it was on a virtual machine. reload and reinstall. Reply 175 Gilles Detillieux February 4, 2010 at 5:25 pm After accidentally rebooting the wrong server a couple times, forgetting what I had ssh’ed to, I decided on a handy fix, this /usr/local/bin/confirm script: #!/bin/sh echo "You are about to run the command '$*' on hostname." >&2
echo -n "Are you sure this is what you want to do? (Y/N): [N] " >&2
case "$x" in [Yy]*) exec "$@" ;;
*)      echo "Command '$*' aborted." >&2 ; exit 1 ;; esac Now, on my servers I add these aliases to root’s .bashrc file: alias halt=’confirm halt’; alias reboot=’confirm reboot’ That’s saved me a few times since then. The .bashrc file also already had the “alias rm=’rm -i’” in there, which at first I hated, but learned to like it after it too saved my skin on more than one occasion. Reply 176 Vivek Gite February 4, 2010 at 6:06 pm @Gilles Nice hack. I wish I had some sort of mod points here :) Reply 177 iKay May 19, 2010 at 11:02 pm Nice! That’s a very clever idea indeed. Noted; and will be using this in future :) Reply 178 amanda January 7, 2011 at 8:21 pm That is absolutely brilliant. I’m snagging this if you don’t mind. Reply 179 Mike February 9, 2010 at 7:36 pm I accidentally deleted /etc. I had restored /etc to another location and just went in and didn’t put the correct path in the command. That was a painful mistake. Reply 180 Salim February 10, 2010 at 10:01 am Accidently rebooted servers…best is uname -a before you hit reboot command. Wrongly plugged Sun M-series with one input 110V and 220v on the other. Server wouln’t start whatsover you do… Best backup is dump or ufsdump (solaris), most of the time tar and cpio may lead you to loose your job. Reply 181 Otto February 11, 2010 at 2:00 pm Adding a user to a group: # usermod -G groupname username If you forget the -G, you’ll have all the current user groups replaced by the one you thought you were adding. Very tricky. Reply 182 nathan February 13, 2010 at 12:37 am @Otto — I did basically that to myself with sudo,  # sudo usermod -G groupname myusername  I was immediately no longer a member of ‘wheel’ and was no longer allowed to sudo. Reply 183 Sarper February 14, 2010 at 10:07 am I have screwed myself most ways mentioned. My shining moment was building a firewall for a local government site. They said port 1049 was experiencing lag, so to debug I decided to clear the firewall. INPUT and OUTPUT were set to reject and I typed: iptables -X really bad part? It was a no-access VM, had to call the host to chroot and release. ALWAYS DO THIS:  iptables -A INPUT -j ACCEPT && iptables -A OUTPUT -j ACCEPT && iptables -X  Reply 184 Anonymous coward February 15, 2010 at 12:45 am tar -czf dirname1 & tar -czf dirname2 & tar -czf dirname13 & #Seems like all of them have been tarred and gzipped rm -rf dirname1 rm -rf dirname2 rm -rf dirname13 #Oh, some of the larger directories weren’t. Reply 185 Michael Shigorin January 9, 2011 at 9:08 pm Too bad, you should be running more or less serious hardware (for that mistake’s scale) to handle a dozen of active I/O+CPU jobs efficiently. Otherwise (e.g. on a older dual socket or even modern quad core) it would rather increase seek contention and scheduling overhead thus total execution time. …or wait(1) for them to finish at least. Reply 186 Andreas February 15, 2010 at 7:03 am I wanted to format my sd-card but I typed: dd if=/dev/null of=/dev/sda instead dd if=/dev/null of=/dev/sdc and watched my system beeing formatted in front of me. Reply 187 Steve March 18, 2011 at 6:47 am I did almost exactly the same thing, except I was copying a bootable image to the card. I ended up overwriting my primary hard disk’s partition table. It would have been easy to recover the data, had I not partitioned as XFS!!! Reply 188 Fool February 17, 2010 at 10:56 am On a Personal PC, not so much a linux mistake though… This was in the age of laptops moving to having NO more floppy drives, I needed some way of removing grub as it was no longer needed. Since there were no floppy drives I was searching for an alternative to the win98 bootdisk so I could run a “fdisk /mbr”. Found a nice little program that I burnt to a disc, that would allow me to modify the boot sector. … accidently ended up wiping the entire partition table! Reply 189 mordovorot February 18, 2010 at 5:49 pm Here is what I did: - Backed up /etc directory on a Sun Solaris server: oldbox> tar cf etc_oldbox.tar /etc - Transferred the tar to my home directory on the new server and extracted it to get some config files: newbox> tar xf etc_oldbox.tar The newbox was killed. Why? The native Sun tar doesn’t remove the leading “/” by default, as GNU tar does. As a result, the whole /etc content was completely overwritten with files from another server. Linux people, be careful with Sun Solaris! Reply 190 Mike Lama February 19, 2011 at 9:53 pm It’s not a Solaris fault. Even on Linux, tar will assume the base path of the argument. ALWAYS ALWAYS use . as the letter of the argument Reply 191 Daniel February 19, 2010 at 12:56 pm I saw a fresh installed centos system broken by command: yum remove python :) Reply 192 Katzider February 19, 2010 at 5:59 pm #init 0 On remote clustering server… Reply 193 Mike Davies February 21, 2010 at 9:05 pm This was years ago and now I think back about what an idiot I was. I created a user, mike of course in a box running DEC 4.0E and I decided I wanted more permissions so I decided to make it user 0, just like root. The system actually asked if I was sure I wanted to do this. I said yes….It took me 2 hours to get the system back up and running because it changed every file in the system that was owned by root to mike. dumb, real dumb. Being superuser can be a disaster! I use dd a lot now and make many backups. Reply 194 Kang February 24, 2010 at 3:09 pm Stupid french AZERTY keyboards have the * key located just on the left of the ENTER key. while typing some : mv /path/to/file /usr/local/ I pressed the ENTER key… pressing the * key at the same time :) It ended up executing : mv /path/to/file /usr/local/* which moved every dirs in /usr/local into the last directory in /usr/local Since then, I’m using only QWERTY keyboards ^^ Reply 195 james February 24, 2010 at 3:57 pm On at least three entirely seperate occasions I’ve been on the command line within a C project I’ve been writing and gone to remove backup files: rm *~ but been distracted after typing the asterisk and then switched back to the terminal and immediately pressed enter forgetting about the tilde sign. Immediately deleting weeks worth of *.c files. Luckily I found ext3undelete, and while I did recover my files, it painfully insisted upon recovering the entire partition. Which of course happened to be the largest partition. So I then spent the rest of the evening not only waiting for it to restore the files I actually wanted, but manually deleting everything it restored which I had *not* deleted in the first place so the partition I was restoring to, did not run out of space before the files I actually deleted were restored! Reply 196 zeve March 4, 2010 at 8:45 am I did: # route -f on my early first days, ended up with no routes. Reply 197 philbo March 4, 2010 at 2:49 pm A nice old one – not by me but by a less-than-worthy colleague, who liked to do everything as root on the server box: WHAT HE MEANT : last | grep reboot WHAT HE TYPED : last | reboot Reply 198 Micha March 4, 2010 at 7:45 pm “Best thing” happend to me. Tried to clone a Serversys after 12 hours configuration. But typed: dd if=/dev/sad of=/dev/sda bs=1024 Work was done for nothing. Samething even happend on Windows Server 2008 Backup Tool. What have I learned, check your syntax and read warnings when they appear … they are usefull at least for something … Reply 199 Nick March 8, 2010 at 6:15 am I work as a student worker at a university and as a web developer there have some limited sudo access on the production web server. They upgraded to a new server and didn’t warn me that they hadn’t made bash the primary shell. So, naturally I typed the same commands I always have to change ownership of a folder to myself so I could make some changes to files and inevitably changed ownership of the whole server to myself! Oops! sudo chown -R user folder [tab] / didn’t execute the same as sudo chown -R user folder/ Reply 200 Barius March 8, 2010 at 2:22 pm We had a cas of  rm -rf /  at work recently, but thankfully I was not the culprit. After that, they renamed the OPS team to OOPS. My fav was a laptop which dual-booted windows 2000 / linux. I installed VMware under w2k with raw disk access, which allowed me to boot the linux partition as a VM while runnining windows. This worked great for about a year, until I gave the wrong partition number to mke2fs and formatted the NTFS as EXT2 while w2k was running. W2k didn’t even notice until it’s fs cache was cleaned out by a reboot. After that it was–how shall I say–completely fscked. Reply 201 Kenneth Heal March 9, 2010 at 9:19 pm Once I installed a Linux server and forgot to log out of the local console. I noticed this and foolishly decided to pkill all the users processes. The user happened to be root and one of its processes was sshd; thus meaning I locked myself out of the box. Reply 202 matt March 12, 2010 at 3:50 pm The best one I have seen (and had to rebuild the server afterwards was someone’s Q&D tidy up script to clean old files out of /tmp… cd /temp; find . -mtime +30 -exec rm -f {} \; Practically everything on that line was fine… except it failed to complete the change directory and then went it’s happy way from / deleting anything that hadn’t been changed in the last month until it hit something important at which point the serverwent down. It was run by cron as root in the wee small hours so no-one spotted the effect until the next morning. The best thing about this apart from it not being _me_ that executed the command… the management believed the line that it was a virus… on AIX a decade ago. Reply 203 matt March 12, 2010 at 4:02 pm Oh, I forgot the other really funny error… customer who wouldn’t use vi… copy /etc/passwd to a PC using Windows built-in ftp, add a user in to it in notepad and then use ftp to put it back… and added a ^M to all of the entries! A call in late afternoon… ‘I can’t log on. Help!’ Reply 204 George March 17, 2010 at 7:59 pm First week with my new Ubuntu… - Program: Libc6 needs to be updated to run - User: - User: sudo apt-get remove libc6 - System: WARNING: DELETING THIS LIBRARY MAY AFFECT YOUR SYSTEM. TYPE ‘I’M AWARE OF THIS AND WANT TO CONTINUE’ IF YOU WANT TO PROCEED. - User: types ‘I’M AWARE OF THIS AND WANT TO CONTINUE’ … - System: *sigh*… ok… deleting everything that depends on C (the whole system) - User: open eyes, stomach twists, push “Power” button in panic - User: restart computer, nothing works. Grub had problems, can’t access windows installation. Doesn’t have Live CD. Has to ask to friends for a Live CD, and explain the whole story every time. Reply 205 Tom June 9, 2011 at 10:35 pm Sounds like I know that. Few years ago as newbie linux admin tried to upgrade very old Slackware router with single IDE drive. I discovered that openssl library has bug reported and wanted to upgrade it. But system said: first upgrade libc6 (or libc5, whatever). So I did it. It FAILED. Rebooted box (it has high uptime, nobody rebooted it long time). System didn’t come up. Also, I saw on screen: IDE TIMEOUT.. blach blach. Drive died. I had backup machine with clean Debian installed on mirrored raid, but still missing some things.. Lerned how Iptables works within 1,5 hrs :))) It happends.. almost exacly 28001 hours ago, becouse this server is still up and this is hard drives “Power on hours” attribute. Reply 206 I know this guy... March 17, 2010 at 11:49 pm Can’t top the keystrokes, but think I’ve got a top 10 spot for consequences: One of those ‘start rebuilding the wrong box in a failed redundant pair’ scenarios. Sadly it was the dispatch note / purchase order processing system for largest European lights-off central distribution center in a very big industry sector, and config had been ‘evolved’ or time. Result: they couldn’t dispatch anything and had to turn away every delivery for a day and a half. We are talking about a hundred cross-continental trucks sent back to their depots empty, or still full. I could tell you that it wasn’t me, but I don’t expect anyone to believe that. Moral: Never, ever, ever trust a sticky label – look at the prompt, and if the prompt doesn’t tell you: change it so it does. SVN’ed config would have massively reduced down time too. Reply 207 Clark March 22, 2010 at 2:58 am Was on a server, one of a dozen for TRAINING airmen and soldiers on a military base, not to track weapons or shoot anything. The same training scenarios have been run for months. Logs (multiple hundreds of megabytes were collected per day) are kept for at least a year. I wanted to copy a day’s log collection and reset the log queue so the next daily logs don’t get too terribly big for analysis (search for unauthorized logons.) I entered the command: # copy {logfile} {logfile.bak}; rm {logfile}; touch {newlogfile}; {start proven analysis script on logfile.bak}; Copy is a DOS command. Rm worked. I had no more logfile for one day of training. I immediately and voluntarily told the information assurance team lead that I accidently deleted this one log file. I was escorted off base within 15 minutes, no longer employed. This is a true story. I wish it weren’t. There is no humor in this event. Lesson for my next job: the rm command is the enemy when combined with idiot information assurance staff members. Avoid it. Use the mv command instead. Reply 208 Michael Shigorin January 9, 2011 at 9:29 pm I’d say no, rather “use && instead of ; (or don’t chain up stuff at all)”. mv can leave you longing for rsync if it bails out half-way through a large tree to move… of course, for a single file it’s rather about “use mv but don’t chain”. That lead probably wasn’t an idiot, just an army man. Still take my condolescence. Reply 209 Gerard March 22, 2010 at 10:31 pm My worst ever … in “/root” doing chown -r root:root .* .* also includes .. :) Reply 210 naomi March 25, 2010 at 5:14 pm update Customers set Surname = ‘Smith’ Not strictly linux CLI but … Cold sweat is a very apt description! Reply 211 Anonymous March 26, 2010 at 9:22 am nice mistakes Reply 212 Pradipta March 30, 2010 at 10:47 am I have put command “#last|reboot” to know the last login information on a production box and ended up with rebooting the server with 18 VPS. Reply 213 Rules to live by March 30, 2010 at 11:02 pm Yes, I have done some excellent fatfingers, IDtenTs, etc…more importantly here is what I do now to help combat human error…my personal favorite is /sbin/service network down and forgetting you are not on console…Doh! 1. Measure twice cut once – Look carefully at the commands prior to running them, don’t ever hit unless you really know what the command will do. 2. Build a practice – Being a SysAdmin is like being a doctor, think about how you do what you do, do it consistently…document it in a wiki ideally. 3. Take the wiki documentation you create and automate what you do in scripts so life is easier, and there are less mistakes, as you test your scripts on a lab machine. 4. NEVER TEST ON PRODUCTION 5. Create build plans for what you plan to do if it is complex…refer to your docs when you do it, ideally test your docs on a lab server or a DEV server prior to doing it on PROD. 6. Play nicely with others. 7. Hire junior admins to do the junior stuff to keep the senior admins to mentor, and do senior stuff… Reply 214 ju March 31, 2010 at 8:04 am Created perl script to fetch files from remote server but forgot to add check of directory existence. All of the remote files were lost because my scp command was overwriting them on the destination path. Reply 215 dexter April 2, 2010 at 12:04 pm I typed the 3 fingers salute on a windows server to log in on a kvm switch, but unfortunately, I was on the ERP under RHEL, the trap wasn’t desactivated… reboot took 30 minutes, it was while a work experience and I was alone at the I.T. department. Reply 216 NightDragon April 3, 2010 at 7:33 am In my company the default set up for desktop systems was a mv and rm without -i … so i wanted to delete some files … the usual mistake (a space between dir and the wildcard)… and thats it Reply 217 MEH April 7, 2010 at 6:06 am I was administering a centos- box remotely via ssh and trying to resolve a RPM hell of one sort. Finally got things down to ssh version conflict and decided to unistall it. # yum -y uninstall openssl I was bit puzzled to notice that my ssh- session was terminated and I had to find someone to access the box via console to install ssh back so I could restore all the damage done so easily. Reply 218 Chris de Vidal April 8, 2010 at 1:20 am Dump rocks, but I found out the hard way of its Linux limitations. Reply 219 Chris de Vidal April 8, 2010 at 1:23 am Clark: SAD story. Lesson I have to keep learning is not to chain commands with ; but && (you are not alone). Reply 220 Dave April 14, 2010 at 6:43 pm I loved this. I wish I could remember some of my specifics, but I’ll just have to add a general comment. It applies to anyone sitting at a console – and I learned them the hard way a long time ago. Never test a script or program on a production server. It WILL burn you eventually. We’re human, and we make mistakes. Test every file manipulation command by first listing the files that will be affected without actually modifying anything. Let your users know you’re taking the production system down well in advance, so they have plenty of time to prepare. Emergencies do happen, but you’d better be able to explain to the CEO why his administrative assistant’s work was suddenly interupted, or the presentation they were giving to a customer that just flew 12,000 miles to see it, was interrupted. Never run through the hallway like you’re racing to put out a fire – it scares people. Calmly act as though you have it all under control and use the extra time to THINK about what you can do to resolve the crises. Never go to your boss with an ill-formed, by-the-seat-of-your-pants analysis. If your wrong, you’ll look like a reactionary or a chicken-little. Take the time to think, analyze, and consult w/ peers. You’ll look more professional. Reply 221 Mr Z April 14, 2010 at 7:21 pm Dave, I agree completely, even the part about not scaring them LOL, but you did forget one thing: Have a backout plan and TEST that plan on the test system prior to even thinking about touching the production system. When you are certain that plan B works it’s really easy to look calm while you stop to get a cup of coffee on the way to the data center!! Trust me on that one. Reply 222 Mark Johnson April 16, 2010 at 7:23 am Setting up a testing server I wanted to copy /etc from another machine. On the machine I was copying to I ran mv /etc /etc.old instead of cp /etc /etc.old Reinstall, then… Reply 223 Mike April 20, 2010 at 8:38 am Why didn’t you just ‘reverse’ it then: mv /etc.old /etc ? Reply 224 Hans Henrik May 7, 2010 at 7:16 pm ah, or cp /etc.old /etc Reply 225 Marco April 16, 2010 at 4:20 pm While cleaning up someones mistake of running this on a live production box: rm -rf / home/[...]/file (note the space) All that was left was an SSH connection and existing services – the box was maybe half operational with no one onsite at the DC, restoring files to the box from another similar production machine with RSYNC. Went to restart a daemon with: ps -aef | grep (service) | awk ‘{print$2}’ | xargs kill -9

Ran the angry loopus but forgot the grep

ps -aef | awk ‘{print $2}’ | xargs kill -9 Within moments init was killed and the box was lost. Reply 226 Joey Adams April 16, 2010 at 4:29 pm This these aren’t Unix mistakes, but I was backing up files from my old Macintosh (Mac OS 7.6), so I sent them via FTP, but the program I was using defaulted to text mode. The real mistake was not knowing about md5sum . Also, I used TI-Connect to back up 2 years of BASIC programs from my TI-89. After backing up, I wiped my calculator. I wanted to move the backup archive to another directory, so I issued Cut from TI-Connect so I could Paste it into another. Paste didn’t work. Reply 227 Joey Adams April 16, 2010 at 4:30 pm Oops, should have proofread. s/This these/These/ Reply 228 Shehzad April 17, 2010 at 10:50 am With great power comes great responsibility. Once I run, rm -rf * and then later found few directories were required. (luckily already had backup of them all) Reply 229 me April 20, 2010 at 6:22 am made a script to delete files older than and put script file in that folder when script was old enough… Reply 230 Christian-Manuel Butzke April 20, 2010 at 8:33 am A perfect evening. At 3AM in the morning I finished revising some last minute changes on my local ubuntu virtual box, pushed them to git, updated the dev servers AWS instance, rechecked and then, ready to sleep and shut down my max, i entered a “init 0″ to shut down my local ubuntu box… good night… unfortunately it was not the ubuntu box i shut down, but the AWS instance. As AWS instances will be deleted on termination and shutting down an instance results in termination of the instance, the complete dev server + all the data was gone… No backup ( the instance had a ebs connected to contain the db data, but a year ago we switched from postgres back to mysql. for some pleasant reason, only postgres was using ebs…) I would have had much more fun in executing “sudo rm -rf /” manually… This enter key is quite a dangerous weapon….. Reply 231 Yehosef April 20, 2010 at 9:13 am I was making testing some changes to the database schema locally for a live remote database. After doing one of the tests, I was going to drop the local database and restore a previous version from a backup file. Unfortunately, I had both the local and remote database admin panels open in a separate tabs in my browser and I dropped the wrong (live) database. I had made backups of the live database shortly before this, but there was an error in the dump and only half the database was there. And the datacenter didn’t realize that we had wanted them to back up the database dir also… Reply 232 yehosef April 20, 2010 at 9:14 am I realized it’s not really a unix command line mistake – nevermind. Reply 233 slayedbylucifer April 24, 2010 at 5:45 am I guess this is the dumbest one. yesterday I installed RHEL-6 Beta and was logged into it from home via ssh. I had an X session active on the server which I left running yesterday when I left the office. /home is a LV and I wanted to reduce it. but i could not as it said volume in use. so i thought running killall5 will kill my active ssesion X Session (as lsof showed only my X-session is using /home) and will leave my ssh session from home running. haha, I ran killall5, and my putty got disconnected, I can no longer connet to my server. I will be going to office on monday and will start the ssh service. Fortunately, this was a test system I had configured to playaround with RHEL-6. Reply 234 matt April 26, 2010 at 6:44 pm when i was new to linux i used rm /home with no -i by default…. what was meant was ls /home read before you hit that enter key!!!!! Reply 235 Sepiraph April 30, 2010 at 5:55 am Mine wasn’t a Unix mistake but a IOS mistake … The date was Christmas’s eve, 2009. I was doing some firewall change for a friend I knew back in HS for his company router. I put: deny x.x.x.x 255.255.255.0 instead of deny x.x.x.x 0.0.0.255 since I didn’t use the reload timer command … he had to drive to the data center… Reply 236 Win May 5, 2010 at 2:59 am > Conclusion: > 3. Never use rsync with single backup directory. Create a snapshots using rsync or rsnapshots. Can you elaborate a bit on this? Not too sure what you mean. Reply 237 Michael Shigorin January 9, 2011 at 9:48 pm Maybe “you can ruin a single backup directory with e.g. rsyncing an empty dir onto it”. With snapshots (think cp -al), one has a bit more trouble destroying that backup — still replacing a heavily hardlinked file’s contents will ruin its contents in all the hardlinked snapshots. So disk based backup is complemented really nice by tape one (modern tapes can take around 800Gb uncompressed data at ~100Mb/s, and tape changers cost somewhat less than a fortune by now). Bacula is highly recommended and worth the time. Reply 238 Valeri Galtsev January 16, 2012 at 8:40 pm Hm… somehow rsync works differently for me (on FreeBSD, Linux – CentOS 6, Mac OS 10.6). In particular, rsyncing empty directory onto not empty directory doesn’t change anything. I.e., rsync works as documented: it copies over to destination _only_ files that have older timestamp on destination, leaving everything intact. If one wants to also keep an older copy of file being updated, one can add -b option (which will rename older “file” into “file~” before copying newer version. [valeri@point ~]$ mkdir test1
[valeri@point ~]$mkdir -p test2/test1 [valeri@point ~]$ touch test2/test1/file
[valeri@point ~]$rsync -avu test1 test2 sending incremental file list test1/ sent 57 bytes received 16 bytes 146.00 bytes/sec total size is 0 speedup is 0.00 [valeri@point ~]$ ls test2/test1/file
test2/test1/file

- you see, the “file” still exists! I use rsync for years, I never had anything trashed by it. But I _did_ my share of other flops described here!

239 dani May 6, 2010 at 9:38 am

last |reboot

last |grep reboot

while logged as root on critical production system … downtime were about 40 minutes :S

240 Simon May 10, 2010 at 1:21 am

In my home directory, I usually have a couple of .torrent files named “[isohunt] foo.torrent”, “[isohunt] bar.torrent” and alike.
Even if you’re sure you have many files starting with the same letters, don’t type
$rm $iso* In one case, there was only one such file, so what I did was essentially rm \[isohunt$ foo.torrent * instead of$ rm $isohunt$\ *

erasing my entire home directory (at least without subdirectories). What a shame

241 Alex May 13, 2010 at 9:34 am

I had two similar PCs where I had to install Ubuntu and similar software. I am a lazy boy ;), so I’ve installed Ubuntu on one of them (let’s say BoxA) then I inserted HDD from second one (let’s say BoxB) to BoxA and run:

dd if=/dev/sda of=/dev/sdb bs=1M

to clone this HDDs. For some reason HDD with working system was sdb after reboot. So in result instead of two working PCs I’ve got two clean HDD drives.

That was really epic fail :)

242 Valeri Galtsev January 16, 2012 at 8:56 pm

Yes, linux has quite a few weird (well, illogical) things. You had one drive, it was assigned sda device name. You added second drive to clone firs onto, but when you booted _linux_ live CD, former sda device was assigned sdb name and second drive: sda. Expect dd cloning to fail unless you know linux id weird here or better test which device is which. The same is (at least was for quite some time) about network interfaces. Linux names devices in an order of discovery of devices, only it reverses it as if it pushes devices into stack, and assigned names later when pulling them from stack. BSD in this respect is much better: it assigned device names according to their physical place in hardware.

243 Michael Shigorin January 16, 2012 at 9:22 pm

Valery, please note (as was noted to me before) that this page is explicitly for recalling troubles and not trying to market one’s crap over someone else’s crap, especially when one blames Linux instead of his own krivye ruki.

With FreeBSD in particular, you could end up with a kernel panic *just* by going over IIRC 7 vinum’s software RAID volumes back in the day, or by plugging a USB storage device into a router not so long ago.

Just in case, I did mess with SATA drives too by plugging “the next one” in a socket with the lower number (that’s pretty easy when one got 6 or 8 of them but quite feasible with even 4 or 2 when there’s no decent light source, you see). *BUT* I didn’t reverse the dd if/of so far due to a habit of “fdisk -l” and other double-checking before things like that.

Please stop spreading that lame misinformation on device numbering as the bus scan order can change with BIOS, kernel, or kernel option *BUT* you’re wrong again, uchite matchast’ (e.g. udev’s KERNEL variable description for a start).

One of the fundamental mistakes a *NIX admin can do is listen carelessly for some local “authority” who would “back” their words by being loud and proud, and not by being actually knowledgeable and reasonable (the test is “why?”). That’s pretty wide-spread in Russian-speaking *BSD circles, unfortunately.

244 Cody January 16, 2012 at 9:43 pm

Funny you’d say that Michael. I actually came to this thread for this same reason, but got caught in some other messages I wanted to reply to.

Indeed, there’s such a thing as labels.. uuid’s and so on. Using a different OS for not knowing how to fix a problem when its readily available knowledge seems silly to me. All OS’s have their own flaws and strengths. So does however each human.

But that said – I used to love bsd and scoff at linux. I was even called an elitist by some and frankly I do not blame them. My attitude combined with really extreme sarcasm, I did seem like one! I think one ofthe biggest things I learned (after years of blindness) is to use what works for you! As it happens I much prefer gnu libc than bsd’s much less useful libc (maybe improved by now ?). And this is coming from someone who used tothink, say, C++ is bloated (more specifically OOP’s). My reasoning or even my “defence” was that even Linus says it is. A silly and stupid logic there (more like no logic). I knew that much (it was more of well ‘if he sees it..’). However, as a friend said to me: you mean the person who created a very bloated OS ? Its true, Linux has a lot of stuff that a lot of people will never use. That doesn’t make it bad or useles.

Point was well taken and I now love OOP (it’s very beneficial, its more type safe [in C++ versus say C], and really it gave me something new to learn!).

In other words: Linux has flaws. BSD has flaws. Windows has flaws. MacOS has flaws.. NOTHING is perfect. Use what works for you.

245 Michael Shigorin January 16, 2012 at 10:11 pm

Yup, nothing is perfect under the moon.

Many of my mistakes could be less harmful to me — and others — if I learned it earlier that knowing the *weak* sides of what is available and avoiding them is way better than knowing the strong sides and just relying on them…

Thanks Vivek, it was a decent idea for a blog post to share the bumps earned and prompt us colleagues to do the same ;-)

246 Valeri Galtsev January 18, 2012 at 6:41 am

Wow, i didn’t mean to cause an explosion, sorry everybody… And compared to you, Michael, kernel expert, I’m just a humble sysadmin. I only put my dirty hands into kernel once, when we had 32 bit box with 8GB of ram: to get more than 1 GB for user data. I do use Linux a lot, as it’s something that just works for me. Way back I remember 3 year uptime of some of our Linux boxes. A couple of years back I started to seriously look at better alternatives, when Linux became more like windows: every 1.5 Months on average: kernel security update (==reboot)… Respectfully, – Humble Sysadmin.

247 Valeri Galtsev January 18, 2012 at 7:12 am

I don’t mind if moderator deletes my posts: I agree with Michael, my posts are just junk compared to elegant and instructive shell errors found here.

One thing I couldn’t buy as ultimately devastating though:
rm -rf /
- if I ever manage to do it as root on my *nix box I expect /bin, /boot, and part of /dev gone (and whatever else could be in / alphabetically before /dev on that box). Then the device hosting root filesystem will be deleted, and this will be end of my trouble. The rest: /home, /lib, /lib64, /sbin, /tmp, /usr, /var will stay intact. Other opinions?

248 Michael Shigorin February 7, 2012 at 5:33 pm

Just try that (not exactly that command but you’ll figure it out) in a virtual machine, then think of mmap’ed files, open handles, cached filesystem metadata.

On the bright side, on Linux at least one can salvage a (wrongly) deleted file at times by knowing it is still open by a running process, SIGSTOPping that process just in case, examining /proc/THAT_PID/fd/ symlinks and cat(1) the contents of the needed one, conveniently marked as “(deleted)”, into a safe place.

I’m not a kernel guy either, only fixed iso9660 perms back in 1999 or so for localhost :) but I know freebsd FUD when I see it, and if you failed to find a distro that works for you (like I did in 2001), don’t blame “linux” for it — it’s just not professional in the first place. Ну, не стоит уши под лапшу подставлять и дальше её тиражировать.

Back to topic: on Linux it might be safer to check /proc/mounts and not mount in case one has to doublecheck: I once had a trouble with a recently cloned hard drive after a reboot, blasting the wrong one with dd(1) after having missed the *real* mounts state (don’t remember the details but LABEL and UUID won’t help to differentiate between bit-per-bit copies, obviously). There’s a tendency to have /etc/mtab just symlinked to /proc/mounts though.

249 monkeyslayer56 May 13, 2010 at 5:47 pm

i accidently blocked windows from squid proxy… 99% of the computers on the net are windows…

250 Ruban May 14, 2010 at 9:58 pm

I usually work night shifts, was doing healthcheck as usual on my office servers. There was one time where i press command top to check the memory usage.

Saw that mysql process Cpu usage at 99%, i thought by killing the mysql process it will free up the resources and we can start back later.

Once i kill the pid, i start receiving alerts the server is down and all the transactions was pending at GUI.

Paniced and called my senior at 4am, luckyly he thought me how to start back the mysql process. ^_^

Learned from mistakes. ^_^

251 Michael Shigorin January 9, 2011 at 9:59 pm

Heh, fixing what ain’t broken (admin-wise), and then postponing the “how *exactly* can I start what I try and stop now”. It’s like speeding the crossroads, might just make it 90% of the time but die the tenth time…

252 iKay May 17, 2010 at 4:03 pm

Where to start, where to start!

Having a lot of servers from one provider, all with similar hostnames, and reinstalling the OS of the wrong server.

Rushed setting up a server; normally I setup alias for cp so that it runs cp -R. So I need to back up an SQL database. I go in to the MySQL data directory and run cp database_name /root/db_backup/database_name. I proceed to run killall mysqld and then rm -rf database_name.
Reboot the server and SQL comes up all is fine so killall mysqld and cp /home/db_backup/database_name /var/lib/mysql/. Bring back up MySQL and then try to hit the website. Realise that I have only copied the files and no directories, the database is incomplete and site is destroyed. No other backups.

Setup a script to automatically ban IP addresses on 5 failed login attempts to SSH and theres no timeout to remove banned addresses. I also blocked my server provider from getting into the box via SSH key. I changed the root password then logged out of the box. Went to sleep and forget what I changed the password to then after 5 attempts I’m locked out. Give my provider what I think is my root password and after 5 attempts they are locked out at the NOC.

253 Michael Shigorin January 9, 2011 at 10:02 pm

sshutout at least has timeouts and a whitelist, albeit I managed to help a colleague lock himself out for something like 6 attempts at his password :)

254 Navneet May 17, 2010 at 4:43 pm

i want to run unix commands with multiple options…like
snmpwalk -Cpublic 192.168.1.1 and get the result on a webpage….how do i give the options..

255 JP May 18, 2010 at 6:58 pm

crontab -r instead of crontab -e
doh!!

256 crontab doh'er May 20, 2010 at 2:45 am

crontab -l before any crontab -e is a friend to all :)

257 Mike -r May 25, 2010 at 1:26 am

I now have this nickname at work. On deployment did a crontab -r on a critical system with no backup of the crontab easily accessible. The worst!!

258 Jeez Man May 20, 2010 at 2:54 pm

Jeez Man! Going about advertising your mistakes, especially where you rebooted the Oracle DB Box! I wouldnt employ you!

259 Exolon May 22, 2010 at 12:04 am

Then you wouldn’t find anybody to hire (or you’d hire a liar who claimed never to make booboos). Everyone makes mistakes. _Everyone_.

Probably the best one I made in the last while was removing a program that didn’t work on my Mac at home… I typed in “rm -rf /Applications/”, thought that I’d typed the initials of the program and hit tab to complete it, then hit enter. It didn’t match anything of course, so about 3 seconds passed while it deleted programs before I noticed and hit ctrl-c. It got as far as C*, so I was able to re-install the stuff I’d nuked :/

260 Juraj May 21, 2010 at 1:53 pm

Why anyone insists on using > and >> operators instead of proper text editor, is beyond me…. even if it would save some time (I doubt), it is not worth it.

261 iDale May 26, 2010 at 5:54 pm

Very much worth it from within a well tested script. I agree with you in spirit, though: using them manually is just asking for it!

262 riffraff June 17, 2010 at 8:48 pm

You must not have very many servers to look after.

263 Cody June 3, 2011 at 2:07 pm

Well you should learn it rather than fear it.

It’s not hard. It’s not elusive. It’s not difficult to learn. Editors have their uses. But if you’re also afraid of > or >> then I guess you’d be afraid of sed and awk or pipelines or any command that edits a file. Think of sed -i. One screwup and you could wipe or make useless, an important file or many files. Yet, I guarantee it’s not only FAR faster, it is worth it and among the best solution for many problems (multi file search and replace and similar). And as for > and >>, here’s another example of how knowing how it works, saved my hide.

Imagine this: mistake in or missing entirely /etc/fstab. In my case, I think the root filesystem entry was gone but this was years ago so I could be remembering wrong. I may have even made a mistake. Whatever, the important point is, I knew how to fix it. It was on a FreeBSD box is all I remember. Regardless: you don’t always have editors. One such reason: think of linked in libraries. Another: a not mounted file system (possibly this is the most relevant reason in the case I fixed).

So how did I solve this issue without having to reinstall (remember: no editor!) ? Simple:

cat > /etc/fstab <> for multiple lines.

Yes, I actually reconstructed /etc/fstab by knowing how to use the shell.

So that’s one reason people insist on using it: it is VERY useful and not really hard to keep straight. It reminds me of the quote that goes along the lines of ‘unix is user friendly, but it’s very particular about who it is friendly with’.

264 Cody June 3, 2011 at 2:10 pm

Of course, that was misinterpreted by the site.

Bleh. it should be two left arrows and then “EOF”.

E.g.:

cat > /etc/fstab << “EOF”

EOF

Hopefully that comes out properly this time.

265 AmbientAngels May 22, 2010 at 5:13 am

A few years ago when I was still relatively new to linux, I had my windows drive mounted as /windows/c on the system. I wanted to delete my wine windows directory. So I moved to the .wine directory and ran rm -r /windows. I never had the opportunity to abort as I had left the house after running that command.

To this day I’m still grateful I had gotten into the habit of backing up data on a regular basis.

266 me May 28, 2010 at 6:19 pm

I liked this one:

rm *#*

The intention was to remove all files containing ‘#’ sign – these were some temporary ones. Unfortunately, my shell interpreted ‘#’ as beginning of comment. Guess what happened :)

267 Bodsda June 2, 2010 at 6:29 am

rm -rf * ~/junk

Never put that asterisk on the wrong side :)

268 iMadalin June 9, 2010 at 2:34 pm

can’t stop laughing :)) you guys are great :)

269 TheEye June 2, 2010 at 5:23 pm

instead of ‘crontab -e’ I mistakenly hit the ‘r’ instead of ‘e’, and removed the root cron completely

Whoever wrote that to allow a remove without verification, bad, very bad…

270 Dave July 22, 2010 at 10:10 am

Yes, I’ve done this too many times. R and E are so close on the keyboard and no are you sure??

271 Anonymous June 3, 2010 at 2:16 pm

rm -r /var/* instead of rm -r ./var/*

272 dudhead June 5, 2010 at 2:01 pm

When a s/w installation went wrong, I decided to delete the incomplete installation, which included a /bin directory, then merrily type y for all files when asked if I wanted to delete it….lost system /bin, couldn’t do anything. Got Ubuntu live, copied /bin from a sister machine, then manually created all symbolic links. Found I couldn’t use su anymore, then gave chmod u+s su (and other similar files). Recovered all!

273 Anonymous June 5, 2010 at 5:55 pm

You got way lucky on that one. Reminds me of the time I ssh’d into the mail server to work on an Inbox and didn’t exit out. At closing time I found an open terminal on my laptop and typed sudo /sbin/shutdown -h now….it wasn’t my laptop I was shutting down.

274 Chris M June 8, 2010 at 6:48 pm

On FreeBSD and new to rsync, set up rsyncd.conf with a new module section named svntrac to allow syncing from path /usr/local and including two subfolders (one for svn repositories and one for trac repositories). On client machine for backup as root ran rsync -avzr –delete server::svntrac /usr/local.

It backed up my svn and trac repos fine, but deleted everything else from /usr/local ! All installed programs gone just like that (including rsync!)

275 Mark Scholten June 9, 2010 at 9:43 am

I once used rm progname * instead of rm progname*.
Just one additional space in between the progname and the asterix.
Unfortunately the system was used by two departments. Each department thought that the other one made backups of the system. So nobody did. Took me two weeks to rewrite the code. Since that day I always use rm -i progname* and check doublecheck.

Years later, I accidentally put the wrong permissions on /etc/passwd. So no one could login into the system. Not even root. Luckily some IBM guy came along on the same day to repair a harddisk. He knew a back-door into the system and saved my day.

Sofar only 2 mistakes in 20 years of Unix. Not too bad.

276 Simon June 9, 2010 at 3:22 pm

I was once working on a mounted samba share when I discovered two directories which seemed to have identical content. Let’s call them foo and bar. To make sure I did a diff foo bar on the directories, returning no differences.

In order to free those 5GB of disk space, I continued to rm -rf bar when, after about 10 seconds, it struck me that I was working on a samba share and aborted the operation, luckily.

Next, I logged in using SSH and discovered that bar was only a symbolic link to foo, a fact that is hidden from the user when working on a mounted share :-/. Well… the 1.5GB that were already deleted could be recovered for the most part, but I certainly learned a lesson. Don’t trust samba shares ;-)

277 Russell June 9, 2010 at 10:01 pm

LOL!! I hope I never do one of those mistakes!!

278 atlgnt June 11, 2010 at 6:45 pm

One time I was copying some neat stuff unix commands along with the process steps I documented.from a journal i had on Lotus notes…was going to paste into a txt file as I moved to a new mail system….. well I pasted the info into the worng window…..as i had several windows open at the time…. one of the commands was a shutdown command i had in some processing steps to prepare for some work to be done on a unix server……so I shutdown a production server,,,amazing how fast it shutdown….not so amazing how long it took to come back up…..lucky for alerts…instantly got an email saying system was down….i said ..what idiot shut down that box….oopps …that idiot was me……lucky most folks were gone for the day….so it was not as bad as it could have been

279 Jason Barnett June 15, 2010 at 5:51 pm

chown -R jbarnett /

yeah… it didn’t end so well =/

280 oops June 17, 2010 at 4:56 pm

chmod -x /bin/chmod

281 Andy T June 17, 2010 at 7:42 pm

I love this one most of all because it made me think about the fastest way of recovering the situation: fire up the filesystem tools and toggle some bits? su and then cat the contents of /bin/chmod over another less important system executable? something much simpler I’ve missed? some reason a rebuild was unavoidable – please expand because I know it’ll be useful…

282 Sam Watkins June 21, 2010 at 12:33 am

to fix that, you could do something like this:

cp -p /bin/sh ~/chmod
cp /bin/chmod ~/chmod
~/chmod +x /bin/chmod

It would be more difficult if you had done this: chmod -R -x /bin
In that case you might be able to rescue it with the current shell and something in /usr/bin, or else use a rescue disk such as knoppix, not sure…

283 Mike June 17, 2010 at 8:17 pm

My personal favorite that I did fairly recently was changing my login password, and forgetting to change my ecryptfs password before restarting… When I restarted, I could login with my new password, however when it would try to pass the new password on, it wouldn’t match the ecryptfs password, and it could not mount my /home directory… sucked!

I then made things worse by logging on to a different account, su-ing to my main account, and trying to change the ecryptfs passphrase to match the new password I picked…

I ended up getting cought in a loop of changing passwords and su-ing into parrallel accounts… I eventually gave up on it and formatted/reinstalled…. SUCKED!

284 riffraff June 17, 2010 at 8:52 pm

Did the killall thing on an AIX box. Entered ‘killall’ just to get the syntax and switches. Next thing I know, my ssh connection is gone and people are calling about the server being down. Fun times.

285 kajienk June 18, 2010 at 11:56 am

My best error was following.
I reinstalled Linux and copy home folder backup from ntfs partition. While I was removing the annoying executable flag from all files with something like
chmod -x * -R
I ended removing executable flag in whole /bin :(

Especially /bin/bash is very troublesome :)) I couldn’t log back I couldn’t do anything. Installation again….

286 Eric June 21, 2010 at 11:24 am

I have done the “userdel” one. But I am wondering how to delete user home directory only while keeping their mail spool..

287 8bit July 6, 2010 at 3:15 pm

Done this a few times – thought I was logged onto a Red Hat box and done “init 5″. Surprised to see server start to shutdown, instead of transitioning to multi-user with graphical login. Realise it’s a Solaris box, runlevel 5 on Solaris is shutdown and power-off…. :|

288 Karin July 11, 2010 at 7:19 pm

I was moving some files and accidentally typed this;

mv / /home/….

Everything under the root directory was moved to the home directory. Yeah!

289 edward July 12, 2010 at 10:26 pm

I typed
chmod -R USER:USER /* instead of chmod -R USER:USERS ./*

changed the whole dang webserver to a single owner, that wasnt root.
All 87 domains. :(

290 dhx July 16, 2010 at 10:06 pm

I have done a lot of “the classics” but my best was :

being in /backup
meaning to write rm -rf www/
i wrote rm -rf /www
guess what, it was nice..
but the worst part was a second later I saw my mistake, and wrote the right command deleting the backup _ALSO_
just then I realized what have I done…

291 Dave July 21, 2010 at 12:11 am

“chown -R root:root /” instead of “chown -R root:root .” == reinstall of whole system and a night without sleep :-)

292 Dave July 21, 2010 at 12:22 am

Also did “cat db_backup.sql > mysql” instead of “cat db_backup.sql | mysql” as root and destroyed the server binary requiring a complete rebuild from source

293 giany July 22, 2010 at 7:37 am

I told a client to do a : chmod 600 /root/.ssh/authorized_keys2 and he did :

chmod 600 / root / .ssh / authorized_keys2

294 Douglas July 22, 2010 at 8:20 pm

I had a years worth of MySQL backups using the XML format, unfortunately i had failed to read the FULL manual and therefore did not know that while MySQL would in fact write to an XML file, it could not read or import an XML file for the version on the server.
Several panicked hours later, I had a working setup of the latest version (from the website, not pretty), and had managed to import and the XML file just to reexport them in a format readable by the version we had on the production server.
Lesson learned: always read the full manual before trying new or “better” features. We all thought the XML format was great, more portable, etc…
Oh, and the newest version (as of May 2010) could not import the files directly, they were too large and in the wrong format, I had to write a Perl script to do everything in chunks.

295 Elton July 29, 2010 at 4:28 pm

My keyboard is under my desk, in a sort of a drawer, and the monitor is fixed on the wall, so I have free space to work on my desk.

I was trying to solve some problems in my laptop with help from people at freenode. When I had to do
[code]
su

[/code]
in the laptop, I typed in the wrong keyboard and my root password (which was the same for the desktop and the latop, for the last time ever) went directly to the channel.

296 Jared July 30, 2010 at 4:54 pm

Long ago I learned to always lift my finger completely from the shift key immediately after typing the ‘*’ character.

Here’s what I meant to type: rm -f *.txt

Here’s what happens it you type the ‘.’ without ensuring you have completely lifted your other finger from the shift key:

rm -f *>txt

You get a directory with only one file in it,called ‘txt’, with 0 bytes.

That was a painful (never to be repeated) lesson.

297 Andy July 30, 2010 at 10:46 pm

Thanks god, the earth is still running…

298 Bruce K. August 4, 2010 at 5:52 pm

This is really pitiful … such terrible work habits and carelessness. Certainly no one is perfect, but all one has to do is be deliberate and look carefully at what they type instead of getting in a mad rush that will set you back hours or days.

One thing I have done is to create a super-prompt on root account:

>>> [515] user@machine 2010-08-03 13:09:30 [515]
>>> /home/user
>>> $The “>>>” is not part of the prompt of course. This tells you exactly where you are, what command you are in, what time you did your last command. Next … when you want to know what files you are going to operate on such as “*”. instead of doing the command “rm *” … do “echo *” to see the file list. Then backtrack in history and edit the command. One thing that has bitten me before is getting too fast on history editing … especially in multiple windows. If you use the same history file for every login you run the risk of thinking you are repeating a command when you may be repeating a command that was typed in another window and added to the history file. The way around that is to create separate history files for every login in your init scripts. You must manually delete the old history files at some point, but sometimes it is useful to be able to search for a command you used in the past at some point. All the rest of it is question of being here now … not going so fast you do not have time to think and see what you are doing. Computers are unforgiving. Reply 299 Marcus August 6, 2010 at 3:57 pm While root I typed: kill 1 instead of: kill %1 The screen went very blank, and then there was a lot of yelling from users on other terminals… Reply 300 Jack August 12, 2010 at 5:11 pm Not an Unix mistake but I’ve deleted the wrong LUN from Storage side, the redo log from Oracle, It scared the crap out of me. Luckily the dba managed to rebuild redo logs. Reply 301 Evan Richardson August 14, 2010 at 7:10 am Not a unix command, but thought I’d share mine. I was RDC’d into a remote box, and after having updated some info on it’s host dhcp server, I wanted the machine to pull down new changes. Instead of typing ipconfig /renew, I typed “Ipconfig /release” needless to say, I had to go to the remote location and type ipconfig /renew myself =( Reply 302 Jimboooo August 15, 2010 at 9:29 pm Removing some core files from a host… # find / -local -type f -name ‘core*’ -exec rm -f {} \; # Phone rings – DBA complaining about his database not working. “Which database?”, I ask. “Core”, he replies. Reply 303 Mr Z August 20, 2010 at 2:12 pm This one has had me laughing for days now. I’ve been pretty careful for many years now. Worst I’ve managed was somewhat trivial: Pulled the cover from a running Sun E-250 to see why a fan alarm was going. The interlock switch helped me to get down time to change the fans. When I first started using Ubuntu, playing with it a bit I decided that I didn’t want to learn python, so removed it. If you’re not sure why that’s bad, try it :-) Reply 304 Anonymous for safety August 19, 2010 at 4:23 pm I managed to get physically assaulted (kidding, was just a a pretty hard slap on the back of the head), joked at for 6 months, and get a to-this-day epic dressing down from the boss in front of all my coworkers for a very simple, and idiotic mistake. Had an SSH terminal for our DNS / DHCP server, which also doubled as a mailserver. I opened another tab inside the terminal window, to another server I was gonna install (it was supposed to be the new mail server). As a joke, I called a coworker nearby, and said “watch this!”. To his horror, I proceeded to rm -rf / in front of him, apparently on the production server. (you can see where this is going) He went white. I laughed and said “heheh gotcha. that was the old server. its now all ready to reinstall!”. He goes, “nooo, it was the production server, are you MAD?” – while turning from white to red. After a couple of “no it wasnt; yes it was”, I looked back at the screen. Sure enough.. I typed the rm-rf into the wrong tab. Icing on the cake? The work order was install a backup system on the dns/dhcp server, and migrate the mailserver to a new machine. Obviously there were no backups. Punishment: having all my coleagues leave early for the day, with instructions to me of “you will get out of here when the everything is back up!”. Was a loooong weekend. According to the boss, I wasnt fired on the spot only because “well, at least we know YOU will never ever do another rm-rf withought thinking twice or thrice…”. Reply 305 brux August 19, 2010 at 9:53 pm 231 Anonymous for safety …. Sorry to pick on you, and don’t mean to, but this is one really big source of mistakes on computers, people getting emotionally involved. Like playing around … you were thinking more about other people and making a joke than what you were doing, so you were and will be bound to always make mistakes like this. Like playing around and not realizing what window the focus is in, or that your ssh has time out and you are now back on the original machine, or whatever. With computers you have to think very carefully about what you are doing, and then look at it, and even test it if you can before you run a command that does something complicated, or any kind of “write” operation. You also have to think about the ease of recovering the data should something go wrong. Before we did upgrades on user’s machine we used to just naturally assume users were lying about have local data on their machines, so we would do a backup image to a admin server just in case. That saved a ton of data from people who sometimes did not really understand the difference, or were not totally thinking. I used to get kidded about my seriousness and the fact that when we did common operations, upgrades, etc, I would look at a command line and ask everyone there if it was OK. I got neverending shit about that, until a few times when people did not do that we lost customer data. We were doing an upgrade once and one hot shot admin was ready to hit the return key to start it. I asked him if there were backups and he said he did not know, but nothing is going to go wrong. I told him to make sure and backup the machine, and of course, you know what happened, becuase he got his emotions, his arrogance involved in it and could not stand to be questioned. When I am hiring I try to look as best I can for this trait, because it is the number one problem with a good admin … that and just plain crookedness or dishonesty. Why are their so many anti-social sys-admins? ;-) Reply 306 Marcus August 20, 2010 at 2:01 pm No offense, Brux, but isn’t this for people sharing their own mistakes? He clearly knows he screwed up and how, or he wouldn’t be here. We already know you’re good because you’re on a site that’s about improving your craft, but do you have a mistake of your own you can share for our education and entertainment? Mine is at #228, fyi. Reply 307 brux August 19, 2010 at 9:57 pm by the way … i really love the individual icons that this site creates for its commentors … can someone please, please, please, email me and tell me what that is … it’s really cool and I’d like to use something like that myself …. please!!!! very very cool! Reply 308 Shehzad August 20, 2010 at 6:14 am BIG mount blunder: Very old existing mount was: /mnt/mountpoint1 /mnt/mountpoint2 After long days, once I was experimenting with SSDs and mistakenly mounted it on /mnt A script scheduled in cron using mountpoint1 and mountpoint2 broke and take me many hours to fix!!! Reply 309 Sam Watkins August 23, 2010 at 1:21 am One time while I was somewhat mentally ill (no excuse really!) I was using my father’s computer with Mac OS X. I had created some files at various places on the system, and I was thinking I want to remove all the files I created. So I thought, well I can just run rm -rf /, that should delete just the files which I have permission to delete, and fail for others. Not such a good idea, since he had an old filesystem mounted 777 without proper owner or permissions! Fortunately I did stop the rm process before it trashed everything. Luckily he did have backups of those items early in the alphabet, and didn’t lose anything important. Reply 310 Meteor Jenkins August 24, 2010 at 7:03 am I had booted a fully working Windows XP box with my rescue USB stick, to show off to my friend, showing him various command-line stuff. Then – as a regular user – I typed:$ dd if=/dev/urandom of=/dev/sda

Telling him that regular users don’t have permission to alter the hard drive directly, I hit return.

Uh-oh. No permission denied message. Hitting Ctrl+C, I realise my friend has just lost his partition table and Windows installation. Luckily, the data partition was separate and recoverable by TestDisk.

I had added my user to the ‘disk’ groups months ago without realising it.

311 glued August 24, 2010 at 7:08 am

Not as awesome as your stories, but paraphrasing my own coding mistake:

[i_am_doomed]
$while read LINES > epic_fail_command_omgwtfbbq_why_did_you_put_me_here_stoopid_glued > do > ……. > alerting_script${LINES}
> done < ${file} *goes to lunch* Hilarity, 100,000 alert emails, tech calls, and a low yearly appraisal came after. Reply 312 Sam Watkins August 24, 2010 at 7:16 am I love this one :) Reply 313 Wayne August 26, 2010 at 5:06 am Edited /etc/hosts.deny ALL: ALL Edited /etc/hosts.qllow (made a typo) ALL: 10.1. Logged off system to test, couldn’t get back on. Reply 314 digitalsushi August 28, 2010 at 1:30 am function halt (){ echo “really?” read ans if [ "ans" = "yes" ] then /sbin/halt; fi } Reply 315 CP September 4, 2010 at 9:53 pm Poignant post, very sincere, lots of classics here! To add, I wanted to deliberately delete an old 486-based debian install by typing rm -rf /etc (and so on) while the system was running, and I was surprised by how resilient it was and kept itself alive. Maybe it’s better to avoid the mistake than committing and having to learn from it. Reply 316 Wolf Halton September 5, 2010 at 5:18 am I think this thread should be required reading for all students taking a linux administration class. These are classic blunders. Thanks for the OP and thanks for all the comments. Reply 317 Elton Carvalho September 5, 2010 at 3:26 pm Just made this one. Not exactly a command-line mistake, but will do. Got a new hard drive and moved the contents of the / and /home partitions from the old one to the new one. In order to boot from it, I should clone the MBR, so I wouldn’t need to go through the hassle of setting up grub. And there I went: dd if=/dev/sdb of=/dev/sda bs=512 count=1 Note the “512″ instead of “446″. Those 66 extra bytes had the old disk’s partition table and it was written over the new disk’s. Of course I hadn’t backup what I was about to rewrite. Now I’m doing all the copying again and adding “get a LiveCD which supports ext4″ to my to-do list. Reply 318 bart September 6, 2010 at 8:09 pm This one was fun: sudo chmod -R 777 / instead of sudo chomd -R 777 ./ .. Reply 319 bob September 8, 2010 at 6:26 pm First n00b error, that end up discovering a weird bug in RHEL: Typed reboot on the wrong server As soon as I realized the error, I typed - init 5 Runlevel showed 6 5 This locked the computer totally. Couldn’t reboot it anymore, couldn’t use it etc. Deadlock. I had to physically reboot the machine. Second error was a better one. I was working offsite and decided to do some modifications on /etc/passwd using sed. So I made a backup: mv /etc/passwd{,.old} Bad idea: nobody could log anymore, root included, on the machine. We had to boot single user and restore the file for it to work. Reply 320 nikin September 14, 2010 at 9:37 am Yes. i have done some of these to :D Nowdays i have backups off the more important things. offsite every week, and offline every now and then. For work material, like source files and databases on a dayly basis. It did save my ass a couple of times :) Reply 321 Dab September 14, 2010 at 2:21 pm Once a time I copied some text in a technical deployment website to paste to a word document. Then I worked on something else and (I thought) I copied something. when I paste the stuff to my unix session it resulted everything from the deployment page was pasted there… everything was invalud UNIX comamnd except 1 line….. \rm ~ Reply 322 Ameth September 15, 2010 at 10:31 am The only system-crashing thing I have done was to install busybox in what was supposed to become a initramfs, but forgetting to chroot. Nothing of value was lost, though. Also, I once took a backup of a MBR on the same disk it belonged to, before whiping the old one (I don’t clearly remember why, but I think it had something to do with dual booting windows). Not being one to panic, I opened my own MBR in a hex-editor, found some common patterns, booted up the wasted computer with a live CD and made a small C program that searched through the disk for something matching a boot sector, finding a single one some 80 gigabytes out. The whole room applauded (… I wish). Reply 323 Nihal September 18, 2010 at 7:14 am instead of umount /foo/bar i did rm -fr /foo/bar to “remove” the drive :D Reply 324 Dallas September 21, 2010 at 3:05 am Hi, I’m adding my horror stories too. in konqueror, delayed mouse activity.. managed to move /usr/lib somewhere (was logged on as root for some particular reason). reinstall:( manual rpm upgrade of glibc rpm. In the days prior to yum/smart etc. issue was The update was i386 file, not i686 as installed. caused the system to fuad! used recovery disk :) Reply 325 erik September 23, 2010 at 8:28 am AIX root login by default has no / as its home directory. so: 1. logged as root 2. cp -pr /tmp/root_home /root 3. cd 4. rm -rf * duh! now first thing i do when doing a fresh AIX install is to create /root and make it root’s home directory (via post install script) lots of classic above. fun but painful when it hits you. thanks! Reply 326 jfc September 23, 2010 at 11:53 am i had setup a chroot to test something and mounted proc under this directory. after testing, i did a rm -rf testdir. wondering, why the rm took so long, i saw the mounted proc :( i had to restore the /home directory from backups Reply 327 naikta September 29, 2010 at 3:50 am Once upon time in my earlier day’s in unix I was working on HP-UX box . I wanted to reboot the sap application server but halted it. Fired–> #reboot -hy 0 instead of #reboot -ry 0 No console, no datacenter person available in weekend to hard-boot. Extended 1 hrs activity to 10 hrs. Reply 328 Sparcrypt September 29, 2010 at 11:36 pm Cleared all routes from the machine while accessing it from another subnet. Oops. Good lesson in NEVER typing any command in live unless you know exactly what it does. Reply 329 Linus October 8, 2010 at 11:44 am I wanted to delete ~/etc I was in the Homedir and Typed rm -rf /etc Instead of rm -rf etc Luckily it was just a Test-Server Reply 330 Ashish October 11, 2010 at 7:48 am Guys, Kindly be very careful when you work as a root user. My recent mistake on linux box was as below. I wanted to check the reboot history for the system, and the blunder that i made while issuing the command is: The correct command is: last reboot The command i entered: last | reboot Hence, the system got rebooted. It was a production system, and was a big escalation. Be cautious while issuing commands when working as a root user. Reply 331 Taiko October 15, 2010 at 10:42 am Wow, this thread seems to demonstrate admirably that Unix interfaces are indeed cryptic and dangerous. Reply 332 Michael Shigorin January 9, 2011 at 11:30 pm It’s habits, lack of experience, ignorance, showing off that hurt. Interfaces are important but second to all of that (reboot could check if stdin is a tty, but then again an ancient one wouldn’t and one might be less careful expecting it to be somewhat fingerproof). Reply 333 Marcel S Henselin October 16, 2010 at 9:38 am Hi folks I set up a linux machine without monitor to react on ctrl-alt-del to poweroff /etc/inittab: ca::ctrlaltdel:/sbin/shutdown -h -t 4 now unfortunately I had TWO keybords for TWO servers and no KVM switch – guess what - I wanted to log on to a Windows 2000 Server – pressed ctrl-alt-del and heard the beep from the linux machine – just while compiling a newly set up kernel. since then I use KVM switches most of the time :) Marcel Reply 334 Christos October 20, 2010 at 11:23 pm Locked out due to firewall reconfiguration. Done that. Quite some times. I nevel did liked firewall and firewall don’t like mw either. rm -fr to the wrong path. Not really, but there was a case on a an old netra machine, were the filesystem was corrupted and when i issued ls in /tmp i could see the whole / filesystem underneath it. “just some crazy inode mixed-up resulting in ghost entries” i thought. Luckily there was a recent backup around :) And the most embarrassing mistake thus far, I once was in a telco just before it went online. and one of the sun clusters there, had some strange network issues. “The problem lies in the arp cache, one network guy suggests. After clearing some arps and still no avail, i decide to clear everything from the arp cache! There goes the cluster interconnection, and i ‘m experiencing my first cluster split brain. One node immediately panics, while the other one is ..well no at it’s best. I rebooted the whole cluster. Fortunately it only took about an 30 minutes and resolved that network issue, which up until now remain a mystery. Reply 335 JohnK October 21, 2010 at 7:30 pm At 2am typed init 0 instead of init 6 – doing scheduled upgrade including kernel. No one in the building had access to the server room. Had to drive 90 minutes into work to simply push the power button… In my first sys admin job I did not quite grasp the concept of a dumb terminal…I wanted to reset the terminal so I typed reboot while logged in as root…The terminal did not reset but the server did… Reply 336 khh October 22, 2010 at 10:04 pm My worst one was while I was working on the build/compile system of a program. The ./bin/ directory was filled with files, but I wanted to see what files the build system itself copied there. But instead of rm -rf ./bin/* I typed rm -rf /bin/* And as my luck would have it, I’d been doing some work requiring the root earlier and forgotten to exit. Had to reinstall the operating system, but at least I was able to backup all files and settings. Another good one was su -c “passwd” and pasting the password. I don’t know what the I pasted, but it sure wasn’t the password I had in mind. Other than that I’ve done the “sudo ipconfig eth0 down” and the reboot on the wrong box. Once I wanted to play a prank on my friend and sent him an “:(){ :|:& };:”. I should have done that, it ended up causing him a lot of problems. Reply 337 danO October 28, 2010 at 3:46 am thanx for jinxing me. did a rm -rf tragedy today on project files 2 days after reading this post. Reply 338 gormux November 2, 2010 at 10:49 am Once, I wanted to reset my FreeBSD system to a clean state. Usually you just have to rm -rf /usr/local Unfortunately, this time, I did a rm -rf /usr/lib Damn tab-completion… Reply 339 Pawel November 9, 2010 at 10:44 pm Few years ago: UPDATE email_accounts SET vacation =’some text’; instead of: UPDATE email_accounts SET vacation =’some text’ where id = ‘some_id’; About 3000 accounts where changed. Hopefully, we’ve had backups. Reply 340 Andy T November 14, 2010 at 7:31 pm Aaaaaaaaaaaaaaaaaaarrrrgh! Just hosed my main dev instance’s /usr/include dir There’s a very good reason this was not a stupid mistake, but when I told it to my dog, he got up and went into another room so it clearly is not very convincing… OK, it could have been worse – but since it should be recoverable, I thought I’d post the fix: I remember thinking it was odd that with such a long list of mistakes, no-one had ever posted the quickest fix for their problem – I may be about to find out… OK, so I’ve never put anything in /usr/include manually (I’d use /usr/local/include for that) so rpm or yum should be able to rescue me (dpkg or apt for Debian). I’m assuming that only packages with ‘-devel’ in the name will deposit files in /usr/include. (-dev for debian) I’m in a rush so I: rpm -qa | grep devel | cut -d ‘-’ -f 1,2,3 then mess around until I have a list of names ending in -devel which yum will accept. I then create a new xen instance of the same linux flavor. Making sure my hosed xen dev instance and the new xen instance are updated to today’s release, I issue ‘yum install (package)’ for each of the identified packages on the new ‘box’. I then cross my fingers and scp everything from the new instance’s /usr/include to the hosed box. I’m going to do this tomorrow, not now. If you see another post, that means it didn’t work… Reply 341 Hollygirl November 19, 2010 at 2:07 am On a Solaris Box running production cluster Oracle databases – trying to get date, time and other misc output (after typing many misc commands in a row and not paying attention anymore…) typed: > hostname -a opps – changed hostname to ‘-a’ followed shortly by P1 Oracle database crash. Reply 342 Hollygirl November 19, 2010 at 2:11 am A very common mishap *(one of my power-users did this – took about 3 hours of my time to fix): ftp the /etc/password file from a Unix box to a Windows box so it can be edited. Edit the /etc/password file in notepad or wordpad (which likes to add line-breaks etc and other formatting). Save it and ftp the file back to the Unix box. Viola – noone can log into the box and noone can su to root. There is no way to fix this but bring down the box, go into maintenance mode and edit the /etc/passwd file manually (remove all the blank end of line characters from each line of the file). Reply 343 pepoluan November 24, 2010 at 11:20 am Hahaha… wonderfully instructing “OMG” moments there, thanks for sharing :) Hmmm… let’s see, what mistakes I have did… Locking myself out from an SSH session … check rm -rf * while on /etc … check > instead of >> … check rm /etc/sv1.* . instead of cp /etc/sv1.* . … check (The last line was me copying some custom config scripts; I usually prefix them with the server name so I can find them quickly. The last line has the added bonus of removing files I’m still working on in my ~ . *sigh*) Nowadays, I always install my Ubuntu inside XenServer, and religiously create snapshots before thinking of doing drastic things :) Reply 344 Derrick November 29, 2010 at 4:55 am My first major n00b blunder: Wanting to move the files and folders of a blog up one directory, I typed: mv /* ../ Notice the missing period, should have been: mv ./* ../ VPS backup on production server:$5/month.
Lesson learned: priceless

345 DB Man December 1, 2010 at 3:52 pm

I was working at a company which had a home made ticket solution, the database used mysql4 (MyISAM, AUTOCOMMIT=1) and a flaw in the design, a column which just included integers was created as varchar(10) and I was to update a phone number for a customer in the database:

UPDATE table SET phone=’1234567′ WHERE ticketid = 123;

This resulted in that all customers got the same phone number, it wasn’t trivial to get the phone numbers back, we managed to get most of them from a backup and the rest from Apache logs.

346 Ali December 11, 2010 at 6:29 pm

I was working on a linux box, and besides other things, just wanted to review my scheduled tasks; typed crontab -r (instead of -e) and all my tasks wiped out.

347 Derek December 11, 2010 at 11:42 pm

crontab -r is the worst :( no warning. I’ve made it a habit to always type crontab -l before ever typing crontab anything else :)

348 Karl December 14, 2010 at 3:23 pm

I issued a “shutdown -h” on a remote system when I meant to use “shutdown -r”
It’s a bit embarrassing to have to phone your hosting provider to turn it back on :P

349 Ian G December 16, 2010 at 12:15 am

On my first day as an admin, I deleted emacs. Fortunately my co-worker was able to restore it.

Another time, I mistyped a sql command, and reset all user’s scores on a production machine. Luckily, we had a backup… but that didn’t prevent users from complaining that they had lost points!

350 Brian December 16, 2010 at 3:55 pm

I went to backup passwd group and shadow because we were setting up a new server and needed to copy over the users and when i ran the tar command i did this
tar -cf /etc/passwd /etc/group /etc/shadow user_backup.tar

luckily passwd had been backed up into passwd-, I phoned the datacenter and had to get somebody to boot to single user mode and replace the file….

351 aussierob December 21, 2010 at 2:20 am

I keep several clients’ data on my testing server, testing data in /aaadata.
I keep customer1 data in /cu1data and customer2 in /cu2data etc.
when I want to swap test sets of data.

One day I confused being remotely logged onto customer1 live server one day…
and then the mv /cu2data /aaadata which (thankfully) did not work.

Luckily I quickly figured out where I was,
did a mv /cu1data /aaadata to put it all back,
kicked everyone off, rebooted and got away with it !
Close one !
(I sitll df -v before these moves – a little paranoia is good for you )

352 Qasim December 25, 2010 at 5:56 pm

… Scary!!!

353 Alex J Avriette January 7, 2011 at 9:20 pm

A lot of your mistakes are pretty tame compared to some I’ve seen in the wild and some I’ve committed myself.

When I was really young (this was my first Unix machine, a Sparc 2; I was maybe fifteen or sixteen…) and not only didn’t understand Unix permissions and was very frustrated by them. I figured that the easiest way to make sure I could access everything I needed to was to say

chmod -R a+rwx /


I was root, on what had to be Solaris 2.4 or 2.5, and let the command run for a while before thinking, hm, maybe this is a bad idea. The original “logic” was, “well, since I’m the only person using this machine, why shouldn’t I have permissions to read and write everything?” — I completely failed to understand what the execute bit did, for starters.

Permissions were so hozed there was no option left but to reinstall the machine. Learned that mistake but good.

354 Michael Shigorin January 9, 2011 at 5:16 pm

Using *the* simplest command which can make a backup one knows *is* key, especially if tired already (it’s antagonist to being responsible).

I’ve hosed a production backup server’s disks (what an irony) quite recently due to:
- we installed on a single HDD of two as they still await for a hardware RAID controller
- I did mess half a year ago trying to “at least make a backup” (dd’ing sda to sdb)
- I did pay attention to replacing UUID-based mounts by device-based
- I did look into /etc/fstab and mount before proceeding with dd (all clear)
- I did *not* look into /proc/mounts (where it was a rootfs UUID-mounted from sdb)
- I did *not* perform a most basic off-host backup I easily could, with rsync

What was still on the positive side, the tapes weren’t harmed by that dd (20 gigs in when it finally struck me), and there was somewhat older snapshot of /etc/bacula, and while ls would segfault already (had to use echo *) some other tools still worked off the damaged rootfs and so some more parts were salvaged but it was a reinstall and waste of time, even if not hitting users heavily.

And yes, I did off-host and off-site backups after reinstalling on software RAID. Working/commuted iKVM did help immensely either.

A habit of running a primitive local snapshot after considerable changes still holds:
# tar zcf /root/BAK/etc-hostname -s-date +%Y%m%d.tar.gz /etc

355 Michael Shigorin January 9, 2011 at 5:32 pm

> dd if=/dev/urandom of=/dev/withManyData count=1024 bs=1024
> i forget it….i the last command i haven’t way to rollback only reinstalling
Not exactly: DO NOT REBOOT while you still have the kernel which mounted the filesystems while all information to do that was available.

STEP AWAY from console, have some tea, sit down and try to calm yourself.

Then estimate the consequences of losing the info and if it’s valuable, consider the possibilities to save it.

Second HDD is OK if it’s already mounted, otherwise consider networked backup (rsync/scp). USB HDD/flash might not work already if you had the luck of damaging kernel module file contents which would be needed to use them.

Then backup up /etc, /home, /var or whatever might still be readable.

Then have some more tea — unless the downtime is really pressing on you or you have 99% solid backups. Maybe you’ll remember some more stuff.

Only then say goodbye to that filesystem and reboot, afterwards only salvaging tools might be able to help (testdisk is easy-to-use but limited, gpart is a PITA but did help me to recover partitions after installing a distro on workstation disk not the test one; photorec reportedly helps with salvaging files from damaged filesystem).

356 Michael Shigorin January 9, 2011 at 10:53 pm

Aside from firewall/sudo lockouts, chown -R smth .* (back when it did go from /home/smth to /home, which was a minor disaster), and aforementioned “dd backup server down” case (BTW I did ask a colleague to advise/witness me then took a cup of tea then mailed those concerned in half an hour when downtime and reinstall was considered inevitable), I could recall these…

One of my first Linux systems (a RHL5.x, libc5) was hosed by very much wanting to upgrade xmms-0.7 to xmms-0.9 from a “pirate” “Red Hat Linux 6.01″ (actually 5.9, a 6.0 beta) CD where it was linked against glibc2. Well, rpm did try to stop me. And I thought to cheat it around. After rpm -Uvh –force glibc*rpm finished what it was ordered, even ls would blow up. It would be only years later that I would know how to recover from that situation (like, boot the 5.x installer — there were no livecds yet it seems — mount that filesystem, copy glibc rpms there, rpm2cpio | cpio -id at least the /lib/libc.so and VIP friends, and then try booting off that root to rpm -Uvh –oldpackage that glibc*rpm — or keeping rpm-static at hand in the first place). Well, at least glibc2 is actually nice regarding backwards compatibility.

Several years later there was a Very Important Dump sitting in /tmp with stmpclean set up to shoot down month-old crap there; the next day was a moment to remember. My fault for placing it there in the first place but had also to talk with folks who set it up so in packages for the distribution installed.

On the same job, we once had to take development environment into production real fast — and chose to continue running backend on an office server where it was already deployed. Then we had tough time moving it to production servers at colocation, and then summer came with increased power consumption for downtown’s conditioning systems. One day we were running on all the office UPSes with two-minute downtimes to change them (no second PSU to juggle cables, and we didn’t chain up UPSes for the reason I don’t remember already)… the decision to move production services to production environment became a bit more obvious ;-)

On a community-built LUG server I was relying on a donated DAC960 controller to handle a separate SCSI HDD holding the root filesystem for an FTP server for some time — until it began to sneeze on a virtually new 18G drive and fire a single physical volume from its single logical volume… since virtual environments (linux-vserver back in 2004, openvz by now) lived on ATA/SATA drives (there was a mix back then), I would end up with perfectly working but unmanageable server until that half-a-meter board was retired to a stand. I thought it’d be cool to run IDE+SATA+SCSI, in fact it wasn’t — and “coolness” is a bad factor to account for.

Didn’t reboot/halt the wrong system so far — a friend of mine did, and he told us the short story to remember (thanks Nick!). Didn’t get caught with crontab -e/-r so far either, as well as prepositioning / when it wasn’t meant to. Well, lots of wisdom read, thanks.

Bonus tracks:

A colleague was experimenting with FreeBSD 4.x softraid (vinum) on a production server with some 4 HDDs and found it the hard way that after hitting some pretty low limit of volumes the kernel just froze up. He didn’t expect that in all honesty but we were down for that night.

Another anecdote was of a junior who decided to test how the hardware RAID5 works and pulled a drive from production server, then pushed it back and pulled anoter one. The very same moment the array was ruined — poor guy apparently didn’t understand that it takes time to rebuild an already degraded array after pushing first drive back, and that he was unneccessarily risking double fault even if he didn’t pull the second drive (if one of the remaining spindles would be close to fault it might not bear the added load of rebuild and go down bringing the whole array with it, again).

PS: yes, I did notice the difference in stories where there was a backup (“phew!”) and when there wasn’t. Taking care to review and test backups, *especially* after reworking the storage scheme (e.g. Bacula won’t descend into mounted filesystems by default, and moving data to a separate filesystem might also prevent it from being backed up), is really worth the trouble. Backups are sysadmin’s children: he tends for them, then they tend for him…;

357 James Alvarez January 12, 2011 at 5:50 pm

When managing multiple SSH console, my always mistake was to type into the wrong SSH console, which is completely insane. :)

358 /activate brainless actions January 17, 2011 at 11:03 pm

Well, I have Used Ubuntu for a few years and mine (among many) was I was trying To add a repository for my anti-virus and I added the repository for all of the Debian volatile project. It was a huge update and then nothing when i restarted.

359 W8Hosting January 19, 2011 at 1:13 pm

Command on wrong server: done

I would rebuild my backup server that at that time wasn’t fully working. I didn’t realized that I was logged in on the main cPanel server. Result: Angry customers and some downtime.

360 Ulric Eriksson January 24, 2011 at 10:19 pm

My most remarkable command line mistake happened sometime in the eighties. I didn’t know there was this thing called “Usenet”. Trying to delete some file, I accidentally typed rn instead of rm. That mistake cost me countless hours over the next ten years or so.

361 AlexUnix March 4, 2011 at 2:31 pm

Nice way of praising the Internet! :))
rm is not even so bad after all! :P

362 Aura January 26, 2011 at 4:37 pm

I was going to wipe a little USB-stick and did dd if=/dev/urandom of=/dev/sde on the production server. Ops!

Lesson learned: Use very obvious, colorful, different prompts for each system. At the time I had the same .bashrc on multiple systems.

Another mistake I did was to add “exit 0″ in some function in my .bashrc, when I really wanted a “return 0″. It took me some time to realize why I was instantly kicked off everytime I was logging in via SSH. Haha!

363 Aaron Samuel February 9, 2011 at 8:41 am

On Debian

rainofkayos@karma ~ [4398/255] % hostname -f
karma.lan

On Solaris:

0@solaris-rain:~[root@solaris-rain /]# hostname -f
0@solaris-rain:~[root@solaris-rain /]# hostname
-f

Result?

Wasn’t bad service wise , was a weblogic instance running in a massive cluster of about 10 instances. The members just dropped it from the cluster when the hostname changed to -f. I noticed immediately.. lol naturally because hostname returned nothing,, it was a quick puzzled feeling, then one of the “oh crap” feelings, =). Luckily that client (no names) had the shell prompt configured with hostname, and I could easily see what i needed to change it back to by looking at the previous commands I had run.

Embarassing wise, super, about 20 minutes after I had “cleaned up the issue” , I thought it just looked like a blip and no one had noticed. Which in general was true, the client hadn’t noticed it at all, however a senior of mine and some what mentor, walked over to me, pulled over (rolled over) an office chair, sat down, looked left, looked right, ensured we weren’t on attention lane, and said in a strangely nice yet taunting way, “so… (long pause) dont always run commands as root… (another long pause) … and dont do hostname -f on solaris.” He then got up, and walked off back to his office.

LMAO,,, _fail_…

What did i learn?

Ofcourse one would think the point is dont run hostname -f on solaris. But actually I learned from this and from the years, dont abuse root or super user access. Take advantage of proper permission usage.

Cheers~!
Aaron.

364 jimmy jam February 9, 2011 at 4:21 pm

My best mistake while learning Linux at work was accidently deleting a config file in sites-enabled using winscp! Straight away the clients website was unavailable and I had to put my thinking cap to good use. So I managed to copy the contents of another file and reconfigure. It was a tough learning stage, the power of Root should never be underestimated!

365 Tof February 9, 2011 at 6:15 pm

Hi all,
So funny ! My vote goes to:
- typing in the wrong console (i rebooted the main oracle DB production server once … all the operators where coming to tell me there was a problem with the DB … oups .

My second favorit : the rm -fr /stuff * (with the space before *) …
Even been carefull after the first time i made it about 8 years ago … i reach to do it 3 more times … in production environement! Sure your happy to have some backups then !

366 Dan Sichel February 10, 2011 at 12:01 am

keep config files in CVS. Nice. Simple and useful as H*LL now and then. Thank you. Chuckled at the DNS zone file overwrite. What a difference a greater than sigh makes!

367 Nimda February 11, 2011 at 12:04 pm

The reboot command on solaris has no time factor option, but any followed text is interpreted as the kernel that will be loaded after reboot. So without knowing it…

My Mistake on a Solaris box, dont ask me why but i typed…:
#reboot now

“Cannot boot now”

oh well …. :)

368 Muppet February 17, 2011 at 3:33 pm

Shutdown ssh daemon instead of restarting it apply a config change… remotely.

369 giggity February 18, 2011 at 3:37 pm

Got into work one morning..
Logged into the main production server (at another site), and found a new directory:

/aaaaaaaaaaaarghhh_dont_delete_me

“Oh, you found it then!”

Apparently, he’s been removing a user from the system the previous evening and did a:

rm -rf / home/username (yes, with a space between / and home)
Fortunately, since we were using amanda, there was a /amanda directory full of backup related files, which gave him a few seconds of Cntrl-C time :-)

Hence the new /aaaaaaaaaaaarghhh_dont_delete_me directory, full of small, random files, just to provide a few more seconds of Cntrl-C time should anyone repeat the command!

Oh, also did a ‘rm -rf /’ (knowingly!) on an old SunOS4 server that was being shipped offsite, we were rather disappointed to find that it thrashed it’s disks for half an hour before just returning to the ‘#’ prompt. I think the only useful thing we could do with it afterwards was ‘echo *’ :-)

370 Anders February 23, 2011 at 10:57 am

I once did rm -rf .*/ instead of rm -rf ./*

Luckily I was doing this as a non-privileged user, and only lost some of my (backed up)
personal files.

371 Dennis February 24, 2011 at 1:49 pm

Played with some commands a long time ago on one of my test systems.

# cd / ; chattr +R *

forgot all about it and then finally rebooted the system a few weeks (months?) later…
…BIG MISTAKE…….

372 Deltaray February 24, 2011 at 7:01 pm

Dennis, what does chattr +R do? R is not one of the attributes listed in the man page.

373 cybernijntje February 26, 2011 at 1:46 pm

Correct syntax is: chattr +i -R *

*sigh*

374 BBHoss February 27, 2011 at 5:57 pm

CVS is horrible for configuration. Use git instead. Also for your QT PROJECT!!!

375 George February 27, 2011 at 9:34 pm

AFAIR, the “Unix Haters Handbook” has one from alt.risks where the user had typed:
rm -rf *>o (fat fingering the SHIFT key and getting > instead of .

Instead of deleting *.o, everything is deleted and you get a zero length file called “o”.

376 Graham February 28, 2011 at 8:19 am

My main mistake was being super user and issuing rm -r * from the root directory, instead of a temp directory I wanted to get rid of. Unix does not run very well with no files at all.

377 pepoluan February 28, 2011 at 8:56 am

That’s just happened to me last week! Granted it was not really my fault ;-)

At the staging area (where we prepare boxes) we have no KVM switch. Due to a stack of new hard disks, the keyboard needs to be placed one in front of another (instead of right in front of each monitor). I had earlier moved the Windows box’s keyboard in front. Without my knowledge, when I was out my colleague switched the keyboard to do some Linux stuffs. I went back, and tried to log on to Windows. A yell from behind me confirmed that I’ve done something terrible…

… lucky it was still being installed, so nothing important is gone. The inconvenience of having to reinstall was promptly forgotten when I treat her to a nice dinner ;-)

378 Tom May 2, 2011 at 10:59 pm

KVM switch is danger thing. I had connected 2 x Linux and 2 x Windows to one KVM. The screen was blank.
Just hitted CTRL+ALT+DEL becouse wanted to login to Windows machine…
… caused immediate reboot of main Linux router. Fortunately, it tooks only 2 minutes to boot up.

379 Rick February 28, 2011 at 1:20 pm

If there is any reason to worry about what I’m doing I test commands like rm by substituting ls first.

Reason to worry? You’re root, that’s reason enough to be extra careful.

380 Mike March 7, 2011 at 11:36 pm

Forgetting sudo on the first run has kept me from doing some really stupid stuff, like removing my /etc folder or similar. Damn was I glad to see that “Permission denied” after rereading the command and noticing my (almost) fatal mistake.

381 z March 9, 2011 at 7:14 pm

# crle -l /appdir/lib

This is effectively replacing (not appending) the application library path to the system library path configuration. (Solaris)

Care to guess how many system utilities this mucks up? This sent me looking for a miniroot boot image as the errant change survived even after hitting the big red switch.

382 shola benjamin March 14, 2011 at 11:37 pm

how can i get unix operating system for my pc, please try email me

i really appreciate this forum

regards

383 Christos March 15, 2011 at 6:43 pm

I’d recommend Solaris 10 for x86 (or x64)

384 Cody March 22, 2011 at 1:05 pm

Excellent!

This reminds me of when I told a friend a way to auto-log out on login (many ways but this would be more obscure). He then told someone who was “annoying” him to try it on his shell. End result was this person was furious. Quite so. And although I don’t find it so funny now (keyword not as – I still think it’s amusing), I found it hilarious then (hey, was young and obnoxious as can be!).

The command, for what its worth :

echo “PS1=kill -9 0” >> ~/.bash_profile

Yes, that’s setting the prompt to run the command : kill -9 0 upon sourcing of ~/.bash_profile which means kill that shell. Bad idea!

I don’t even remember what inspired me to think of that command as this was years and years ago. However, it does bring up an important point :

Word of the wise : if you do not know what a command does, don’t run it! Amazing how many fail that one…

385 Cody March 22, 2011 at 1:13 pm

Hmm. Note to self : be careful which one you reply to (noscript might have had a play in this). To the op – if you would delete this, I’d appreciate it as i put it in the right section now (and delete this too, obviously).

386 PrometheeFeu March 24, 2011 at 3:54 am

Here is my favorite:

I was tasked with compiling a list of certain functions in our source code. I was slowly refining the list removing stuff. Then, I typed:

$grep “pattern” list_file > list_file You might know it means it was all gone. Reply 387 Nate March 24, 2011 at 9:32 pm my developers love doing this as root: cd (non-existent path) chown -R root:root chmod -R 777  instead of chown -R root:root (non-existent path) [so it fails] Reply 388 kalyani March 29, 2011 at 12:26 am sir am kalyan, i have i problem, am creating folders in home directry but they r not visible in unix terminal wen i enter cmnd cd /home ls no files were opened wts the prob…. Reply 389 Aaron March 29, 2011 at 4:57 pm who owns the folders and who is trying to view them? Dont create everything as root and if you do, understand that you may need to change ownership or change permissons on the files. /home on linux is the location of$USER home directory. It should be owned by the user that needs to use/view/modify it, actually IT should be the home directory of the user.

I.E.

user name aaron

/home/aaron

ownerships by default atleast on a debian system would be

aaron:aaron

however lets say you make aaron under home and the perms are

root:root

aaron will not be able to do anything with this folder when he is logged in, furthermore, he will get errors on login saying he has no home.

390 Michael Shigorin March 29, 2011 at 6:03 pm

Hey, now that was a tip to get into this list with yet another story ;-)

/home holds homedirs, but usually isn’t any user’s homedir itself. Tampering with its permissions might feel very wrong with other users.

2 kalyani: do “cd; ls” — plain “cd” is equivalent to “cd ~” or “cd $HOME”, and this isn’t a place to ask (or give) advice as I was already kindly pointed at — head over to linuxquestions.org or whatever forum suits you better. See also e.g. http://www.tuxfiles.org/linuxhelp/cli.html (googled up as “linux command prompt howto”). PS: anti-offtopic: my latest colocation visit (after IP-KVM session but no remote boot media available) was due to hda->sda issue (2.6.18 would still use legacy IDE driver for an IDE CF root device, while 2.6.32 went libata); I did it “the smart way” and replaced “boot=/dev/hda” value with whatever UUID “blkid /dev/sda1″ returned earlier while modifying /etc/fstab for old-or-new kernel setup. The thing is, one doesn’t really want a bootloader in _partition_ when MBR isn’t prepared to boot off that partition. And I managed to get that MBR LILO to L 99 99 state when the backout entry wouldn’t be of any use… Reply 391 cayey April 3, 2011 at 7:01 am is is possible that “shutdown immediate” in oracle automatically runs? Reply 392 aaron April 4, 2011 at 8:46 pm Don’t elieve so other than some script external to oracle. You Cal also check the logs as oracle would log shutdowns. Reply 393 Paul-Willy Jean April 6, 2011 at 8:22 pm Best one yet, I had setted up a firewall at with ipfire the informatic club of my CEGEP and found out that my dhcp server would end it’s leases every 2 hours. I then decided to set the max lease time to 0, thinking that it would remove any limits. It didn’t take long to find out that it was releasing new ip adresses to the computers on the local network on every second, rendering it inaccessible. I had to go directly on the machine to change it. Reply 394 m47730 April 7, 2011 at 4:20 pm One of my worst mistake: i’d like to delete all hidden directories… rm -fr .* but “..” and “.” are counted in the glob! argh! m47730 Reply 395 Tom April 13, 2011 at 10:44 pm The worst thing I ever made caused a disaster. There was one, single SATA in the server for few users, where I was thinking it was proffesional SCSI (same name: sda, 1st mistake). It was containing Samba with roaming profiles on reiserfs partition. One day one user asked is it possible to recover deleted file. Answer was: maybe. I found command to rebuild reiserfs file structure, which can help recover data, something like this: reiserfsck –rebuild-tree -S -l /root/recovery.log /dev/hda3 where /dev/hda3 was /home. Unmounted /home and did it. Unfortunately, there was a BAD SECTOR on the drive (I didn’t know that) which caused command to interrupt. All data went gone, including all raming profile’s data from stations next day. No recovery possible. No backup. Remember: just tell them it’s not possible… and pay attention. Reply 396 Sam April 14, 2011 at 7:01 am The only thing you did wrong was NOT HAVING A BACKUP!!!!!!!!! There are hundreds of ways your data can be lost, and only one way to prevent it: backup. Reply 397 Michael Shigorin April 15, 2011 at 3:48 am Backup and RAID are orthogonal: one will lose data added after the last backup if the only disk (or degraded RAID) goes down or a filesystem gets seriously corrupted. BTW my another similar fault was trying to mess with a damaged filesystem on its native drive (which developed read troubles) and not on a _copy of_ a copy of that block device… that sort of extra duplication very much pays off when you badly wish to get two minutes back in time when data was still relatively close at hand. Reply 398 Michael Shigorin April 14, 2011 at 7:41 am Heh. I’ve stumbled onto somewhat similar problem last year: we’ve got “master” git directory on a mirrored “enterprise class” SATA drives holding reiserfs (one colleague insisted on that) and backed up nightly with Bacula; I’ve moved it onto mirrored SAS drives for performance reasons (and ext4 for sanity reasons). Of course that means a separate filesystem. And it wasn’t until another colleague asked me to restore some file from backup that I’ve realised that git backups effectively stopped the day we did the move (something like a month ago back then). Bacula honestly warned in reports that it didn’t descend in a mounted filesystem, and docs stated clearly that one should explicitly either specify such filesystems, or tell it to cross mount points (and care for /proc and friends on his own). Mind you, it was proper hardware, mirrored disks, scheduled downtime, an extra copy taken just in case — but yet another gotcha and only myself to blame. Lesson learned: don’t only backup, do verify what is extractable. Especially after storage-related changes. Reply 399 Matou April 14, 2011 at 3:51 pm Quite a long time ago, I discovered once the magic of /etc/inittab, in the “default” line. I changed to ‘s’ for “single user mode by default” and did what I had to do, reverted to the “norrmal” mode, typing “m” for “multi-user mode”, then rebooted the server. Since no line matched this strange code ‘m’, it was just starting NO tty. Hopefully, I had a similar server 10 miles away where I could make a bootable tape… Reply 400 Ken April 21, 2011 at 4:43 pm I once ran a Windows server box, didn’t make any mistakes, but Windows took care of corrupting all data anyways. Restored from backup and running websites on a Redhat box now. Reply 401 subhajit April 22, 2011 at 6:01 am how can i configure a local dns server in my home pc where i use ubuntu 10.10? any give the brief funda of naming a server like(127.0.0.3)? why we use this kind of naming system?what r the facilities and why? tell me how i can configure a ip address? n how can i create a lan comnnection with tectniacal terms n eclanation? one more qsn is for any of the above qsn understanding of .NET language is it essential??????n books name where u got that n how u learn that?i want to learn all that procedure????????? Reply 402 Michael Shigorin April 22, 2011 at 1:11 pm As you see this blog post is about mistakes made; yours one is not searching for your own question *before* asking it to anyone who might have already answered many times :-) http://lmgtfy.com/?q=how+can+i+configure+a+local+dns+server I make this one more often than I’d consider ok, so bonus hint is “DNS HOWTO”. Re .NET, I’m happy to _not_ have made a mistake starting with something but LISP. One might enjoy SICP book, see http://mitpress.mit.edu/sicp/ (it also happens to be one of top google results for “sicp”, incidentally). Reply 403 Skydev April 28, 2011 at 7:22 pm After going from Windows to Linux typed pkill /? on a Solaris box trying to get help. Ended up with a system reboot. Reply 404 Lord_Pinhead May 2, 2011 at 10:10 am Hehe, nice ones :D My best kill of an Server was: ./reset_postfix >> /dev/null after destroying /dev/null, i have to reboot the server into an rescue system and create a new /dev/null with mknod: mknod /dev/null c 1 3 chmod 666 /dev/null Reply 405 Iain Kay May 2, 2011 at 4:54 pm Rofl at this one that’s an absolute classic! Destroying /dev/null :P Reply 406 Cody June 3, 2011 at 1:53 pm As is already noted or referred to: writing to /dev/null writes nothing to nothing (essentially that’s the end result). But even then, not only would > imply the file would exist after (even if empty – which /dev/null is anyway), >> would just append instead of overwrite. But this means the file exists. And hey, look at this: # mknod -m666 /dev/null c 1 3 mknod: /dev/null’: File exists Now, if you were to actually rm the device, that’s another story. And in that case, you could just use mknod to set mode. Also, if you have selinux enabled you’ll need to do more or else you’ll get denials. Nice try Pinhead…. Reply 407 Tom May 2, 2011 at 10:35 pm Few years ago, friend of mine accidentaly did another, funny thing. He connected one port from Netgear 24-port industry-entry switch to another port on the same switch. In result, this caused route loop, pushing packets to go over and over in the same switch rendering all network devices inoperate within minutes (!) This also can happend when “smart” switch uses LACP and 2 or more cables to connect to another switch, when it doesn’t have function like “IEEE 802.1D Spanning Tree Protocol (STP): provides redundant links while preventing network loops”, You forgot to save the settings and switch has been restarted… so always remember to click damn SAVE before You get out of the work! Reply 408 Kosta Cavic May 27, 2011 at 7:29 am Tom, Same thing happened to me few years ago, when a tech guy from Telco company connected same cable on the same switch within my client’s VDSL network causing so called “broadcast storm”. He’s explanation was that he intented to made VDSL modems work constantly good by forcing them into broadcast to speed up VDSL network based on analog telephone lines :). When we put firewall device on front of the VDSL network, firewall device was rendered unusable within minute by detecting some kind of DoS attacks and bringing down ethernet ports, both LAN and WAN. Investigation took us a day to find segment (switch) that is causing problems. I know that this post is based upon Unix-Linux mistakes, my senseirly appologies, but it is usefull to know what kind of weird problems can prevent network from operating normal. Kosta Reply 409 Kosta Cavic May 26, 2011 at 1:14 pm After entire night spent at client’s site doing migration of Oracle DB from W2K3 box to RHEL 5.5 box (with Jboss on it), arround 06:00 i’ve just wanted to delete some symlink and ended up doing something like this “rm -rf /etc/* /some_sym_link (notice blank space). I wasn’t sure what i wanted to do with this command because i was to tired. It was production Jboss server with no backup and it should run Oracle too. Luckilly i had still W2K3 box to continue work with no downtime. Within an hour everything was OK on W2K3 box, but still needed to reinstall Linux box with Oracle and Jboss. It happened 3 months ago. Regards, Kosta Reply 410 Kosta Cavic May 27, 2011 at 7:46 am Guys do not force Unix or Linux box to do S.M.A.R.T. checks unless you are 100% sure what you are doing. Few years ago i forced Unix or perhaps Linux box, cannt remember, to do that and it rendered system disk unusable within two weeks. There are conf file which can be used to configure such option but that option should be enabled in BIOS also. Forgot to do that and ended up with problem :). Regards, Kosta Reply 411 Josh May 30, 2011 at 8:13 am damn redirects have gotten me more than once #echo ‘ServerName 127.0.0.1′ > /etc/apache2/apache.conf #echo ‘UUID=xxxxxxxx /data/disk1 ntfs-3g defaults 0 0′ > /etc/fstab I learned my lesson and stopped being lazy, just open the file and edit… Reply 412 Tom May 30, 2011 at 11:16 pm I’m here again. Just lerned something new. Be careful for UUIDs of Your volumes and LVM2 snapshots. Recently happend to me that both: original partiton and its snapshot has been mounted at same location!! /etc/fstab entry was: UUID=123456….. /var/lib/libvirt ext4 defaults …. but also there was entry in /etc/rc.local mount -t ext4 UUID=123456….. /var/lib/libvirt this should just mount same partition twice. But this not happend. First mounted was snapshot, I think. Luckily, the LAST mounted partition was original /var/lib/libvirt and then Libvirtd started. I was wondering long time am I going to loose data after delete of snapshot… no file size increase, becouse of RAW format already filled… but checked modification date and time of files when remounted to separate temporary folders. This happend on Ubuntu Server 10.x in test env. By the way, anyone know why read/write snapshoted LVM volume is REALLY slow? Reply 413 Shahina Shaik June 3, 2011 at 11:32 am I was checking whether is possible to recover files which got deleted using “rm -rf *” command. i was in my home folder and i created a test folder and i created two test files inside the test folder(Actually after creating the test folder i forgot to change the directory, so i was still in home folder). And i issued the command “rm -rf *”. Silly mistake… :) Reply 414 markus June 5, 2011 at 6:08 pm I’ve accidentally run commands on a remote shell instead of local machine more times than I care to admit. Ive learned the hard way to be VERY careful when using chmod -R, and rm -rf. Thankfully its never happened on a production server, although I have annoyed a friend or two who have asked for help setting up servers. Reply 415 hpavc June 10, 2011 at 6:10 pm ./script.here | tee * a coworker did this to me, overwrites everyfile in the cwd with the output of ./script.here. this can also be pretty crazy intensive if you have a crazy amount of logfiles for example and lots of output. Reply 416 D.plomat June 14, 2011 at 5:03 pm stop the wrong server… done. firewall lockdown… done. But the most outstanding i had was initiated by a buggy KVM switch: We had 3 identical Debian Lenny server in production (redundant+load-balancing DB servers) we wanted to dist-upgrade to Squeeze… We did it the obvious way, choosing a low-load hour, upgrading first one while keeping the two other in production… Well in fact that *** KVM switch actually broadcasted the keystrokes to the 3 servers. We noticed at the kernel update that all were rebooting, then we got phone calls for the service not being available. Even more outstanding, is the dist-upgrade went equally well for the two other and as soon as they’d rebooted they were fully-functional… great testimony on the reliability and ease of upgrade you can expect from this distro, the first time i’ve seen such a potential big-disaster mishap turning to negligible impact unattended. (lol, after the adrelanine rush when you’re waiting them to boot, you can enjoy the ineffable feeling of having succeed a flawless unaware upgrade of a full pool of production servers…) Reply 417 Spinifex June 17, 2011 at 10:08 pm This is wonderful. Absolutly wonderful. I don’t feel like such a fool after all. I have done a few of these. Reply 418 wolfric June 19, 2011 at 9:41 am instead of ifconfig eth0:1 bla, doing eth0 bla…. yeah that wasn’t pretty. did > instead of >> once as well Reading these comments, it sounds like this might help alias rm=’sleep 2&&rm’ Reply 419 James June 26, 2011 at 2:53 pm has anyone else noticed that very few of the comments on this page use full English???? it’s scary…. and one made no logical sense until i imagined a few commas. also, i’ve done the remove everything ….. Reply 420 fooboo June 26, 2011 at 7:39 pm Capital ‘h’ for “Has”. One question mark is enough to denote a question. Capital ‘i’ for “It’s” since it is the beginning of a sentence. Three full stops ‘…’ denote an ellipsis not four. Capital ‘i’ for “I imagined…”. Capital ‘a’ for “also” as it’s the start of a sentence albeit a poor one. Capital ‘i’ for “I’ve” and another ellipsis with too many full stops/periods. Yes, I suppose it is fun commenting on other people’s poor grammar. Reply 421 Sumner June 27, 2011 at 4:57 pm It’s crucial to stay aware of context! My coworker wanted to delete the contents of the current working directory, so he typed “rm *”, but he didn’t have the necessary permissions to remove those files as himself, so he moved on to: su - [root password] rm * There was just a little problem with doing that. The “su -” command moved him to the root directory, and shortly thereafter, he realized the system was no longer bootable. Reply 422 Amit Chaudhary July 1, 2011 at 5:12 am Hi, I did rm -rf /abc/* / Just out of hurry and my machine was dead.Luckly it was local machine.One day rebooted the server thinking working on local machine and bang i rebooted the nagios monitoring server :) Reply 423 dagon July 1, 2011 at 1:39 pm I removed a lot of files by accident. This learned me to create a wrapper for rm which copies the files into “trash” directory instead of physicaly deleting them. Reply 424 Cody July 1, 2011 at 4:34 pm Hm, why that when you have these options to rm (assuming you do; else ignore) ? -i prompt before every removal -I prompt once before removing more than three files, or when removing recursively. Less intrusive than -i, while still giving protection against most mistakes (E.g., just alias rm to ‘rm -i’ for example – like: alias rm=’rm -i’ in your login script (.bashrc or equivalent)) The problem with such approaches, of course, is if you change accounts, move to a different machine that doesn’t have the option or an alias set up, you could end up in trouble if you rely on them. Reply 425 Oisín July 1, 2011 at 6:54 pm “The problem with such approaches, of course, is if you change accounts, move to a different machine that doesn’t have the option or an alias set up, you could end up in trouble if you rely on them.” True. Best to have a unique alias that will fail with an error on another machine. For example, alias rim=’rm -i’. Similarly you could have a “rim” for killing processes (usage: rim [job])… Reply 426 Bill Timmins July 8, 2011 at 10:01 am My favourite experience was the error message when I typed rm -rf * .bak which was .bak not found I realised slowly what it meant :-) Reply 427 edueloco July 11, 2011 at 7:00 pm Guys/Gals, i had a great time reading all you!!! We all have stumbled for God’s sake!!! i remember my, already mentioned, last|reboot’ in a payroll server…. at the end of the month…. LOL’s for all, may we all learn from our mistakes!!! Farewell!! Reply 428 Greywolf July 11, 2011 at 7:36 pm I was watching an admin deal with an interesting problem. We had a file named ” ” (space) in the root directory which he wanted to delete… …you’re way ahead of me. Sure enough, he typed: rm(space)-rf(space)/(space)(return) This system was the last running of its kind in the company, we had no alternate software for it and no way to recover or boot it up again. This was in 1988 at a shop that specialised in doing its own ports of UNIX to just about any piece of hardware you gave it (“give us 7 days and we’ll have a single-user prompt on a crowbar.”). For this one, though, it was so ancient there was just no helping it… Reply 429 Tom July 19, 2011 at 11:22 pm But it’s so easy.. touch ” ” rm ” ” and that’s all.. Reply 430 Sam Watkins July 12, 2011 at 2:57 am One thing I stuffed up at work was dumping a database to text format, then reinserting the records. I think it was with postgresql a few years back. You would expect it could read its own SQL, but we were using a human-readable date format and the timezone was printed as ‘EST’, for Eastern Standard Time in Australia. However on re-importing the date, the DBMS interpreted it as EST in America, apparently. So, I learned always to use ISO-style date and time format, e.g. 2011-12-31 23:59:59. It sorts nicely too. This ‘EST’ confusion continues today: > date Tue Jul 12 12:54:29 EST 2011 > TZ=Australia/Melbourne date Tue Jul 12 12:54:30 EST 2011 > TZ=EST date Mon Jul 11 21:54:35 EST 2011 Reply 431 Sam Watkins July 12, 2011 at 3:23 pm I’d like to propose a new direction for this thread… What, if anything, have we learned from all these many and varied UNIX disasters? Here’s my list so far: 1. Keep multiple remote versioned backups of anything you care about. 2. Keep multiple failover systems ready to replace any critical system in a moment. 3. If a process is complex or time consuming or error-prone, try to automate it. 4. Hire competent netops and coders, not the numbskulls who posted here ;) 5. Do not entrust any shell coding ‘toddler’ with root, DBA powers… or anything else. Please tell me what you’ve learned too… or any other good advice for netops… Reply 432 Mr Z July 12, 2011 at 3:52 pm Do a search on the page for ‘learn’ or ‘lesson’ and you’ll quickly compile a list. One of the simpler things is to make a backup of any file you edit before you edit, delete it when the edits test as good. Test with ‘ls’ command before running the ‘rm’ command – the extra seconds is priceless If you can’t replace it or rebuild it – think long and hard about why you can’t, why there is no backup, and just what in the fuck do you think you are doing by editing/changing anything on it. That goes double if it is a production box. Seriously, why don’t you have a test system? Why are you running a command which modifies or deletes anything if you don’t have a backout plan? Sorry, say that again? Are you sure you want to do that? NO, go to lunch and think about it. If it’s still a good idea, well then ok. Go ahead, run that command with no backout plan. It’s on your head. As for your points, 1=yes, 2=yes, 3=YES, 4=ha ha, 5=hey, wait a minute. Even script gods have bad days, so don’t trust anyone to get it right. Severely restrict root access, DBA powers etc. to only those who must have them. There are some things which should not be at root access levels – if that is inconvenient, reassess permissions for it. If it should be root-access-only reassess why anyone needs access to it. Even if you are a script god/admin don’t do your daily work as root. Use sudo. Restrict root login to ONLY the console, no remote root access. Stuff the reboot command inside an alias script which prompts you with all the right information that you should know before rebooting before it runs the reboot command, such as server name, uptime, number of connected users etc. You might even restrict that command (and others) to a special user that you must sudo to first. On production systems, 15 seconds to access it before you run the reboot command can save your job for you. Now, write a script to implement these changes you have made on any and all production servers. Implement them on all production servers. Prepare your USB controlled NERF missile launcher to destroy anyone who complains. Reply 433 Elvis July 23, 2011 at 12:26 pm Wait… are you *THE* Mr. Z ??? If so, thank you for everything, with thirty years delay :) Reply 434 Mr Z July 24, 2011 at 4:01 am I LIKE my moniker. Apparently 4000+ others like it enough to use it also. If I managed to do something kind 30 years ago, I don’t remember it so I’ll not try taking credit for it. Of course, I can’t remember much of last week, so 40 years ago is going to be a stretch no matter how much red bull I drink. I guess I’m “an” Mr Z, probably not “THE” Mr Z, but he has a good name ;-) Reply 435 Elvis July 24, 2011 at 3:36 pm Anyway, thank you! Being admin for years, I still find usefull things on this site, at least thank you for that. I understand what you’re saying. I can’t remember what I ate three days ago. Mr. Z I was looking for is related to Triad, Google knows, Google knows. I learned from one of his friends that he’s “working some Linux boxes with his father”, that’s why my neural network rang a bell. Best! Reply 436 Mr Z July 24, 2011 at 4:16 am BTW, my comments to you are how I treat system administration. I always assume my assumptions are wrong… ba dump ba. Seriously I always check twice especially when I think I know what I’m doing. This applies to many things in life. If you are building some cabinets or such at home, measure 4 times, cut once. You only get to cut it too short one time. You can cut it too long several times. There is no substitute for a second opinion. Google et al are a wonderful source of second opinions, and like this site, those opinions are based on how to fuck it up. So read these and others and do not make those mistakes. Of course you will, we all will, but if this reduces the volume and regularity of your mistakes then it was well worth putting this blog online. My own personal mistakes: Sun hardware often has interlock switches on the covers. Don’t remove them to look inside of a production box that is on line. TEST your backups. No, I really mean TEST your backups the hard way. This is what test servers are for. If you don’t know your backup works it WILL fail. While you are at it, try a little bit of disk-2-disk-2-tape backup. Multiple copies never hurts and available on network attached disk is a speedy recovery method. Did I mention that you should always have backups – plural. Disaster recovery is a life style, not a SLA requirement. Repeat that to yourself every morning. Whether you care for your customer’s data or not, you SHOULD care about your nights and weekends. Many of which you might spend in a blind panic if you lose your backups, or don’t have any to start with. Oh, on the topic of backups – install a version control system. Use it for all your scripts. Trust me, you don’t want to rewrite them from scratch – ever. Build all your boxen so that they can be replaced in a heartbeat or as few heartbeats as you can manage. Seriously, remember the sanctity of free evenings and weekends at the bar with your friends. These will be violated regularly if you are not ready for disaster(s). Another? High availability systems are worth less the the single router they are connected to. tut tut tut. Yep, I’ve done that one, or rather suffered it. The OSS group or Network group are NOT your friends. Make sure that disaster recovery is THEIR lifestyle too. Trust no one! Reply 437 Beus July 12, 2011 at 3:26 pm As I’m reading these mistakes, I remember making a funny one a few years ago. At university, we had our own server in school’s computer center with some “home-made” utilities, one of which being called “kick”. It served to kill (-9) all the processes of a specified user. This was achieved simply by setting a root suid attribute to the binary, calling setuid() syscall to the specified UID and executing killall -9 -1 to kill everything what kernel allowed it to. It had already been a long time since we created these utilities, partly forgetting its inner functioning, and we freshly introduced our “home-made” kernel security patch that was configured to make the setuid() and seteuid() syscalls ineffective (the syscalls returned no error, but process’s (E)UID remained unchanged) so as to disable any kind of root access via an exploited setuid program – to become root, one had to log in properly via ssh from one of the defined IP addresses. In addition, to make it more convenient for the roots to administer the machine, certain UIDs have had certain capabilities added, e.g. to delete some non-owned files from defined directories (not the system ones) or to kill not-owned processes. I was, of course, one of such users. That day, it was Friday evening, I was somewhere else, working remotely, and when finishing, I was too lazy to exit all my shells and other processes properly and wanted to show my friends a way how to log out both impressively and easily. So I issued kick beus and watched with surprise how all my friends’ screens (logged in on the same machine) went blank suddenly. Then I realized the truth of the matter: kick command did a setuid() syscall to a specified user UID which failed, and, as still having my capabilities of killing non-owned (including root’s) processes, the executed killall did its work properly and we ended up with the machine being dead for the whole weekend and Monday morning. Just the kernel and init survived so we could only watch it responding to ping ;-) Since then, we moved sshd into inittab and later introduced a remote-sysrq target into iptables :-) Reply 438 SW July 13, 2011 at 5:39 am Faulty muscle memory. I once wanted to type vi xxxxxx.c but ended up with rm xxxxxx.c The Return keys are really “Point of No Return” keys with commands like this. Reply 439 alldf July 26, 2011 at 11:31 am When i was young and stupid i accidentaly run poweroff command on remote linux box. I wanted to shut-down my local machine, but forgot i’m logged on remote server. This was at evening just before i leaved to the pub. I have to investigate why this server is not running at the morning and i noticed shutdown sequence in log… From this time I’M NOT using poweroff command. I’m using “reboot” in every case and then shut down machine while rebooting by pressing power button. If i have to use poweroff command (eg. i really want to shutdown remote machine) i double check if i’m logged on right box. Reply 440 Christian Straube August 1, 2011 at 2:58 pm Hi, my last mistake was an “iptables -F” while connecting via SSH :-) After that, the local server support had to perform a reboot the restore the iptables rules… I locked myself out ^^ Reply 441 Philip550c August 4, 2011 at 7:22 am Not to be a dick but you made another mistake, in your article you said “day today jobs” but its “day to day jobs”. Sorry I couldn’t resist. Reply 442 zappafan August 12, 2011 at 6:23 am My first, great, unforgettable mistake: first days in linux, run gparted in a have-absolutely-no-idea-of-what-I’m-doing mood. Just answered “Yes” without reading the warning text: that was the last time I saw a running Ubuntu, on reboot computer said “no partition table found”. Second great mistake: first hours in zsh, totally amazed by its great tab completion system! its damned tab completion made me type “rm /bin/mount”. Reply 443 zappafan August 12, 2011 at 6:41 am I nearly forgot my third great mistake! I was having so much fun on the University gentoo server, and that days were my first very funny days on Apache, I was editing httpd.conf and I didn’t realize I had pressed return key, decommenting a help line… Server restarted and within minutes ALL WEB APPLICATIONS USED BY THOUSAND STUDENTS returned a very lolling SOCKET ERROR! ahahah Reply 444 phord August 24, 2011 at 11:23 am I’m on the phone with a lab tech at a large cable TV company. He’s got a runaway process on their video-on-demand server chewing up a CPU core. me: It’s a non-critical process, so you just kill it and restart it. tech: How do I do that? me: Type “ps -ax | grep processname”. Find the PID. Then do a “kill -9 ” followed by that pid. tech: “kay eye ell ell dash nine” ok. Now what. me: It should restar… tech: Whoa. Uh. Hang on. Something else is going on here. I, uh, I gotta go. me: Ok. call me later when you have time. An hour later… tech: Don’t know what that was, but it’s fixed now. Ok, so I killed that process. me: Yeah? So check if it started back up now. tech: Ok. I remember the pid was “1″… me: hahaha. Wait. you are kidding, right? tech: me: O_o Reply 445 steve August 31, 2011 at 12:56 pm …login… /usr/local/bin/sudo su - cd / ls -al …looking for a txt file… unix2dos * ………ctrl-c……..ctrl-d……….ctrl-z…….go get the installation media… Reply 446 Aaron August 31, 2011 at 11:53 pm One time, when compiling some of my code, I accidentally did gcc -o file1.c file2.c instead of gcc -o file.o file1.c file2.c :( Obviously, I was not happy after that. I have also done the accidental “type command into wrong window” before, except I did it the other way around. I meant to shut down the remote computer (It was another workstation, not a production server), but I accidentally did the shutdown in a window for the local computer instead. So I was the only one that was bothered, but I had a LOT of stuff up that I was working on that wasn’t all saved. End users spared, but I was furious. Somewhat similar, but not necessarily my own mistyped commands: I really like making programs for automation, so I was all too happy to make something when some of the staff here said “We’re tired of shutting down computers in this area manually every night after people leave.” I made them something that would shut down all the computers in their area, only to be used at the end of the day. It was called “SHUTDOWN LEARNING COMMONS” To this day, they still get a new person from time to time which runs it, wondering what it is for. They have shut their area down a few times when they had a lot of people doing work over there. As an added bonus, I made the program initially so that it would run in the context of my personal admin account so that it had privilege to shut down the remote computers. The remote computers were windows xp, and the xp window that pops up warning users of the remote shutdown displays front and center what network account initiated the remote shutdown. I was told one day that someone had accidentally run it again, and that one of the more “important” users was searching for me to “give me a piece of his mind” because he saw my name attached to it, even though I was not even at work when it happened. Since then, I made an account called “shutdownLC” whose only purpose is for that one program. Another time, network switch related rather than unix: I accidentally locked myself out of our core switch when I was changing vlan settings on it. This rendered our entire network basically useless. I know this is one of the more common mistakes, but here’s the worst part: It was an older switch that doesn’t have an RJ45 console maintenance port, so all of our serial cables with RJ45 connectors on the other end were useless for this switch. We had a few old ones in the building, but it had been so long since they had been used that nobody knew where they were and we couldn’t find them. The network was down for a few hours before my boss made a special trip in and made me feel like an idiot after he got there: he unplugged the cables that went to the core switch and plugged them into a different switch then said “This might slow things down a bit, but at least it will work now.” I don’t know why I didn’t think of that… Reply 447 Fab September 5, 2011 at 3:07 pm Had to fix a box that was borked by an admin that (possibly) accidentally deleted /bin/grep the system wouldn’t boot. It was full of errors everywhere and the system was hanging. While troubleshooting in chroot environment that booted from a CD I tried to grep config files to find out to my surprise that grep wasn’t there (2-3 hours down the road of course). Reply 448 Fab September 5, 2011 at 3:12 pm But my personal worst was after one month of software development in C:$ rm * .o
rm: cannot remove .o’: No such file or directory

(hint: the space after the * was fatal)
(Reward: I learned how to make Makefiles after that)
(Reward2: I learned how to make backups as well)

449 Tom September 5, 2011 at 8:25 pm

Some filesystems has undelete utility.
Just turn plug off the power cable not allowing disks to sync or write something, take a deep breath, grab some coffee and try to get data back. I learned this after destroying some data on different filesystems, ext2/3/4/reiserfs, even ntfs.
The more data is written, the more fragmented filesystem is, the less chance is for recovery.
Case is more difficult when using RAID, it usually needs backup system to another array or tape.

450 csm September 19, 2011 at 10:49 am

In my previous company 5 years ago.

Day 1, they made me use a solaris machine to practice while waiting for my project assignment. After a boring day, I shut it down and went home.

Day 2, Cad admin is hunting me for shutting down the NIS server.

:)

451 cr September 22, 2011 at 10:44 am

Locking myself down from remoteserver – done
Removing all file in ~ – done
I accidentally typed

rm -rf * /opt/whatever/

rm -rf /opt/whatever/*

I don’t really know why i did this but it happened..
Since that tim i always have a file called ‘-i’ in my homedirectory.

452 Wolf Halton September 22, 2011 at 10:58 pm

Adding a sudoer to a machine on which I had sudo rights, but not root:

# User privilege specification
cschultz ALL=(ALL:ALL) ALL # this was already here
mjagger ALL=(ALL:ALL) ALL # this was already here
metoo ALL=(ALL;ALL) ALL # the semicolon broke sudo so

453 Robert Parten September 27, 2011 at 4:48 pm

Well let’s see:

1. Shutdown public interface on server – check
2. Locked myself out of a firewall – check
3. Caused metldown of database servers by not reviewing a switch config – check
4. Gave wrong IP to switch in lab and deployed to production – check
5. Shutdown server via ssh shell when I meant to shutdown my machine – check

454 Nirvani September 29, 2011 at 5:14 am

Some mistakes I have done as an admin:

changed the network(eth0) configuration of around 100 servers while data center move and forgot changing the network ID – This was done through a script from a trusted host. Also issued ssh xyz poweroff -f on the admin host which killed the admin host itself.

Copied upto 50GB of data to /tmp as a temporary location(had planned to copy them to another location) and forgot. One reboot and the whole data is gone.

ran crle -l /some/path/to/libs on a Solaris production box and it messed the whole system. None of the other admins could fix it and we had to go for a reinstallation.

455 Nirvani September 29, 2011 at 5:26 am

Another mistake i forgot to mention:

cat > /etc/vfstab instead of cat >> /etc/vfstab

luckily i had another identical machine and copied the contents of fstab. Dint break anything :)

456 Superkikim October 4, 2011 at 2:59 pm

Excellent topic. Thank you for sharing with us. And thank you to all who left some more comments.

I think chown and chmod are by far worse then rm -rf. Because usually, you have backups, and you can easily restore a folder unless you deleted your entire file system. Fast and clean. But chown and chmod ? you’re good to make a full restore…. unless you want to lose some time believing that you may find a way to fix the mess…

Another one I did was a quite simple

apt-get autoremove

Well… It removed automatically alright…. REmoved dependencies for a uninstalled application…. but these dependencies were still required for quite a lot other packages. …. I still don’t know if it was supposed to do that… but I don’t use autoremove anymore …

I’m still new on Linux… Working sporadically on it since 3 or 4 years… Lots of mistake to come… for sure. I’ll keep this post in my bookmarks to come by sometimes and read again.

457 Jack Wade October 7, 2011 at 10:02 pm

Not really a significant mistake in terms of downtime/destruction/damage/etc, but still. So I work for a hosting company, and some of our boxes have upwards of a thousand or so domains on them, including the Apache virtual hosts as well as named zones.

Well, one day I edited a zonefile and restarted named rather than reloading it. named takes *forever* to start up from scratch with that many zones so all the sites were unresolvable for around 7 or so minutes. (Note we don’t have a redundant/slave DNS set up, but we definitely should)

458 Aaron October 8, 2011 at 1:21 pm

A bad mix of screen and rsync. Working on adding a slave database server to a staging environment. opened screen and a term to the existing database, and a term to the new host which was to be a slave. got my rsync command ready, and fired off,, only I was on the wrong DB and using –delete, so i wiped out the staging database in a matter of seconds. I never had so many perplexed developers calling me in my life. =), luckily staging is a dump from prod with some sanitizations run to fudge data so i simply needed to run the process manually. In the long run the dev’s weren’t that upset because i gave them a free day off. woohoo

459 Ken October 10, 2011 at 5:32 pm

Our Linux servers are -csh by default, because our production software uses csh environment variables. My workstation is bash by default (and preferred), so, upon making changes to the .login script, I use this:
if [ $this = "$that" ]
then
# more bash syntax that killed the login

Luckily, not a huge mistake, but it did kill everyone’s ability to login. So, had to mount the home directory as a sftpfs, and fix .login.

Multiple shells.

460 jd October 26, 2011 at 9:14 pm

Well, I do remember having a mc with two panes: a 10 gig folder with some clients’ emails and a 9 gig folder with their backup from a week ago. I went to delete the backup folder … the wrong one.
Yay

461 Lala October 27, 2011 at 4:01 pm

Bought a new disk for the stuff i mirror because home partition on my workstation was quite full.

mkdir /home/lala/dump
mount /dev/sdaX /home/lala/dump
mv /home/lala/mirror /home/lala/dump

Then i created a cron job:

cd /home/lala/dump/mirror ; rsync -rav –delete rsync://somestuff.org/repoX . > /dev/null 2>&1

No problem for a couple of weeks until I had to reboot my box. Next morning, looking at my completely empty home (except some RPMs…), I realized that i forgot to add an entry for /home/lala/dump and a “&&” instead of “;” in the cron job would have been a nice to have.

-.-

462 Blurf October 30, 2011 at 2:10 pm

Ahh..memories…..

Starting to fill up the hard drive on my development machine….
So..company buys me a nice shiny new drive, Install it late the night before, get up early to get some extra work done because I know it is going to take a while to set it all up.

mkfs the first partition on the second drive…..or… is it the second partition on the first drive? …OH crap!

Good news…there is a full backup on tape..
Bad news..it’s offsite, half hour out, half hour back, and 3 hours to restore it, *after* I rebuild a bootable system to restore it to…

It’s the little details that kill you – P

463 Michael November 2, 2011 at 9:30 pm

Digging up old thread, but I have a story that I thought would be more common:

I needed to create an account on the production database server. The admin trusted me, so logged in as root and let me create the account while he fetched coffee. It was running an old version of MySQL and was being fussy with the GRANT command so I just added the row directly to the mysql user’s table. Problem was I got the password wrong, so I thought, no problem, I’ll just update it:
UPDATE user SET password = Password(‘mypass’);

I got distracted by remembering to hash the password, and by reflex hit the semicolon [enter] before catching myself.

You know that moment when you go cold and realise you have just majorly messed up a production server that isn’t even your own and you really ought not to have been on in the first place? Fortunately I had done a select a few lines before and it was in my terminal scrollback. I had to manually enter in the 30 odd user password hashes before the admin got back with his coffee. I never told him.

I have now got in the habit of typing in “\c” and then only after I have finished the whole command and reread it do I remove the \c and add a semicolon.

464 martianunlimited November 4, 2011 at 12:19 am

My personal favourite is accidentally copying the default tcsh/csh prompt from the command line and pasting it. (ya.. i know you can just browse the history, but i think i was working with multiple terminals and trying the command line out in a different shell environment)

prompt> ./my_script.csh
prompt> prompt> ./my_script.csh

Immediate changed my prompt after that and refused to copy and paste command lines unless i absolutely have to

465 laapsaap November 10, 2011 at 8:33 am

All linux administrators will make all of these mistakes at some point. But the difference between a master and an apprentice is that masters only make the same mistake once.

466 Barfo Rama December 16, 2011 at 6:24 pm

The difference between a guru and a master is the guru knows how to fix it before anyone notices.

467 htan6x November 15, 2011 at 9:05 am

i run hostname -a instead of hostname on hpux. Noticed it right away and change back.

468 agreatunixadminIneverwillbe November 15, 2011 at 7:09 pm

I’ve done most all the mistakes mentioned here… for me, the worst was when early in my career I was working in a pre-production environment on a customer site, our “pilot” machine shows off our custom software to customer mgt and execs, lets them demo the software/provide feedback, that kind of thing… it’s hugely visible and must have constant availability. On this project, it’s the only one of its kind at the time due to budget constraints.

So, we spend almost a year developing the software for this particular project and had gone through a couple of rounds of feedback, software updates, etc for the customer to see, play with, approve, etc.

One day I’m on the box in the middle of the day prep’ing for an update later that night and instead of removing the contents of an old version’s backup directory, I run this at root

rm -rf *

and go back to my work, waiting for the command to finish. Only when the command was still running 10 minutes later did I realize what was going on. I stopped the rm command, but the damage was done. The box was hosed when it was supposed to have 24/7 availability.

I happened to be working with a great guy at the time that was superior with Unix. He did some command line magic using cpio, some pipes, and some other stuff to this day I still don’t know what he did that in real-time over the network re-built the the machine, software and all in a few hours. No one knew what happened… we took some heat for the downtime, but blamed it on something else I don’t remember.

I always always always have backups now… local backups, as well as remote backups.. and whenever I design systems, at least N+1 redundancy.

469 wharfie November 21, 2011 at 6:53 pm

Two of my favorites, one by me and one by the “boss”.

I was in charge if Usenet for a pretty big ISP. We had two identical Usenet servers, huge systems for the time, dual Xeons with hundreds of gigabytes of storage and as much ram as the rest of the machine room combined. We’d test upgrades on one, get it all working, then change the IP and swap the cables so that the test box became the production box. Well, one day after a successful upgrade I issued the command to remove and rebuild the news spool on the test box. Of course, I was *actually* logged into the identical-in-appearance production system. Tech support started getting calls 5 seconds after I hit enter. The moral: at least change the prompts or something to be different on test/production…

Another time I worked for a bunch of scientists and wasn’t “really” the admin. My boss was the sysadmin and I was the lab assistant, although he didn’t know much about running a computer so I did everything. But he insisted on asserting himself occasionally just for appearances and one day informed me that he’d added a cron job to “clean up” during the night. Next day we came in to a server filesystem that was completely empty except for /bin/crond and /bin/rm… The guy had written a shell that ended with (cd $HOME;rm -rf *). Of course, in those days cron ran as root… And root’s$HOME used to be /, not /root :-) Whoops.

470 Joce November 26, 2011 at 10:29 pm

$ls important_* important_1 important_2$mv important_*

…forgetting the destination directory name. Well, some consolation: you lose just 1/2 of the stuff…

471 zorg December 2, 2011 at 12:15 pm

ln -fs ./blabla /dev/null instead of ln -fs /dev/null ./blabla
(ipfw flush; sleep 3; ipfw add allow all from any to any;) &
but forget “&” :))

472 Doug December 13, 2011 at 7:55 pm

I’m not a sysadmin, but am an app admin. When we did a major software platform upgrade on the same hardware I had to get the new code deployed. It was one night before the release and I had to get the deploy script kicked off but it was late and I needed to get home. Screen wasn’t installed and nohup didn’t always work. So I ended up putting it in cron. Then I got sick and was out for several days which was the major release. Everyone kept wondering why one of the clusters kept going down at the same time each day and reverting out code.

473 shaheed December 14, 2011 at 7:13 am

The usual one which most of them has said already

rm -rf * instead of rm -rf ./*

474 matty August 28, 2012 at 11:21 pm

Do you mean rm -rf /* instead of rm -rf ./* ?

475 Pablo December 15, 2011 at 11:50 pm

The most painful form me what discover why an aix server only allow logins from root user…

Some TSM Genius did a chmod 666 /

476 Barfo Rama December 16, 2011 at 6:18 pm

It seems to be possible to comment a working command in hp-ux crontab, creating a syntax error, and thereby disappear crontab on exit. I always make a manual copy now, as well as auto-copy off the box periodically along with /usr/local/bin and such.

477 jason December 19, 2011 at 3:05 am

I forget the exact sequence and not about to experiment, but was trying to upgrade the libc binary on a live machine to gcc3 to compile, I think I just moved libc.so.6 to libc.so.6.old… and ln, mv, cp, etc rely on libc.so, so wouldn’t work anymore. I did manage to google a solution to get the machine going again.

478 jason December 19, 2011 at 3:08 am

crontab -r just has no reason to exist. open the cron file and delete everything if that’s what you want to do. I had a bad experience, and then could never remember if the bad one was -e (erase) or -r (remove). (-e is edit, -r is remove)

479 kishan December 23, 2011 at 10:03 am

i deleted my root dir :|
edit this command too

rm -rf ~

480 Kaeza December 27, 2011 at 6:29 am

rm -rf "$foo/$bar";


Now guess what happens when you didn’t properly define *both* the foo and bar variables, and run the script as root…

Oh yeah… I realized when it was too late:

481 pooh January 2, 2012 at 12:12 am

I am beginner in Linux administration and.. accidentelly I have deleted anaconda-ks.cfg and install logs under /root/ directory:

1. I unzip files with overwride option using ssh
2. I press refresh button in WinSCP to make sure that the files are overwritten
3. I did not realise that WinSCP have changed the directory after refresh :(
4. I found that file anaconda-ks.cfg and two more install logs are not expected and impulsive have deleted those files.
5. I minute later I found that this is mistake – in result there is no install history and template for future installations / possible some company scripts rely on this file.

Conclusion:

Make backups of all important;
Make sure that you are working in the right directory;
Focus your attention on one console and avoid guis for installation tasks;

Good Luck ;-)

482 Ken Mason January 4, 2012 at 7:02 pm

I needed to place server into single user mode. I did it. Then realized I was remote = I watched my ssh term complain of lost connection. Drive to data center. Feeling rather human.

483 Wolf Halton January 8, 2012 at 5:25 am

Virtual Machines have helped with that. With VSphere/ESXi when I mess up my ssh configs, I can just find the windows machine )that is the only OS that runs the VSphere client) and go in the back way to the server – fix my error and go happily on my way.
One time I accidentally used a semicolon instead of a colon in the sudoers file when adding a new admin to a client’s machine. The machine happened to be 1000 miles away in a data center I do not have access to otherwise. Since it was not one of my own VMs, I had to request a colleague in New England go in and fix my error.
It does make one feel human.
Cheers,
Wolf

484 Nicholas Bodley January 6, 2012 at 5:06 pm

[Please feel free to delete this, in parts, or even totally, btw, if it's too verbose.]

Fortunately, the only semi-serious goof I ever made to a system not my own was to drop the backup Flexowriter console typewriter at the early (above-ground) NORAD/BMEWS COC. I was fired. Should have asked for help lifting it off the floor.
Hit cork tile over concrete. I also wiped some customer data (not important) from a mag. drum-based data-storage device (SEMA, quite obscure) that I was trying to interface to an IBM electromechanical tab-card machine with lots of internal sparking contacts. They weren’t really concerned.

ThinkGeeks, iirc, is where you get T-shirts that say “I void warranties”. While that applies to me also, mine should say,
“I disable operating systems
(Only my own)”

At least three times, I have destroyed my Linux installations (and one Win XP; sorry).
Being over-tired and doing something risky is just stupid, when one is free to hit the pad.

Linspire was a lovely Linux distro., very nice user community. It faded when its developer died, and his son really didn’t want to continue. I hosed mine via emelFM, a twin-pane file manager, new to me. Had one pane up showing /, the other /home. Wanted to recursively ensure that all of /home was owned by {user}, not root. Chose wrong pane, because UI didn’t really make clear which pane was active. I forget details, but permissions for the system were hopelessly hosed. Backups? Good idea, but not yet…

I think it was XP; I had about six partitions, and some unallocated space. Made a partition there to put Linux into. Did a mkfs.ext3fs on what I thought was the free partition. Unfortunately, partition “index” numbers (such as /dev/sda2) don’t necessarily follow the physical sequence on the platters. That can be an horrible “gotcha!”
I got my ext3fs, OK, but XP was puzzled by the leftover hybrid partition type.. Backups? Not yet… (Most of the data is probably between the superblocks and inodes, still recoverable.)

While I like the Unix philosophy of avoiding user feedback unless wanted (I’m 75, and have some feel for the command line on, say, a DECwriter), I do think it shouldn’t be that difficult for the (GNU) mkfs commands to warn the user that a partition about to be formatted contains data. Afaik, # rm -rf actually does ask whether you’re sure, at least in some distros.

I tend to delete Bash history any time I’ve used an rm command that could cause future grief. All too easy to poke the up-arrow and hit Enter.

Discovered Parted Magic, iirc; small bootable CD, actually a workable small distro in its own right. Wanted to see what the GUI for deleting the partition table looked like.
OK, so /that’s/ what it looks like. Over-tired; clicked on [execute] instead of [dismiss] (actual names differ). Should clone that HD and try C. Grenier’s [testdisk].

OK, although have long believed in backups, and too rarely done them, decided to back up openSUSE 11.1. Real fool, doing it in a very makeshift fashion (I know better). Krusader, twin-pane file mgr. Active. F5 is Copy, F6 is Move, iirc. Meant to hit F5, hit F6, instead. Watched progress, and suddenly files in / started to disappear.

So, at last, I’m running Mint Linux (do investigate Cinnamon!), and do need to remember not to waste 100s of GB backing up, say, a downloaded, but outdated Linux .iso file.

Last, had several 100 GB of mostly non-valuable videos and stills on a 1-TB HD. Hosed it (tried to expand a partition, but machine timed out and shut down during resizing). Used the wrong recovery app. written by Chr. Grenier with no luck, of course. Wanted [photorec], but was trying to do it with [testdisk]. No luck; segfaulted testdisk once. Finally tried
# dd if=/dev/null of=/dev/sdb BS=1M (details from memory, maybe not totally correct). Took over 3 hours, and, this time, I had disabled machine shutdown after 2 hours; set it to “never”. Only then did I realize that I should have used [photorec].

485 Cody January 6, 2012 at 7:41 pm

Quick response, some may be vague. Hope its helpful.

Re: “Afaik, # rm -rf actually does ask whether you’re sure, at least in some distros.”

It’s an alias. rm -i does that. Do be careful you don’t rely on that though, because when the alias isn’t there.. well you can be surprised.

As for mkfs – define “data”. Best option for future is to (of course make sure its not mounted too – may give notice on that but not done it manually in some long while) check disk free on that mount point, e.g.:

df -h /

would show how much is in use / free on the root volume.
(that’s the other thing. if a partition is not mounted, nothing will be seen data wise – not at the higher level anyway).

And for backup, depending on size of what needs backup, a few thoughts:
cron job that backs up nightly/weekly, whatever ? (or even something like the program bacula).
Also, you can save a dump of the partition table, boot sector and so on (not useful for everything, however it can be of use at times).

One other thing about backups : when recovering data, never write to the disk its on, including restoring or installing the program to recover with to it.

And on another note, regarding permissions. You could always do:

chown -vR user.group /home/
or
chown -vR user:group /home/

Just make sure you never do chown -Rv user.group .* or similar (in general be very careful with recursively changing ownership).

486 Cody January 6, 2012 at 7:42 pm

Hmm, last two commands got messed up. I stupidly used arrow brackets. I should have used the html codes.

after /home/ should be the user’s directory.

487 Been there done that January 9, 2012 at 5:29 pm

Working for a hosting company has given me some interesting experiences.

Like the time a customer support agent gave a customer the instruction (via email) to simply execute on his CentOS box [rpm -e XXX]. Apparently the customer didn’t understand so the customer support agent typed in [rpm -e ] and then pasted [rpm -e XXX]. I spent well over 6 hours getting RPM functional again and I can assure you it is not a fun experience.

Or the entertaining situation created when a customer support agent wanted to “grow” an existing partition for a customer on a second drive by creating a new partition with the same name and then mounted it over the existing partition. Nothing was lost, but you should have heard the customer scream about his “data loss”. Luckily, mounting a partition in this fashion doesn’t erase data but simply hides it – yet it still took several of us a couple of hours to figure out what had actually transpired…

488 Kiwi Nick January 16, 2012 at 7:41 am

A few that I’ve seen happen

1. SQL: DELETE FROM some_table; — missing WHERE clause. Bun fight for the next 3 hours for the trained monkeys at the client to get the backup tapes for a restore.
2. There is a Linux installation where almost EVERYONE has the same home directory (no need to store user-specific stuff on this machine, they all share the same .profile). From force of habit: userdel -r minion; ## because minion had left the company. -r wiped out the shared home dir for all users, including all the DBF files for the production Oracle database. About a day’s downtime.
3. The same guy made a second mistake: logged in remotely to a Unix server, he goes: ifdown eth0; ## then intended to ifup eth0; to pick up a config change. But he couldn’t get his prompt back after the first command, I wonder why ;-) He rang up and told someone to force the power off, then on.
4. An entire group of suburbs without phone service for a day: some vague rumour about something misconfigured.
5. One that I did myself: very early days of Unix – confusing rm instead of mv. It was only a single file deleted (the intended destination didn’t exist, and rm croaked with the error, which alerted me to the mistake). But it was still 2 hours recreating the C-program assignment from memory – some one else’s who I was helping.
6. Another one I did myself: swapping between the CVS storage area, and the equivalent working directory in my home directory: wanted to delete part of the working directory tree and start over: rm -fr stem_of_tree; ## but in the CVS area, not mine. Got someone to restore from backups. It was a pretty rudimentary CVS setup.

489 Kiwi Nick January 16, 2012 at 8:09 am

Now for some prevention.

• mv -i ## always use -i
• rm -i ## at least when you have your L-plates on (L-plates is Aus slang for “learner driver”).
• Only execute something as root if it won’t work otherwise; I generally have two windows open: my normal user and the root user, and I do things like ‘ls, cat, less’ in the normal window. Then I do things like vi, or rm as root (but I do a ls just before the rm, as root).
• I configure my root windows with a different background colour.
• pwd; prior to rm or recursive commands.
• prior to a recursive command on a small directory tree, find . -print; or, find . -type d -print; see how many files will be affected (or for slightly larger situations, the second one gives the range of directories to be affected).
• When I write a script to rename files or change them in a robotic manner, I put an echo in front of the intended command, so it will give me the commands that are going to happen. If satisfied, I remove the echo, to let the command actually happen.
• If I’m editing files in a robotic manner (by script), I create the new files with a backup extension (eg file.cxx becomes file.cxx.new). Then I diff the various original files against their new ones, to ensure it is as intended, then I rename the files, mv -i ; and keep my eyes peeled as I say yes to each one.
• The > is your enemy. If I want to create a brand new file by redirection, I … cat brand_new_file; (expect a not found error), then some-cmd > brand_new_file;
• If I want to append by redirection, I make a backup of the target file regardless.
• When I edit a config file in /etc (or anywhere), I … cp -i file.conf file.conf.20120116; named after today’s date. The dates aren’t that truthful, but it’s miles better than anything.
• Create dedicated users for running VMs and similar (especially if they are shared). That’s the approach Apache uses (usually automated).

490 Beus January 16, 2012 at 4:32 pm

- Adding echo before the actual effective command when making any automated file-manipulation script and only removing it after several satisfactory test-runs is exactly my approach, too :)
- If creating a brand-new-file from redirection, I use the advantage of tabkey completion ;)
- When editing config files, I usually just copy & comment out the original line(s) and then edit – it saves time spent by diff-ing the versions. This way I sometimes get configs a few times larger than they have to be, but an occasional cleanup of unnecessary stuff during the next edit takes care of that :-)
- As to the root access – I never have any root-shell window at all and do everything under my user. If anything really needs root privileges, on machines not needing top security I avoid having to type the root/sudo password each time by just running “do command parameters” where “do” is a tiny hyper-simple program in my ~/bin (which I have in $PATH) running the rest of its commandline under root privileges (has chown root:special_group and chmod 4510 (r-s–x—) ), basically going like this: setuid(geteuid()); execvp(argv[1],argv+1);  Mine is extended by running the shell if run without parameters but I almost never leave such a shell open unused. I even used this on some servers – being executable by only a certain group together with hiding it in odd locations provided high enough level of security (even if someone gained an access to one of the root-allowed users’ accounts, would have to know what to run). - Anyway, what contributes most to the commandline safety is the distinction in prompt – I always set it up to show hostname[tty]:dir> for me and root@hostname[tty]:dir> for root, sometimes in different colour. This way I’m always warned that it’s a root-shell before issuing a command. Benefit of always seeing the hostname and directory prior to even starting to type a command or manipulating files does not need any explanation (or if hostname is not distinguishing enough, just use any string to identify the machine) and knowing the tty proved very useful when hunting processes. So, the prompt consumes the whole line sometimes (in some deep subdir trees) but the information it provides is worth that space, and it’s more convenient and safe than to check for “pwd” or “hostname” each time, what you can forget sometimes (especially when tired, what is the worst time to run dangerous commands) :) Reply 491 Kiwi Nick January 16, 2012 at 8:28 am A few more, for major reconfigurations. • During major reconfigurations (using a liveCD), I mount everything -r (readonly) unless the task demands I write into that partition. • Mount a partition and have a nosey prior to hosing it (sometimes it won’t work coz it’s never had anything in it, but still: if the mount unexectedly works …) • Prior to removing a large directory tree, chmod 000; and/or, mv -i; to a new name, and leave it for a few days, to see if anything stops working. • rpm –test as a non-root user, prior to the real rpm. Ironic: there’s an advert below this box saying OMG’S of the week (Telstra). Yes, it’s a bit like that. Reply 492 Oisín January 16, 2012 at 5:35 pm “Mount a partition and have a nosey prior to hosing it (sometimes it won’t work coz it’s never had anything in it, but still: if the mount unexectedly works …)” Good point, which saved my skin a few times… “I’ll format that for the new backup partition. Well, I’ll just check if there’s anything on it and… wait, isn’t that my Windows partition? That’s not possible, because /dev/sdb is supposed to … oh, I plugged in that thing and… yikes.” Reply 493 Allen Garvin January 19, 2012 at 5:54 pm I once untarred an archive in the root dir. This was long ago, on HP-UX 8. It changed the permissions of the root (/) to 700. Since I was logged on as root, I saw no change in behavior. However, everyone else began seeing very strange behavior, and no one else could log on at all (it would seemingly log them on, then drop them). It took about 20 minutes of looking, before finding the problem. I did the killall thing once long ago on Solaris, too. It taught me to switch to using pkill everywhere (that had it). Can’t tell you how many times I installed Solaris in the 90s with a fresh install, and then forgot to enable nis, and had to drive back to the data center afterwards because you couldn’t log in as root in telnet. Reply 494 Programie February 7, 2012 at 12:25 pm You are currently in your systems root directory (/). You want to delete all files in /tmp but typ “rm -rf /tmp/ *” (Note the space between / and *) After the command completed successfully, you try to change to another directory. But you can’t. There is nothing! It doesn’t happen to me yet… but hey, the life can be long. :D Another mistake which is possible: You are on a remote machine (via SSH) and think you are on your local machine. You want to re-partition your hard disk… Reply 495 Valeri Galtsev February 7, 2012 at 3:42 pm Hm… with a little knowledge of unix one can realize that rm -rf / is _NOT_ as devastating as many think here it is (I would add that as unix certification exam question). Let me re- post here what I already posted: “… One thing I couldn’t buy as ultimately devastating though: rm -rf / - if I ever manage to do it as root on my *nix box I expect /bin, /boot, and part of /dev gone (and whatever else could be in / alphabetically before /dev on that box). Then the device hosting root filesystem will be deleted, and this will be end of my trouble. The rest: /home, /lib, /lib64, /sbin, /tmp, /usr, /var will stay intact. Other opinions? Of course, you loose the system on the fly, but you can mount the partitions of the drive on another system, and you will see other filesystems mentioned above intact. Reply 496 cyclooctane April 5, 2012 at 9:22 pm The only problem with the above logic is what happends when someone lets the command run to completion. (I have seen exactly that scenario happen when a young sysadmin failed to realise in time what he had done) Anyway. My list Running comarnds on the wrong box (done) Locking my self out of a box with a firewall rule (done) ifdown eth0 when logged in on this NIC via ssh (done) chown apache:apache -R / var/www/html/* (done) dd if=/dev/zero of=/dev/sdb (when it should have been /dev/sda) I am sure there are many more that I can not remember off the top of my head. I remember these because they were the most painfull. :( Reply 497 Thomas Bliesener April 9, 2012 at 5:42 am > Locking my self out of a box with a firewall rule (done) >ifdown eth0 when logged in on this NIC via ssh (done) These are my favorite ones on very remote systems. ;-) 1 Set a reload of the firewall in the crontab 25 mins in the future 2 Set a reboot in the crontab 30 mins in the future. 3 Set an alarm 23 mins in the future on your desk. 3 Do changes only on the command line first and when they really really work, modify the config file. Reply 498 Tom February 25, 2012 at 7:24 am I stopped the iscsi daemon on the wrong machine because I was typing in the wrong shell. This resulted in bsod’s on a lot of VM’s that were running production Reply 499 LennoN February 28, 2012 at 12:57 pm Editing the pam.d/auth despite all the warning about it :) Cant even log into my root :) Reply 500 d March 4, 2012 at 7:41 pm chmod -R 000 / Reply 501 sachin March 6, 2012 at 4:21 am hi all , i was doing a database activity . i rarely use window machine .that night i was using windows and login using putty. i was just searching history where i found /etc/init.d/mysql stop i select that command and again start searching history even i dont know what i was searching then suddenly i press right mouse . in putty doing so is executing the selected command and within no time the database was down that was production database.. conclusion: never select orkeep any command in buffer . regards, sachin singh Reply 502 Cesar Chirino March 8, 2012 at 3:43 am One of the most absurd things had happened to me is that some years ago, the operating system let me create a file named “*”. Ha ha ha!! Imagine the efforts to delete, rename, move or something that file! Finally got backup file and got last changes lost… Wrote some programs again Reply 503 Cody March 10, 2012 at 1:24 pm Re: Imagine the efforts to delete, rename, move or something that file! Like this ?$ touch \*
$rm -i \* rm: remove regular empty file *’? y As I recall there was always a way to remove/etc a file by that name. There’s also similar tricks if a file – exists, e.g., most programs (gnu versions at least, cannot check others nowadays) allow — to signify end of – options. Reply 504 Cody March 10, 2012 at 1:44 pm Oh, and another way (just for reference). You actually had it kind of (hint, you quoted it). Another shell feature :$ rm “*”

That would also delete it (look up in e.g., the bash man page QUOTING). Note that it refers to this :
The special parameters * and @ have special meaning when in double quotes (see PARAMETERS below).

* by itself is not a parameter, so it’s not referring to that. See Special Parameters section for more info.

505 Daniel Hoherd May 4, 2012 at 5:10 pm

Regarding funky characters, sometimes you can use a ? in place of the problematic character, or use some other substring that will match. For instance, if you wanted to remove a file called “~*_blah” you could “rm ./??_blah”

506 Cody May 4, 2012 at 6:51 pm

You should be very careful with ? though. Reasons – although it won’t match a ‘.’ at the beginning, it would match the dot for file.ext if you specified, file?ext or a second dot.

That may not seem like a problem, but imagine this:

touch \.\?
( create a file called .? )

rm -f ??
(won’t delete the file, would delete files [that you have permission to delete] that have two character names; and if you specify -r it would do same for directories too, minus ..).

Sure, you can do ./.? but… hope you don’t do rm -rf without escaping the ?

Contrived or not, it’s best to escape characters or make absolutely sure it won’t do anything else other than you intend (which you even mention in a different post). Okay, so if you have a letter after, fine, but then you might be tempted to specify .?* and there’s another issue there. So yes, it can be done, but it should be pointed out it is risky.

507 soundman2020 March 12, 2012 at 2:37 am

Then there was the time that I wanted to use the tcsh shell for root login, instead of the standard bash shell. So I carefully edited /etc/passwd and changed the relevant part to “/bin/ctsh” instead of “/bin/tcsh”. Guess what happened next time I wanted to log in as root …

508 Programie March 12, 2012 at 8:48 am

I know that. And I also did some mistake like that.

My suggestion:
*NEVER* close you current opened shell before testing the new configuration in another instance. That also helps while configuring the SSH daemon. ;-)

Another mistake I did in the past but: Changing the SSH port without updating the firewall rules. :D
I ended with a real headless system. I was happy about the possibility to use a VNC remote console. ;-)

509 Daniel Hoherd May 4, 2012 at 5:08 pm

Something I’ve done in the past as a safety net for firewall edits (which isn’t fool proof but worked for me) is to open a screen session on the server whose firewall you’re modifying, then do “ssh -R 29922:localhost:22 some.other.server”, then if the firewall prevents inbound connections you can still connect through that pre-existing remote tunnel.

510 Cody March 13, 2012 at 5:50 pm

Maybe you know of it now, but.. if not:

Next time, instead of editing the passwd file, try invoking the program ‘chsh’. There’s also ‘chfn’ (for change finger information). Both have been around for aeons.

511 Collin B March 13, 2012 at 10:20 pm

I set the /bin/chmod to not be executable due to a failed copy and paste and got my commands switched around. Took some outside the box thinking but I used perl to reset it. I now use puppet to handle my configs, I use git to clone my puppet configs to a local user environment, test those changes then use git and push those changes back to puppet and update configs that way. Definitely a smarter way to go.

512 Marcus March 15, 2012 at 6:03 pm

Backup, backup and backup. Right! The most important thing. :)

513 Vincent March 29, 2012 at 6:01 pm

I wanted to change permissions on all hidden stuff of a user’s home directory :
chmod -r 644 /home/user/.*
Had to build and run a perl script to clone permissions on all /home from the last backup…

514 Vincent March 29, 2012 at 6:08 pm

btw you should try :
$cat > -r hello world Ctrl-D Mouhahaha Reply 515 Cody March 29, 2012 at 8:14 pm And ? What’s your point ? You miss several things, among them : - Many commands, rm included (and other similar basic utils) have the option ‘–’ to say stop processing – options. - It’s not hard to delete, ie : rm — -r - rm -r won’t work (because missing operand). - If you specify, say ‘rm -r ‘ it’ll remove the directory and skip the -r file. - List goes on. - See the first point. In other words: Nice try, but your attempt at being ‘clever’ actually is the opposite. Maybe you should try testing your cleverness first next time ? The words that come to mind of what it makes you look like otherwise, is.. well, I won’t even go there. Reply 516 Cody March 29, 2012 at 8:15 pm Right, html. I should say ‘rm -r [enter]‘ won’t work because it will be missing an operand. Reply 517 Cody March 29, 2012 at 8:24 pm Oh, and clarification of why it’ll skip it. If you don’t have it aliased, it might be an issue (but again – see first point). However, anyone with say rm -i as the alias for rm (which is, though not the best choice – it is still a default on say red hat systems and potentially others). In all cases then, it’s pretty silly to think it’s going to cause an issue (oh and as the person above you said: backup). So, yeah, it’s not an issue to delete files with special meaning characters. It’s also not the 70s when more was possible. I mean even these days you have less permission (as non root) to write to consoles and so on (say, echo or cat a file > /dev/pts/ …). Reply 518 Oisín March 29, 2012 at 8:35 pm rm ./-r ? Reply 519 Cody March 29, 2012 at 8:40 pm Another nice example indeed. And interestingly, I just tried unalias on rm, and it was still smart enough to process it as the option -r rather than play around with the file. Really though, I shouldn’t be surprised given how programs do process arguments passed to the command line. It makes perfect sense given it starts with the – character. In short, his idea is just very flawed from the start. Reply 520 Stefan April 7, 2012 at 10:46 am not linux related: i cut of the power cord from our primary core switch to the ups, while we re-configured the second switch. downtime about 20 minutes Reply 521 Luigi April 21, 2012 at 1:41 pm One little program that will save a lot of time: molly-guard. I found it on Aptitude and asks you the hostname of the machine you want to shutdown/reboot. This should adequately prevent accidental shutdowns Reply 522 Thomas Bliesener May 4, 2012 at 7:08 pm Sometimes the filename is so ugly that you can’t even type it. In these cases you can address the file by its i-node: 1. Catch the i-node with ls -i 2. find . -inum -exec rm {} \; This trick is desribed in “Unix Power Tools” from O’Reilly. Reply 523 Oisín May 5, 2012 at 3:28 am That is a lovely trick! Pity about the slightly painful find/rm syntax. Reply 524 Cody May 5, 2012 at 12:02 pm Yes, and that book happens to be quite good. I have an old edition of it and its still good. You could of course use the ‘-delete’ option to find to make it ‘easier’, but observe the man page warning on it first (do man find and then type /-delete to search for that). Reply 525 Jon May 6, 2012 at 9:02 pm Another one for mixing up terminal windows here – I had two terminal tabs open in OSX. One local, one SSH onto the server. The local mySQL had some changes I wanted to put onto the server, and I wanted the local to match the server more closely (apart from the changes). I was about to copy the server’s httpd.conf over to local, so I went to delete the local copy beforehand. cd’d to the right folder and went to delete it…. but between the two I’d swapped to a different window and then back, losing track of which terminal tab I was in. Deleted the httpd.conf file on the server and had to shut the box down while I rooted out the backup… then had to port all my changes over again. Reply 526 dn3s May 13, 2012 at 6:24 am my worst mistake is typing > instead of >>. i use >> way too often for my own good on files i really shouldn’t use it on. bash completion is the second worst, when i type rm startOfFilenam[tab] [enter without looking]. Reply 527 eeq May 14, 2012 at 6:41 pm It wasn’t all my fault, only about 98 – 99%. Someone had somehow created a file with zero length and the name “\”, without the double-quotes of course. Well, that just had to go. On about the fourth or fifth try I noticed it appeared to be working but taking way too long to delete one file. Ctrl-c. Real fast. And of course I was root. Was I ever lucky. I wiped the OS but stopped it before any user data was lost. The lead sys admin moved that HD to slot 1, put a new HD into slot 0, and handed me the OS CDs. I spent the rest of the evening re-installing the OS. Reply 528 Rich May 18, 2012 at 5:59 pm I’ve done some of these… ‘rm -rf fileprefix *’ late at night when trying to free disk space (instead of ‘fileprefix*’) removed all our corporate binaries; fortunately, backups had just completed and restore went quickly. Here’s a trick I depend on every day: All my ssh/xterm windows are color-coded – production server windows are loud and glaringly obvious, no matter what I am doing. Makes me nervous just typing in them… Reply 529 10speed June 11, 2012 at 12:49 pm My all time favourite was a co-workers mistake. DELETE FROM table name WHERE id – 1000 yup, he tried to remove all coloum 1000 bur ir qas the onlt one left. He luckly with a bit of scripting and his long buffer we were able to resore in just a few minites. Lesson learned select * before delete :) Reply 530 Phord June 12, 2012 at 6:53 pm Ouch! I’ve had some SQL typos like that, too. Lesson I learned is to always BEGIN TRANSACTION; before any data-modifying statements. If I have several steps to do, I’ll do something like this in Notepad or vim: BEGIN; DELETE FROM table name WHERE id – 1000 ; ROLLBACK; – COMMIT; I try this a few times until I get it right. Once I do, I remove the ROLLBACK and run it one more time. BEGIN; DELETE FROM table name WHERE id = 1000 ; – ROLLBACK; COMMIT; Reply 531 Remco June 17, 2012 at 5:24 pm My worst mistake ever was when I wanted to remove all DOT-directories (.esd-1000 etc.) in my /tmp-directory. cd /tmp rm -rf .* As this took more than the expected very short time period, I realized that .* also expanded to .., so I effectively did an “rm -rf /tmp/..” which is equivalent to “rm -rf /”… As I didn’t know what files were gone, I reinstalled my system. Nowadays bash prevents this from happening: rm -rf .* rm: cannot remove directory: .’ rm: cannot remove directory: ..’ Reply 532 Zeddarn June 18, 2012 at 2:15 pm Looked in as root, and deleted password file from RHEL 6, didn’t know how until when I restarted the server. Luckily I copied the file from another server and everything was back. Reply 533 Stv T June 20, 2012 at 10:08 am Way back, I was trying to debug a ‘.cshrc’ which was not working correctly and wanted to see where it was hanging. I put ‘#!/bin/csh -x’ at the head of the file and started a new session. Unremarkably, I was left staring at a blank screen with no output, but succeeded in exhausting all process slots, preventing anyone else from logging in and slowing the machine to a crawl. Luckily I had another window open and was able to remove the ‘#!/bin/csh -x’ and kill all my processes, although it took about 30 minutes. Never done it since !! Reply 534 Lou Marco August 4, 2012 at 2:09 pm Not really quite a command line screwup, but some of you might find it humorous. It does involve volcopy, at least (remember that one?) I used to manage a room full of 11/780′s and 750′s. They all used those old washing machine sized RP05 drives, think they held like 100 MB. They set the bus ID with a big plastic plug with a number, like a kid’s building block, that you stuck into a hole on the front. They used a removable platter stack – you could open the top of the drive like, well, a washing machine, reach in and unscrew the platters, and the whole thing came out. I took advantage of this to do full backups. Every Sunday night I’d come in, swap a spare set of platters into one of the drives, volcopy another drive to it, then just switch the ID blocks. The new copy became the old /usr or whatever, the old platters went on a shelf as a known good backup, and the now vacant drive was ready to run the hext backup. Lots faster than the 9 track tape drives I would have had to use and I could run more than one at once. Except one time I ran the volcopy before I remembered to switch ID blocks. All volcopy cared about where “volumes” – disc partitions – and the garbage scratch disc I’d just told it to back up to my /usr drive had perfectly valid partitions on it…. Whoops. Reply 535 okobaka August 13, 2012 at 6:46 pm 1. alt+f4 on putty hurts 2. rm cd /path/to/dir # someone called me on the cell while executing rm and after that i typed cp /path/to/dir (alias rm -r) 3. again rm dots, stars, spaces and other weird stuff with char coding ctrl+c and ctrl+z is your friend My way to prevent rm was touch 0; chmod 000 0 in directory i want to protect. Secret zero file. Same with: * remote mistakes, iptables, ifconfig, route, dhcpd. * partitioning (you want write to disc? yes, oh no no no … ), cfdisk, MBR, lilo * cat binary files, forced me to drop a shell because i didn’t know about reset/restart * wrong umask in root profile, as root chown or chmod -R /home/user/, hip hip horray for symlinks, * remote rsync * Whats going on? Something is not working? Why i cant do that? lets reboot – never reboot when something is cooking. First check if you have enough space to write your damn files on partition! Permissions and so on … … for sure i did more. I would like to say now i’m fine with mistakes because of double check everything, but that rm pissed me today, i’m going to block some arguments to rm – damn it. Nothing important was lost. Reply 536 kiran October 5, 2012 at 2:43 am while something went bad with a human err, its always better not to reboot.. I agree :) Reply 537 Behrooz August 16, 2012 at 1:33 pm My list: rm -rf / chmod 1700 / dd if=/dev/sdb1 of=/dev/sda dpkg -r$(dpkg –get-selections| awk ‘{print $1}’)#wanted to reinstall them afterwards Reply 538 Steve Holdoway August 30, 2012 at 5:56 am I’ve made a few (along with most of those above) , and had some done my predecessors… Overlapping partitions… the first one fills up, overwrites the start of the second one, which was when we found out that his backups didn’t work either. In the days before vipw, editing the password file on a full partition. Believing the old 2 volume System V ringbinder which told me to format, not mkfs a partition. At least my backups worked that time. Steep learning curve, that. A colleague that realised that hey could replace /etc/passwd from his PC ( DEC Pathworks anyone??? ). Unfortunately a DOS formatted /etc/passwd is as much use as no /etc/passwd. Re: the tar/untar problems with resetting the CWD permissions. Now you know the reason no to create an archive as tar cfz arch.tgz . – always add a directory above, or use tar cfz arch.tgz * .[a-zA-Z0-9]* – obviously ony use the second pattern if there are hidden files to transfer. Reply 539 Hoover September 1, 2012 at 9:23 am A long long time ago when I was learning my ropes as a DEC VAX / Ultrix admin, I hit the BREAK key on the VT220 terminal expecting to return to the LAT server prompt in order to open a new connection to another VAX (all serial terminals connected to a terminal server back in the days, must have been the late 80s or thereabouts). However, I did not know that when you hit BREAK on the terminal connected directly to the server’s serial port, it, well, … breaks. ;-P The machine was our main facultiy student server and had about 30 people logged in at the time, and as the classroom was right next to the admin room my boss and I instantly heard angry shouts and pitchforks being sharpened next door. He probably saved my life (or at the very least my reproductive capabilities ;-) by intoning in a rather loud voice: “Damn those pesky students! They’ve crashed math6 again!!!” ;-) Good times, good times. Reply 540 Jake September 11, 2012 at 8:14 am I’m just a young guy, however I do enjoy reading everyone’s war stories. Best to learn from listening to the people who have been there. I got a boneheaded one though. While I was updating my system, I forgot to update libc first. Luckily I had a boot disk handy, and was able to roll back the mistake. Cost me about 2 hours of a Sunday. Reply 541 UriH October 7, 2012 at 12:46 pm Man you gave me a good laugh, shutting down the interface you’re connecting from and then you can’t ssh anymore hahaha :D Reply 542 James October 14, 2012 at 3:43 am Two that come to mind, looks they’ve been mentioned already, but… ifconfig eth0 down on a box I’m ssh’d into on the eth0 interface. >_< and then the one that was the worst… mounting a drive to a folder on another drive that had files in it. My FreeNAS server through a FIT along with SMART errors (it seemed to think I was about 500GB short on disk space) :P Reply 543 Bernhard October 29, 2012 at 9:18 am Two more for the history: 1) “last | reboot” instead of “last reboot” … 2) ldd /bin/ls then copied the following output line from this command as root and executed: libc.so.6 => /lib64/libc.so.6. This deletes the contents of libc.so …. Reply 544 woot October 31, 2012 at 2:30 pm Wow if anyone of you was on my company, i think you will be fired very fast. I’m working since 10 years in IT typing commands every day, i never did any of these noob mistake. Reply 545 Cody October 31, 2012 at 8:55 pm You’d have no employees then and you’d be on your own in the end. You’d also eventually (if not already) need to fire yourself. And as for “noob mistakes” – either you’re lying, you’re unaware of mistakes you made or you simply are so inexperienced that the commands you type are incredibly basic. Maybe even a combination of the above. 10 years isn’t all that impressive anyway. And while I try not to criticize based on grammar (no one is perfect and this is more like a discussion forum than a professional document), I’ll make the exception (and I admit I make mistakes too). You might (although anyone with experience knows that to be a complete lie) make no mistakes at the command prompt but you sure make a lot of mistakes in your comment as far as language is concerned. I’ll refrain from pointing them out for the reasons I already mentioned (and I’m not going to stoop so low). I will however point out something else: those who are in denial about such things will never learn from mistakes and will never grow. In other words, they’re incapable of improving and should be asking themselves “why do I bother?” rather than insulting others who are more mature, honest and willing to grow. Truthfully, people with the attitude you have are afraid of admitting to mistakes and often the severity and amount of those mistakes. Does it really make you feel better? I doubt that very much. Hint: humans learn from mistakes. So either you’re not human or you don’t learn often. Reply 546 defenestratexp October 31, 2012 at 9:04 pm No offense, woot, but, if you’ve been working in I.T. for 10 years and never keyed in any mistakes, you’re either lying or not doing anything complex. #!/bin/bash grep woot /var/cache/competent_techs.dat >> /var/cache/lying_techs.dat sed s/woot//g /var/cache/competent_techs.dat > /tmp/temp_list mv /tmp/temp_list /var/cache/competent_techs.dat Reply 547 defenestratexp October 31, 2012 at 9:06 pm Damn! if you grep for Cody in the file on_the_warpath.txt, you will DEFINITELY get a hit. Well said, Cody. Reply 548 Cody November 2, 2012 at 4:58 pm Thanks :) No one is perfect and that includes me. I’m far from perfect. But to insult someone for being human (and therefore prone to mistakes) is quite arrogant and hypocritical. I have made quite some bad mistakes (though most not command line but in source code). Bad mistakes also means basic mistakes and something I had never made even when starting (when you’d expect it to happen). But it happens. I remember being quite tired at the time but that was about it. Later on the problem was discovered and I spent a great deal of effort to track it down and did. What matters is how you respond to the problems you may run into or cause. It’s about responsibility, and always improving yourself (and if you can, others). He or she is doing the opposite. woot reminds me of a bully. Nothing else. And anyone who has either experienced bullying first hand (as victim) or just has some insight to issues in the real world would most certainly know what that’s a sign of: very low self worth and esteem, embarrassed to admit to their own flaws and simply put people down to try to compensate for their own issues. It isn’t even authentic and they know it and everyone else knows it. The bully just is in denial. They may have issues that bother them but they are afraid to get help or whatever else. But in the end, it hurts everyone involved. Anyway – I appreciate the kind words. :) Speaking of such – I love your little script up there! Reply 549 10speed October 31, 2012 at 8:53 pm @woot. You must use windows Reply 550 Russ Thompson November 10, 2012 at 4:35 am Last big blooper was a simple restore of a test database — on the production server. I learned quickly how nice MySQL recovery works! Anyone that can, color code your PS1 strings for different servers, or at least have test one color and prod another. it helps to be able to just glance at the screen and the prompt color tells (yells?) what box you are on. But then again — humanity reigns — see above. Fumble Fingers Thompson Woot: reboot –now Reply 551 ps November 20, 2012 at 1:43 am my clm pkill -fv theapp I added -v the see what processes that where killed but pkill comes from grep so I killed every process _except_ theapp Reply 552 Akim November 20, 2012 at 4:37 pm Good one… Must have been painful, if the server was in production… Reply 553 DotMG November 20, 2012 at 6:44 pm cd /home/$USER; rm -rf *

But /home/\$USER didn’t exist. And the script was in /

554 Akim November 21, 2012 at 11:04 am

… well… should I say you deserved this one … That was my first thought…

And then, I thought that in the end, we all deserve what happens when we do mistakes :)

Reminds me of good all times with format C: /autotest … As efficient as rm -rf /

rm -rf * is quite bad in / for sure. But I wonder if chown -R or chmod -R is not worse…

With rm -rf, you know exactly what happens… You then just have to fix it. with chown or chmod… Hell… you can either decide to waste hours to try to fix it, or … just do an rm -rf and start again :D

555 Akim November 21, 2012 at 11:08 am

An easy one I made yesterday…

Connected via SSH, I made a script with:

iptables -F
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

then a full set of rules including rules allowing traffic on port 22 obviously.

I applied the script. Was working fine, except for one port. So I wanted to log iptables instead of drop. Made a new:

iptables -F
iptables -A INPUT -p tcp -m tcp –dport 22 -j LOG

and some more lines… Executed the script, and …. Well…. Had to go on vcenter to access the VM console and reset my rules…

556 Russ Hore November 26, 2012 at 4:24 pm

Being seen as the ‘Unix Expert’ meant anytime an admin wanted help they would call us and immediately log us in as root.
I wanted to see if a certain command (can’t remember which now) was installed in /bin so without looking typed
cd /bin
ls | grep thecommand
without looking at the screen.
Unfortunately the keyboard was not mapped properly and the “|” came out as a “>” overwriting grep

557 Carl Smith December 7, 2012 at 2:08 am

I once installed a newer version of Python, then removed the older one, which was the system default. Doh!

558 George December 11, 2012 at 2:12 pm

a dangerous one is this one:
rm -rf /whatever
The problem is maybe you’ll hit enter after the / (maybe someone pushes your chair, or whatever). Obviously your system is dead after that.
my solution is to type:
rm /whatever
then use the left arrow, and insert the -rf missing part

559 Ron December 13, 2012 at 10:35 am

For everyone who has done the ‘rm -rf *’ thing … there is a nice trick you can add to all the servers you administer ……

One time only, issue the command touch /-i

This creates a file called ‘-i’ and when the * in an rm -rf * mistake is expanded, it includes the ‘-i’ file name … that gets passed to the ‘rm’ command which treats it as a command line option – and prompts for confirmation of the action.

The result is that you get a second chance :-)

560 Rory Toma February 14, 2013 at 9:06 pm

My all-time favorite mistake:

I was on the main server that served NFS for about a dozen diskless workstations. I was adding a new one, and made a mistake, so I wanted to delete the etc directory, so I typed…

“rm -rf /etc”

“rm -rf etc”

Oops

561 Abhishek Bhardwaj February 24, 2013 at 10:22 am

Its not related to unix but I was taking a backup of .cbs file from cbex tool which is used in Clarify11.5 in production environment.
For taking backup you have to click on Export button but by mistake I clicked on Export/Purge button while it took the backup but it also deleted those files from production.
I told my local delivery team about it and then quickly put the backup files in production again.

562 Russ Thompson April 25, 2013 at 2:03 am

Wrong Box Story:

simple command: mysql <testDataBase.backup

But the production server, not the test server

ug!