My 10 UNIX Command Line Mistakes

My all time favorite mistake was a simple extra space:

cd /usr/lib
ls /tmp/foo/bar
I typed
rm -rf /tmp/foo/bar/ *
instead of
rm -rf /tmp/foo/bar/*
The system doesn’t run very will without all of it’s libraries……

Some classics here :)

public network interface shutdown … done
typing unix command on wrong box … done
Delete apache DocumentRoot … done
Firewall lockdone … done with a NAT rule redirecting the configuration interface of the firewall to another box, serial connection saved me

I can add, being trapped by aptitude keeping tracks of previously planned — but not executed — actions, like “remove slapd from the master directory server”

typing unix command on wrong box and firewall lockdown are all time classics: been there, done that.
but for me the absolute worst, on linux, was checking a mounted filesystem on a production server…

fsck /dev/sda2

the root filesystem was rendered unreadable. system down. dead. users really pissed off.
fortunately there was a full backup and the machine rebooted within an hour.

tar -czvf /path/to/file file_archive.tgz
instead of
tar -czvf file_archive.tgz /path/to/file
I ended up destroying that file and had no backup as this command was intended to provide the first backup – it was on the DHCP Linux production server and the file wad dhcpd.conf!

A couple of days ago I typed and executed (as root): rm -rf /* on my home development server. Not good. Thankfully, the server at the time had nothing important on it, which is why I had no backups …

I am still not sure *why* I did when I have read about all the warnings about using this command. (A dyslexic moment with the syntax?)

Ah well, a good lesson learned. At least it was not the disaster it could of been. I shall be *very* paranoid about this command in the future.

Deleted the files
I used place some files in /tmp/rama and some conf files at /home//httpd/conf file
I used to swap between these two directories by “cd -”
Executed the command rm -fr ./*
supposed to remove the files at /tmp/rama/*, but ended up by removing the file at /home//httpd/conf/*, with out any backup

conclusion: check the directory where the rm command removing files of which directory

rm -rf /dev/{null,zero} $HOME/dev/

instead of

cp -rf /dev/{null,zero} $HOME/dev/{null,zero}

…live cd and mknod savemes

echo > somefileWithManyCodeLines.pl

instead

echo >> somefileWithManyCodeLines.pl

the x-term history saves me !! (preveusly i dump the content of the file using cat)

so doesn’t care what term uses…set the history at the max.

and…. my prefered (many years ago happens me ….could be an excelent way to …really delete & hide information)

dd if=/dev/urandom of=/dev/withManyData count=1024 bs=1024

instead of

dd if=/dev/zero of=/dev/withManyData count=1 bs=512

the last one i callit “dd unleashed” ….

I did a rm -r /usr on my two-years-old gentoo installation. Ctrl-C after one second, most of /usr/bin removed. Running ubuntu since then.

My worst mistake was when I started using Ubuntu and changing abruptly (but willingly) from Windows to Linux. I accidentally deleted the entire filesystem with a command. No backups but it was a clean install.

or chown all the hidden file in a website directory by ” chown user.group .* ” end up changing owner on all the websites .. (one directory up)

one should use :
chown user.group .??*

good job ;-)

Firewall lock out: done.
Command on wrong server: done.

And the worst: update and upgrade while some important applications were running, of
course on a production server.. as someone mentioned the system doesn’t run very well
without all of its original libraries :)

This is how I protect myself http://blog.valqk.com/archives/Protect-yourself-from-accidentally-halting-a-server.-22.html” rel=”nofollow”>from making wrong halt:

in short I use molly-guard

Have done /sbin/init 1 instead of /sbin/init q multiple times.
after siwtching to MS Natural Keyboard.

I invented a new one today.

Just assuming that a [-v] option stands for –verbose

Yep, most of the time. But not on a [pkill] command.
[pkill -v myprocess] will kill _any_ process you can kill — except those whose name contains “myprocess”. Ooooops. :-!
(I just wanted pkill to display “verbose” information when killing processes)

Yes, I know. Pretty dumb thing. Lesson learned ?

I would suggest adding another critical rule to your list:
” Read The Fantastic Manual — First” ;-)

Mistake 1:

My Friend tried to see last reboot time and mistakenly executed command “last | reboot” instead of “last | grep reboot”

It made a outage on Production DB server.

Mistake 2:

Another guy, wants to see the FQDN on solaris box and executed “hostname -f”
It changed the hostname name to “-f” and clients faced lot of connectivity issues due to this mistake.
[ hostname -f is used in Linux to see FQDN name but it solaris its usage is different ]

I was dragged into a meeting one day and forgot to secure my Solaris session. A colleague and former friend did this: alias ll=’/usr/sbin/shutdown -g5 -i5 “Bye bye Vince”‘ He must have thought that I was logged into my personal host machine, not the company’s cashcow server. What happens when it all goes wrong. Secure your session… Rgds Vince

Made a script that automatically removes all files from a directory. Now, rather than making it logically (this was early on) I did it stupidly.

cd /tmp/files
rm ./*

Of course, eventually someone removed /tmp/files..

I run a periodic (daily) script on a BSD system to clean out a temp directory for joe (the editor). Anything older than a day gets wiped out. For some historical reason the temp directory sits in /usr/joe-cache rather than in, for instance, /usr/local/joe-cache or /var/joe-cache or /tmp/joe-cache. The first version of the line in the script that does the deleting looked like this:


find /usr/joe-cache/.* -maxdepth 1 -mtime +1 -exec rm {} \;


Good thing the only files in /usr were two symlinks that were neither mission critical nor difficult to recreate as the above also matches “/usr/edit-cache/..” In the above the rather extraneous (joe doesn’t save backup files in sub-directories) “-maxdepth 1″ saved the entire /usr from being wiped out!

The revised version:

find -E /usr/joe-cache/ -regex '/usr/joe-cache/\.?.+$' -maxdepth 1 -mtime +1 -exec rm {} \;

.. which matches files beginning with a dot within “/usr/joe-cache”, but won’t match “/usr/joe-cache/..”

Lesson learned: always test find statements with “-print” before adding “-exec rm {} \;”.

mv /* /tmp (or some where else).

One more mistake that I do remember is copying a directory to another server but without using the recursive option. That copied the files found at the root but the files stored in the sub-folders were not copied.

I’ve most of these too :D

My favorite mistake was sitting around waiting for 250 or so gigs of stuff to copy to an nfs share, only to find I forgot to mount the remote share and it all just copied into the directory in /mnt instead. Good thing I had a huge root partition…

Whilst a colleague was away from their keyboard I entered :


rm -rf *

… but did not press enter on the last line (as a joke). I expected them to come back and see it as a joke and rofl….back space… The unthinkable happened, the screen went to sleep and they banged the enter key to wake it up a couple of times. We lost 3 days worth of business and some new clients. estimated cost $50,000+

tar cvf my_dir/* dir.tar
and your write your archive in the first file of the directory …

Attempting to update a Fedora box over the wire from Fedora8 to Fedora9
I updated the repositories to the Fedora9 repos, and ran
# yum -y upgrade
I have now tested this on a couple of boxes and without exception the upgrades failed with many loose older-version packages and dozens of missing dependencies, as well as some fun circular dependencies which cannot be resolved. By the time it is done, eth0 is disabled and a reboot will not get to the kernel-choice stage.

Oddly, this kind of update works great in Ubuntu.

# svn add foo
—-> Added 5 extra files that were not to be commited, so I decided to undo the change,delete the files and add to svn again…..
# svn rm foo –force

and it deleted the foo directory from disk :(…lost all my code just before the dead line :(

Typing rm -Rf /var/* in the wrong box. Recovered in few minutes by doing scp root@healty_box:/var . – the ssh session on the just broken box was still open . This saved my life :-P

This is why i never use pkill; always use something like “ps ….| grep …” and, when it’s ok, type a ” | awk ‘{print $2}’ | xargs kill” behind it. But, as a normal user, something like “pkill -v bash” might make perfect sense if you’re sitting at the console (so you can’t just switch to a different window or something) and have a background program rapidly filling your screen.

Worst thing that ever happened to me:
Our oracle database runs some rdbms jobs at midnight to clean out very old rows from various tables, along the line of “delete from XXXX where last_access < sysdate-3650". One sunday i installed ntp to all machines, made a start script that does an ntpdate first, then runs ntpd. Tested it:
$ date 010100002030; /etc/init.d/ntpd start; date
Worked great, current time was ok.
$ date 010100002030; reboot
After the machine was back up i noticed i had forgotten the /etc/rc*.d symlinks. But i never thought of the database until a lot of people were very angry monday morning. Fortunately, there's an automated backup every saturday.

while I was working on many ssh window:

rm -rf *

I intended to remove all files under a site, after changing the current working
directory, then replacing with the stable one

wrong window, wrong server, and I did it on production server xx((
just aware the mistakes 1.5 after typing [ENTER]
no backup. maybe luckily, the site was keep running smooth..

it seems that the deleted files were such images, or media contents
1-2 secs incidental removal in heavy machine gave me loss approx. 20 MB

After 10+ years I’ve made a lot of mistakes. Early on I got myself in the habit of testing commands before using them. For instance:

ls ~usr/tar/foo/bar then rm -f ~usr/tar/foo/bar – make sure you know what you will delete

When working with SSH, always make sure what system you are on. Modifying system prompts generally eliminates all confusion there.

It’s all just creating a habit of doing things safely… at least for me.

The deadline were coming too close to comfort, I’d worked for too looong hours for months.

We were developing a website, and I was in charge of developing the CGI scripts which generated a lot of temporary files, so on pure routine i worked in “/var/www/web/” and entered “rm temp/*” which i misspelled at some point as “rm tmp/ *”. I kind of wondered, in my overtired brain, what took so long for the delete to finish, it should only be 20 small files that is should delete.

The very next morning the paying client was to fly in and pay us a visit, and get a demonstration of the project.

P.S Thanks to Subversion and opened files in Emacs buffers I managed to get almost all files back, and I had rewritten the missing files before the morning.

Some of my more choice moments:

postsuper -d ALL (instead of -r ALL, adjacent keys – 80k spooled mails gone). No recovery possible – ramfs :/

Had a .pl script to delete mails in .Spam directories older than X days, didn’t put in enough error checking, some helpdesk guy provisioned a domain with a leading space in it and script rm’d (rm -rf /mailstore/ domain.com/.Spam/*) the whole mailstore. (250k users – 500GB used) – Hooray for 1 day old backup

chown -R named:named /var/named when there was a proc filesystem under /var/named/proc. Every running process on system got chown.. /bin/bash, /usr/sbin/sshd and so on. Took hours of manual find’s to fix.

.. and pretty much all the ones everyone else listed :)

You break it, you fix it.

second day on the job i rebooted apache on the live web server, forgetting to first check the cert password. i was finally able to find it in an obscure doc file after about 30 minutes. the resulting firestorm of angry clients would have made Nero proud. I was very, very surprised to find out I still had a job after that debacle.

lesson learned: keep your passwords secure, but handy

Hmm, most of these mistakes I have done – but my personal favourite.

# cd /usr/local/bin
# ls -l -> that displayed some binaries that I didn’t need / want.
# cd ..
# rm -Rf /bin
– Yeah, you guessed it – smoked the bin folder ! The system wasn’t happy after that. This is what happens when you are root and do something without reading the command before hitting [enter] late at night. First and last time …

so in recovering a binary backup of a large mysql database, produced by copying and tarballing ‘/var/lib/mysql’, I untarred it in tmp, and did the recovery without incident. (at 2am in the morning, when it went down). Feeling rather pleased with myself for suck a quick and successful recovery, I went to deltete the ‘var’ directory in ‘/tmp’ . I wanted to type:
rm -rf var/

instead I typed :
rm -rf /var

unfortunatley I didnt spot it for a while, and not until after did I realize that my on-site backups were stored in /var/backups …
IT was a truly miserable few days that followed while I pieced together the box from SVN and various other sources …

Heh,
These were great.
I have many above.. my first was
reboot
….Connection reset by peer. Unfortunately, I thought I was rebooting my desktop. Luckily, the performance test server I was on hadn’t been running tests(normally they can take 24-72 hours to run)..

symlinks… ack! I was cleaning up space and thought weird.. I don’t remember having a bunch of databases in this location.. rm -f * unfortunately, it was a symlink to my /db slice, that DID have my databases, friday afternoon fun.

I did a similar with being in the wrong directory… deleted all my mysql binaries.

This was also after we had acquired a company and the same happened on one of their servers months before.. we never realized that, and the server had an issue one dady… so we rebooted. Mysql had been running in memory for months, and upon reboot there was no more mysql. Took us a while to figure that out because no one had thought that the mysql binaries were GONE! Luckily I wasn’t the one who had deleted the binaries, just got to witness the aftermath.

The best ones are when you f*ck up and take down the production server and are then asked to investigate what happened and report on it to management….

Clearing up space used by no-longer-needed archive files:

# du -sh /home/myuser/oldserver/var
32G /home/myuser/oldserver/var
# cd /home/myuser/oldserver
# rm -rf /var

The box ran for 6 months after doing this, by the way, until I had to shut it down to upgrade the RAM…although of course all the mail, Web content, and cron jobs were gone. *sigh*

Remotely logged into a (Solaris) box at 3am. Made some changes that required a reboot. Being too lazy to even try and remember the difference between Solaris and Linux shutdown commands I decided to use init. I typed init 0…No one at work to hit the power switch for me so I had to make the 30 minute drive into work.
This one I chalked up to being a noob…I was on an XTerminal which was connected to a Solaris machine. I wanted to reboot the terminal due to display problems…Instead of just powering off the terminal I typed reboot on the commandline. I was logged in as root…

On a number of occasions, I typed “rm” when I wanted to type “mv,” i.e., I wanted to rename a file, but instead I deleted it.

I have a habit of renaming config files I work on to the same file with a “~” at the end for a backup, so that I can roll back if I make a mistake, and then once all is well I just do a rm *~. Trouble happened to me when I accidentally typed rm * ~ and as Murphy would have it a production asterisk telephony server.

Running out of diskspace while updating a kernel on FreeBSD.

Not fully inserting a memory module on my home machine which shortcircuited my motherboard.

On several occasions i had to use a rdesktop session to windows machine and use putty to connect to a machine (yep.. i know it sounds weird ;-) ) Anyway.. text copied in windows is stored differently than text copied in the shell. Why changing a root passwd on a box, (password copied using putty) i just control v-ed it and logged off. I had to go to the datacenter to boot into single user mode to acces the box again.

Using the same crappy setup, i copied some text in windows, accidently hit control-v in the putty screen of the box i was logged into as root, the first word was halt, the last character an enter.

Configuring nat on the wrong interface while connected through ssh

Adding a new interface on a machine, filled in the details of a home network in kudzu which changed the default gateway to 192.168.1.1 on the main interface. Only checking the output of ifconfig but not the traffic or gateway and dns settings.

fsck -y on filesystem without unmounting it

my personal favorite, a script somehow created few dozens file in /etc dit … all named ??somestrings so i promplty did rm -rf ??* … (at the point when i hit [enter] i remebered that ? is a wildchar … Too late :)) luckily that was my home box … but reinstall was imminent :)

also i’ve done

shutdown -n
(i thaught -n meant “now”)

which had for consequence to reboot the server without networking…

crontab -e vs crontab -r is the best :)

Worst mistake ever, ran fsck -y on an encrypted volume, trashed the whole partition. You can only check the filesystem after its unlocked and mounted as a logical volume.

heheh. yeah, i think we have all been there.

I always change the default prompt on server boxes, preferably to some flashing, garishly coloured text just to remind me i am not at home any more.


chmod -R www-data:www-data /etc/

Later on….

reboot

SSH never came back up. Wouldn't start with those permissions

You can always modify /etc/bash.bashrc and add:
alias rm"=rm -i"
Mine reads:
alias rm="
echo -------;
echo Think before you delete...;
echo Use yes on stdin or -f if you must bypass the prompt;
echo You have been warned!;
echo -------;
rm -i"

It’s not the best fix, and there are better ones out there, but it works fine for home desktops.

Copying many gigs of stuff with dd command. If you mix the source and destination, you very precisely make of copy of the blank disk over whatever it was you thought you were copying.

Doing a lot of copy/paste from installation manuals always risky. Like copy with this not too unusual prompt:

hostname :> /bin/sh

If you mistakenly copy not only /bin/sh you end up wiping out whatever executable you where trying to run ( in this case /bin/sh ) by doing

# :> /bin/sh

i dont consider myself a noob but i did the most noobish thing in the world today

ls -a > ls

Nothing wrong with it, except i was in the /bin directory testing some new shell scripts

I couldnt figure out why my ls command would return a blank on every folder.

One mistake I made was to run “slay” as an unprivileged user. This damn program by default (mean mode) will kill all your own processes in that case. It shouldn’t be shipped like that in a serious distro like Debian or Ubuntu, but they ignored my complaint.

instead of sudo sh -c ‘cat /etc/passwd > /etc/passwd.core’

i did:

sudo sh -c ‘cat > /etc/passwd /etc/passwd.core’

needless to say this made the system useless. i’m not sure why i was even doing this when i could have done sudo cp /etc/passwd /etc/passwd.core. it think it’s because i sometimes do sudo sh -c ‘head -25 /etc/passwd > /etc/passwd.core’. the lucky part was that this server was not in production yet.

One time I typed g++ -o program.cpp program.cpp instead of
g++ -o program program.cpp!

It was a final proyect program, so I had to write it all over again in a couple of hours.

One of my coworkers did: apt-get remove perl
At first sight it does not seem to dangerous, but it removed many programs, including the X environment and even the apt applications, according to him.

I used that command on my old computer when I got a new one, just to see what happened. :D

# cat /dev/nul >/etc/motd
came out for some reason as
cat /dev/nul >/etc/passwd
GOK why. A senior moment near going home time. Couldn’t find any old passwd files around so had to invent one with root in so I could log in at the console and extract the original from the backup tape. ps showed the users as numbers. No one noticed but my boss went into a sweat when I told him the next day :) SCO box. Last millennium!

hah! Hah!…majority seem to hv turtured “rm” to death. Surprisingly “rm” is still around…
anyways my personal goof-up, about twenty years back, when there was no way of knowing where u r in the filesystem – except using “pwd”…no bash, no helpful PS1 configured to show ur current location, etc. (Actually after this episode I created a small script to show the current location).

So here I was on a Xenix 286 system (apparently a unix “developed”/supplied by Microsoft), in single user mode and thinking I was in “/tmp”, issued “rm -r *” (during those days /tmp was just a simple directory)…

Well! Instead of “/tmp” I was in “/” …rest u can imagine…also in those days “rm” was much more powerful…I don’t recall whether “-f” switch was yet invented or for that matter “-i”. Or maybe I was unaware!…”rm” was pretty raw and did what it was asked to do – no questions asked!

But from that day I treat “rm” as my extra-marital wife…be real careful…

route -f xxx

I was trying to add a backup route to a primary sybase server and didn’t include the “hop count”. Since it didn’t work, I tried to “force” it and ended up deleting all routes from the routing table. :/

On a production Oracle RAC server, while connected to the console of one of the nodes through the system controller I typed:

~.

Which sent the server to OBP, halting the OS
instead of typing:

~~#

To exit console and go back to system controller.
Luckily that one time, the other node was up.

Is there a smart way to chmod an entire path, like mkdir -p ?
As I typed , I forgot to remove an usual -R :

cd /any/deep/path/to/allow/
chmod -R a+rx ../../../../..

oups. all the server was chmoded ….
luckily it was on a virtual machine. reload and reinstall.

Accidently rebooted servers…best is uname -a before you hit reboot command.
Wrongly plugged Sun M-series with one input 110V and 220v on the other. Server wouln’t start whatsover you do…
Best backup is dump or ufsdump (solaris), most of the time tar and cpio may lead you to loose your job.

@Otto — I did basically that to myself with sudo,

# sudo usermod -G groupname myusername

I was immediately no longer a member of ‘wheel’ and was no longer allowed to sudo.

tar -czf dirname1 &
tar -czf dirname2 &
…
tar -czf dirname13 &

#Seems like all of them have been tarred and gzipped

rm -rf dirname1
rm -rf dirname2
…
rm -rf dirname13

#Oh, some of the larger directories weren’t.

On a Personal PC, not so much a linux mistake though…

This was in the age of laptops moving to having NO more floppy drives, I needed some way of removing grub as it was no longer needed. Since there were no floppy drives I was searching for an alternative to the win98 bootdisk so I could run a “fdisk /mbr”.

Found a nice little program that I burnt to a disc, that would allow me to modify the boot sector.

… accidently ended up wiping the entire partition table!

I saw a fresh installed centos system broken by command:
yum remove python :)

This was years ago and now I think back about what an idiot I was.
I created a user, mike of course in a box running DEC 4.0E and I decided I wanted more permissions so I decided to make it user 0, just like root. The system actually asked if I was sure I wanted to do this. I said yes….It took me 2 hours to get the system back up and running because it changed every file in the system that was owned by root to mike.
dumb, real dumb.
Being superuser can be a disaster!
I use dd a lot now and make many backups.

On at least three entirely seperate occasions I’ve been on the command line within a C project I’ve been writing and gone to remove backup files:

rm *~

but been distracted after typing the asterisk and then switched back to the terminal and immediately pressed enter forgetting about the tilde sign. Immediately deleting weeks worth of *.c files.

Luckily I found ext3undelete, and while I did recover my files, it painfully insisted upon recovering the entire partition. Which of course happened to be the largest partition. So I then spent the rest of the evening not only waiting for it to restore the files I actually wanted, but manually deleting everything it restored which I had *not* deleted in the first place so the partition I was restoring to, did not run out of space before the files I actually deleted were restored!

A nice old one – not by me but by a less-than-worthy colleague, who liked to do everything as root on the server box:

WHAT HE MEANT : last | grep reboot

WHAT HE TYPED : last | reboot

I work as a student worker at a university and as a web developer there have some limited sudo access on the production web server. They upgraded to a new server and didn’t warn me that they hadn’t made bash the primary shell. So, naturally I typed the same commands I always have to change ownership of a folder to myself so I could make some changes to files and inevitably changed ownership of the whole server to myself! Oops!

sudo chown -R user folder [tab] / didn’t execute the same as sudo chown -R user folder/

Once I installed a Linux server and forgot to log out of the local console. I noticed this and foolishly decided to pkill all the users processes. The user happened to be root and one of its processes was sshd; thus meaning I locked myself out of the box.

Oh, I forgot the other really funny error… customer who wouldn’t use vi… copy /etc/passwd to a PC using Windows built-in ftp, add a user in to it in notepad and then use ftp to put it back… and added a ^M to all of the entries!

A call in late afternoon… ‘I can’t log on. Help!’

Can’t top the keystrokes, but think I’ve got a top 10 spot for consequences:

One of those ’start rebuilding the wrong box in a failed redundant pair’ scenarios.
Sadly it was the dispatch note / purchase order processing system for largest European lights-off central distribution center in a very big industry sector, and config had been ‘evolved’ or time.
Result: they couldn’t dispatch anything and had to turn away every delivery for a day and a half. We are talking about a hundred cross-continental trucks sent back to their depots empty, or still full.
I could tell you that it wasn’t me, but I don’t expect anyone to believe that.

Moral: Never, ever, ever trust a sticky label – look at the prompt, and if the prompt doesn’t tell you: change it so it does. SVN’ed config would have massively reduced down time too.