On the bright side, on Linux at least one can salvage a (wrongly) deleted file at times by knowing it is still open by a running process, SIGSTOPping that process just in case, examining /proc/THAT_PID/fd/ symlinks and cat(1) the contents of the needed one, conveniently marked as “(deleted)”, into a safe place.

I’m not a kernel guy either, only fixed iso9660 perms back in 1999 or so for localhost :) but I know freebsd FUD when I see it, and if you failed to find a distro that works for you (like I did in 2001), don’t blame “linux” for it — it’s just not professional in the first place. Ну, не стоит уши под лапшу подставлять и дальше её тиражировать.

Back to topic: on Linux it might be safer to check /proc/mounts and not `mount` in case one has to doublecheck: I once had a trouble with a recently cloned hard drive after a reboot, blasting the wrong one with dd(1) after having missed the *real* mounts state (don’t remember the details but LABEL and UUID won’t help to differentiate between bit-per-bit copies, obviously). There’s a tendency to have /etc/mtab just symlinked to /proc/mounts though.

rm -rf /

is _NOT_ as devastating as many think here it is (I would add that as unix certification exam question). Let me re- post here what I already posted:

“…
One thing I couldn’t buy as ultimately devastating though:
rm -rf /
- if I ever manage to do it as root on my *nix box I expect /bin, /boot, and part of /dev gone (and whatever else could be in / alphabetically before /dev on that box). Then the device hosting root filesystem will be deleted, and this will be end of my trouble. The rest: /home, /lib, /lib64, /sbin, /tmp, /usr, /var will stay intact. Other opinions?

Of course, you loose the system on the fly, but you can mount the partitions of the drive on another system, and you will see other filesystems mentioned above intact.

Another mistake which is possible: You are on a remote machine (via SSH) and think you are on your local machine. You want to re-partition your hard disk…

I did the killall thing once long ago on Solaris, too. It taught me to switch to using pkill everywhere (that had it).

Can’t tell you how many times I installed Solaris in the 90s with a fresh install, and then forgot to enable nis, and had to drive back to the data center afterwards because you couldn’t log in as root in telnet.

One thing I couldn’t buy as ultimately devastating though:
rm -rf /
- if I ever manage to do it as root on my *nix box I expect /bin, /boot, and part of /dev gone (and whatever else could be in / alphabetically before /dev on that box). Then the device hosting root filesystem will be deleted, and this will be end of my trouble. The rest: /home, /lib, /lib64, /sbin, /tmp, /usr, /var will stay intact. Other opinions?

Many of my mistakes could be less harmful to me — and others — if I learned it earlier that knowing the *weak* sides of what is available and avoiding them is way better than knowing the strong sides and just relying on them…

Thanks Vivek, it was a decent idea for a blog post to share the bumps earned and prompt us colleagues to do the same ;-)

Indeed, there’s such a thing as labels.. uuid’s and so on. Using a different OS for not knowing how to fix a problem when its readily available knowledge seems silly to me. All OS’s have their own flaws and strengths. So does however each human.

But that said – I used to love bsd and scoff at linux. I was even called an elitist by some and frankly I do not blame them. My attitude combined with really extreme sarcasm, I did seem like one! I think one ofthe biggest things I learned (after years of blindness) is to use what works for you! As it happens I much prefer gnu libc than bsd’s much less useful libc (maybe improved by now ?). And this is coming from someone who used tothink, say, C++ is bloated (more specifically OOP’s). My reasoning or even my “defence” was that even Linus says it is. A silly and stupid logic there (more like no logic). I knew that much (it was more of well ‘if he sees it..’). However, as a friend said to me: you mean the person who created a very bloated OS ? Its true, Linux has a lot of stuff that a lot of people will never use. That doesn’t make it bad or useles.

Point was well taken and I now love OOP (it’s very beneficial, its more type safe [in C++ versus say C], and really it gave me something new to learn!).

In other words: Linux has flaws. BSD has flaws. Windows has flaws. MacOS has flaws.. NOTHING is perfect. Use what works for you.

For instance : You know why you need to be root to chown a file even if it belongs to you ? Does breach of security mean anything to you ? Because as I recall, that would (could) be the end result. And yes this is very much related to permission issues.

With FreeBSD in particular, you could end up with a kernel panic *just* by going over IIRC 7 vinum’s software RAID volumes back in the day, or by plugging a USB storage device into a router not so long ago.

Just in case, I did mess with SATA drives too by plugging “the next one” in a socket with the lower number (that’s pretty easy when one got 6 or 8 of them but quite feasible with even 4 or 2 when there’s no decent light source, you see). *BUT* I didn’t reverse the dd if/of so far due to a habit of “fdisk -l” and other double-checking before things like that.

Please stop spreading that lame misinformation on device numbering as the bus scan order can change with BIOS, kernel, or kernel option *BUT* you’re wrong again, uchite matchast’ (e.g. udev’s KERNEL variable description for a start).

One of the fundamental mistakes a *NIX admin can do is listen carelessly for some local “authority” who would “back” their words by being loud and proud, and not by being actually knowledgeable and reasonable (the test is “why?”). That’s pretty wide-spread in Russian-speaking *BSD circles, unfortunately.

I think my favourite story is something that your (Jose’s) story reminds me of. My mom fixes databases and other problems. One time there was a database issue she was debugging. She was talking to the person on the phone. The person kept reading the error he was getting to my mom (over phone). Finally, my mom realized that he had actually printed out the file right – the file is what contained an error – not the printer itself! In other words, he printed out a file that contained the error and he thought that it was the printer having an issue. But hey, he learned from it and we all make mistakes at times. Those who say you should just fire them don’t realize that if that were the case they’d have no employees (and potentially law suits – yes, I’m serious. Some firings over things that may seem harmless can lead to lawsuits, whether its not a legit/unfounded case or not).

[valeri@point ~]$ mkdir test1
[valeri@point ~]$ mkdir -p test2/test1
[valeri@point ~]$ touch test2/test1/file
[valeri@point ~]$ rsync -avu test1 test2
sending incremental file list
test1/

sent 57 bytes received 16 bytes 146.00 bytes/sec
total size is 0 speedup is 0.00
[valeri@point ~]$ ls test2/test1/file
test2/test1/file

- you see, the “file” still exists! I use rsync for years, I never had anything trashed by it. But I _did_ my share of other flops described here!

Good point, which saved my skin a few times… “I’ll format that for the new backup partition. Well, I’ll just check if there’s anything on it and… wait, isn’t that my Windows partition? That’s not possible, because /dev/sdb is supposed to … oh, I plugged in that thing and… yikes.”

setuid(geteuid());
execvp(argv[1],argv+1);

Mine is extended by running the shell if run without parameters but I almost never leave such a shell open unused.
I even used this on some servers – being executable by only a certain group together with hiding it in odd locations provided high enough level of security (even if someone gained an access to one of the root-allowed users’ accounts, would have to know what to run).
- Anyway, what contributes most to the commandline safety is the distinction in prompt – I always set it up to show hostname[tty]:dir> for me and root@hostname[tty]:dir> for root, sometimes in different colour. This way I’m always warned that it’s a root-shell before issuing a command. Benefit of always seeing the hostname and directory prior to even starting to type a command or manipulating files does not need any explanation (or if hostname is not distinguishing enough, just use any string to identify the machine) and knowing the tty proved very useful when hunting processes.
So, the prompt consumes the whole line sometimes (in some deep subdir trees) but the information it provides is worth that space, and it’s more convenient and safe than to check for “pwd” or “hostname” each time, what you can forget sometimes (especially when tired, what is the worst time to run dangerous commands) :)

• During major reconfigurations (using a liveCD), I mount everything -r (readonly) unless the task demands I write into that partition.
• Mount a partition and have a nosey prior to hosing it (sometimes it won’t work coz it’s never had anything in it, but still: if the mount unexectedly works …)
• Prior to removing a large directory tree, chmod 000; and/or, mv -i; to a new name, and leave it for a few days, to see if anything stops working.
• rpm –test as a non-root user, prior to the real rpm.

Ironic: there’s an advert below this box saying OMG’S of the week (Telstra). Yes, it’s a bit like that.

• mv -i ## always use -i
• rm -i ## at least when you have your L-plates on (L-plates is Aus slang for “learner driver”).
• Only execute something as root if it won’t work otherwise; I generally have two windows open: my normal user and the root user, and I do things like ‘ls, cat, less’ in the normal window. Then I do things like vi, or rm as root (but I do a ls just before the rm, as root).
• I configure my root windows with a different background colour.
• pwd; prior to rm or recursive commands.
• prior to a recursive command on a small directory tree, find . -print; or, find . -type d -print; see how many files will be affected (or for slightly larger situations, the second one gives the range of directories to be affected).
• When I write a script to rename files or change them in a robotic manner, I put an echo in front of the intended command, so it will give me the commands that are going to happen. If satisfied, I remove the echo, to let the command actually happen.
• If I’m editing files in a robotic manner (by script), I create the new files with a backup extension (eg file.cxx becomes file.cxx.new). Then I diff the various original files against their new ones, to ensure it is as intended, then I rename the files, mv -i ; and keep my eyes peeled as I say yes to each one.
• The > is your enemy. If I want to create a brand new file by redirection, I … cat brand_new_file; (expect a not found error), then some-cmd > brand_new_file;
• If I want to append by redirection, I make a backup of the target file regardless.
• When I edit a config file in /etc (or anywhere), I … cp -i file.conf file.conf.20120116; named after today’s date. The dates aren’t that truthful, but it’s miles better than anything.
• Create dedicated users for running VMs and similar (especially if they are shared). That’s the approach Apache uses (usually automated).

1. SQL: DELETE FROM some_table; — missing WHERE clause. Bun fight for the next 3 hours for the trained monkeys at the client to get the backup tapes for a restore.
2. There is a Linux installation where almost EVERYONE has the same home directory (no need to store user-specific stuff on this machine, they all share the same .profile). From force of habit: userdel -r minion; ## because minion had left the company. -r wiped out the shared home dir for all users, including all the DBF files for the production Oracle database. About a day’s downtime.
3. The same guy made a second mistake: logged in remotely to a Unix server, he goes: ifdown eth0; ## then intended to ifup eth0; to pick up a config change. But he couldn’t get his prompt back after the first command, I wonder why ;-) He rang up and told someone to force the power off, then on.
4. An entire group of suburbs without phone service for a day: some vague rumour about something misconfigured.
5. One that I did myself: very early days of Unix – confusing rm instead of mv. It was only a single file deleted (the intended destination didn’t exist, and rm croaked with the error, which alerted me to the mistake). But it was still 2 hours recreating the C-program assignment from memory – some one else’s who I was helping.
6. Another one I did myself: swapping between the CVS storage area, and the equivalent working directory in my home directory: wanted to delete part of the working directory tree and start over: rm -fr stem_of_tree; ## but in the CVS area, not mine. Got someone to restore from backups. It was a pretty rudimentary CVS setup.

Like the time a customer support agent gave a customer the instruction (via email) to simply execute on his CentOS box [rpm -e XXX]. Apparently the customer didn’t understand so the customer support agent typed in [rpm -e ] and then pasted [rpm -e XXX]. I spent well over 6 hours getting RPM functional again and I can assure you it is not a fun experience.

Or the entertaining situation created when a customer support agent wanted to “grow” an existing partition for a customer on a second drive by creating a new partition with the same name and then mounted it over the existing partition. Nothing was lost, but you should have heard the customer scream about his “data loss”. Luckily, mounting a partition in this fashion doesn’t erase data but simply hides it – yet it still took several of us a couple of hours to figure out what had actually transpired…