||[folklore] [home] [search]|
Such things happened at least once to every unix person... To me it happened on February 1, 2000, after several years of heavy Unix usage/administration, when I was damn confident in myself and just leniently smiled on all these for-clueless-newbies warnings about not doing things as root.
In the middle of the working day, being a root on the main NFS server
containing all user homes, sitting in
/home/some_user, I typed
chown -R some_user .* and stopped it in 15-20 seconds
when realized that something is going wrong. But you know,
that server was really fast and permissions of the good half of the whole user
space have been modified.
(I recovered of course - by the price of my lunch time).
Anyway, the following classic article from Mario Wolczko describing much more interesting case first appeared on Usenet in 1986.
Have you ever left your terminal logged in, only to find when you came
back to it that a (supposed) friend had typed
rm -rf ~/* and was
hovering over the keyboard with threats along the lines of "lend me a
fiver 'til Thursday, or I hit return"? Undoubtedly the person in
question would not have had the nerve to inflict such a trauma upon
you, and was doing it in jest. So you've probably never experienced the
worst of such disasters...
It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15
BST, to be precise, when Peter, an office-mate of mine, leaned away
from his terminal and said to me, "Mario, I'm having a little trouble
sending mail." Knowing that msg was capable of confusing even the
most capable of people, I sauntered over to his terminal to see what
was wrong. A strange error message of the form (I forget the exact
details) "cannot access /foo/bar for userid 147" had been issued by
msg. My first thought was "Who's userid 147?; the sender of the
message, the destination, or what?" So I leant over to another
terminal, already logged in, and typed
grep 147 /etc/passwd
only to receive the response
/etc/passwd: No such file or directory.
Instantly, I guessed that something was amiss. This was confirmed
when in response to
ls /etc I got
ls: not found.
I suggested to Peter that it would be a good idea not to try anything for a while, and went off to find our system manager.
When I arrived at his office, his door was ajar, and within ten
seconds I realised what the problem was. James, our manager, was
sat down, head in hands, hands between knees, as one whose world has
just come to an end. Our newly-appointed system programmer, Neil, was
beside him, gazing listlessly at the screen of his terminal. And at
the top of the screen I spied the following lines:
# rm -rf *
Oh, shit, I thought. That would just about explain it.
I can't remember what happened in the succeeding minutes; my memory is
just a blur. I do remember trying ls (again), ps, who and maybe a few
other commands beside, all to no avail. The next thing I remember was
being at my terminal again (a multi-window graphics terminal), and
I owe a debt of thanks to David Korn for making echo a built-in of his
shell; needless to say,
/bin, together with
had been deleted. What transpired in the next few minutes was that
/lib had also gone in
their entirety; fortunately Neil had interrupted
rm while it was
somewhere down below
/users were all untouched.
Meanwhile James had made for our tape cupboard and had retrieved what
claimed to be a dump tape of the root filesystem, taken four weeks
earlier. The pressing question was, "How do we recover the contents
of the tape?". Not only had we lost
/etc/restore, but all of the
device entries for the tape deck had vanished. And where does mknod
live? You guessed it,
/etc. How about recovery across Ethernet of
any of this from another VAX? Well,
/bin/tar had gone, and
thoughtfully the Berkeley people had put
in the 4.3 distribution. What's more, none of the Ether stuff wanted to know
/etc/hosts at least. We found a version of
/usr/local, but that was unlikely to do us any
good without a tape deck.
Alternatively, we could get the boot tape out and rebuild the root filesystem, but neither James nor Neil had done that before, and we weren't sure that the first thing to happen would be that the whole disk would be re-formatted, losing all our user files. (We take dumps of the user files every Thursday; by Murphy's Law this had to happen on a Wednesday). Another solution might be to borrow a disk from another VAX, boot off that, and tidy up later, but that would have entailed calling the DEC engineer out, at the very least. We had a number of users in the final throes of writing up PhD theses and the loss of a maybe a weeks' work (not to mention the machine down time) was unthinkable.
So, what to do? The next idea was to write a program to make a device
descriptor for the tape deck, but we all know where
ld live. Or maybe make skeletal entries
so on, so that
/usr/bin/ftp would work. By sheer luck, I had a
emacs still running in one of my windows, which we could use to
create passwd, etc., but the first step was to create a directory to
put them in. Of course
/bin/mkdir had gone, and so had
/bin/mv, so we couldn't rename
/etc. However, this looked like a reasonable line of attack.
By now we had been joined by Alasdair, our resident UNIX guru, and as
luck would have it, someone who knows VAX assembler. So our plan
became this: write a program in assembler which would either rename
/etc, or make
/etc, assemble it on
another VAX, uuencode it, type in the uuencoded file using my gnu, uudecode it
(some bright spark had thought to put uudecode in
/usr/bin), run it,
and hey presto, it would all be plain sailing from there. By yet another
miracle of good fortune, the terminal from which the damage had been
done was still su'd to
su is in
/bin, remember?), so at least we stood a chance of all this working.
Off we set on our merry way, and within only an hour we had managed to
concoct the dozen or so lines of assembler to create
stripped binary was only 76 bytes long, so we converted it to hex
(slightly more readable than the output of
uuencode), and typed it
in using my editor. If any of you ever have the same problem, here's the
hex for future reference:
I had a handy program around (doesn't everybody?) for converting ASCII
hex to binary, and the output of
/usr/bin/sum tallied with our
original binary. But hang on - how do you set execute permission
/bin/chmod? A few seconds thought (which as usual, lasted a
couple of minutes) suggested that we write the binary on top of an
already existing binary, owned by me... problem solved.
So along we trotted to the terminal with the root login, carefully
remembered to set the
umask to 0 (so that I could create files in it
using my gnu), and ran the binary. So now we had a
by all. From there it was but a few easy steps to creating
protocols, (etc), and
ftp was willing to play ball. Then we recovered the contents
/bin across the ether (it's amazing how much you come to miss
ls after just a few, short hours), and selected files from
/etc. The key file was
/etc/rrestore, with which we
/dev from the dump tape, and the rest is history.
Now, you're asking yourself (as I am), what's the moral of this story?
Well, for one thing, you must always remember the immortal words,
DON'T PANIC. Our initial reaction was to reboot the machine and try
everything as single user, but it's unlikely it would have come up
/bin/sh. Rational thought saved
us from this one.
The next thing to remember is that UNIX tools really can be put to
unusual purposes. Even without my gnuemacs, we could have survived by
/usr/bin/grep as a substitute for
And the final thing is, it's amazing how much of the system you can
delete without it falling apart completely. Apart from the fact that
nobody could login (
/bin/login?), and most of the useful commands
had gone, everything else seemed normal. Of course, some things can't
stand life without say
/etc/utmp, but by and large it all hangs together.
I shall leave you with this question: if you were placed in the same situation, and had the presence of mind that always comes with hindsight, could you have got out of it in a simpler or easier way?