Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On Wed, Sep 7, 2011 at 1:00 PM, Bob Proulx <bob@proulx.com> wrote:
> jacques wrote:
>> by error most of the binaries in /usr are erased (killing rm :-(
>
> Everyone has made that mistake at some point. *I know I have!

Not me! Though I did chmod -R /usr once. I noticed it immediately
and cancelled. Most of the commands were broken, though luckily tar
and scp still worked, so I copied over a backup and untarred it. This
anecdote is brought up whenever anyone suggests skipping /usr /bin in
backups is a good idea because the data doesn't change and would be
recovered by OS reinstall anyway (yes I've heard that argument).

Which brings me to another fun question. What's your worst
administration mistake and how did you recover?

-Bryan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAG367gZn9N+V8TUwAMFRa-p4cO6domHYEbMV_X3AY1mN18zKvw@mail.gmail.com">http://lists.debian.org/CAG367gZn9N+V8TUwAMFRa-p4cO6domHYEbMV_X3AY1mN18zKvw@mail.gmail.com

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

> On Wed, Sep 7, 2011 at 1:00 PM, Bob Proulx <bob@proulx.com> wrote:
> > jacques wrote:
> >> by error most of the binaries in /usr are erased (killing rm :-(
> >
> > Everyone has made that mistake at some point. I know I have!
>
> Not me! Though I did chmod -R /usr once. I noticed it immediately
> and cancelled. Most of the commands were broken, though luckily tar
> and scp still worked, so I copied over a backup and untarred it. This
> anecdote is brought up whenever anyone suggests skipping /usr /bin in
> backups is a good idea because the data doesn't change and would be
> recovered by OS reinstall anyway (yes I've heard that argument).
>
> Which brings me to another fun question. What's your worst
> administration mistake and how did you recover?

I once tried to change the ownership of all the files in a user
directory by doing something like "chown -R <newuser> .*" from within
the directory -- I've forgotten what exactly I typed, but my motive was
to get all the "." files included in the scope of the command.

Unfortunately, ".*" includes "..", so the chown command hopped
up to /home, and started switching the whole file system over
to be owned by the new user.

I caught it after it was taking a suspiciously long time, and
after a minute or two, I figured out what had happened.

Nothing focusses the mind quite like screwing up a live
server. After a brief but intense look at the man-page, I figured
out that what I wanted was "chown -hR <newuser> <path/to/target>",
and did that for both my initial task, and to fix the user directories
that had been messed up.

These days, I almost always use verbose options of commands,
if they exist, so I can verify that they're operating in the
expected scope.

-- A.
--
Andrew Reid / reidac@bellatlantic.net


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201109131925.28318.reidac@bellatlantic.net">http://lists.debian.org/201109131925.28318.reidac@bellatlantic.net

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On Tue, 13 Sep 2011 15:15:13 -0700, Bryan Irvine wrote:

> Which brings me to another fun question. What's your worst
> administration mistake and how did you recover?

The worst admin mistake is failure to secure proper backups. Full stop.



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: j4op6m$929$1@dough.gmane.org">http://lists.debian.org/j4op6m$929$1@dough.gmane.org

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On 14/09/11 08:15, Bryan Irvine wrote:
> On Wed, Sep 7, 2011 at 1:00 PM, Bob Proulx <bob@proulx.com> wrote:
>> jacques wrote:
>>> by error most of the binaries in /usr are erased (killing rm :-(
>>
<snipped>
>
> Which brings me to another fun question. What's your worst
> administration mistake and how did you recover?
>
> -Bryan
>
>
On a web server with a 3GB /dev/sda:-
dd if=/dev/sda | gzip | nc my_workbox 3333

At my workbox with a 1TB /dev/sda:-
nc -l 3333 | dd of=/dev/sda *instead* of:-
nc -l 3333 | dd of=bak_sda.img.gzip

first hint of idiocy was when I did a df -h to see how the transfer was
going, and got a screenful of gibberish... :-(
Not fun - cost me several hours to recover.

On the plus side I learnt a lot about testdisk, fsck, and ext4 journals :-)

Cheers

--
"I ascribe to Mark Twain's theory that the last person who should be
President is the one who wants it the most. The one who should be picked
is the one who should be dragged kicking and screaming into the White
House."
— Bill Hicks


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E700AE7.3070107@gmail.com">http://lists.debian.org/4E700AE7.3070107@gmail.com

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

Bryan Irvine:
>
> Which brings me to another fun question. What's your worst
> administration mistake and how did you recover?

I once typed mkfs instead of fsck. Well, at least the filesystem was
clean afterwards.

Lesson: don't start doing dangerous things just before bedtime and after
2-3 glasses of wine.

J.
--
People talking a foreign language are romantic and mysterious.
[Agree] [Disagree]
<http://www.slowlydownward.com/NODATA/data_enter2.html>

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On 09/14/2011 01:15 AM, Bryan Irvine wrote:

Which brings me to another fun question. What's your worst
administration mistake and how did you recover?


On ext4 resizing

- I did not put a swap partition on my laptop, just / and /home
- I decided to shrink /home to put a swap at the end
- fdisk
-- del & new smaller partition for /home, with same beginning
-- new partition at the end for swap (label 82)
- reboot
- fsck failed, resize2fs failed, all failed
- I forgot to backup during the week end
- I lost important documents...

There is a need of absolute "ext4 resizing tutorial"... :-P

--
RMA.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4E704918.9060003@rktmb.org">http://lists.debian.org/4E704918.9060003@rktmb.org

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On 09/14/2011 02:25 AM, Andrew Reid wrote:

These days, I almost always use verbose options of commands,
if they exist, so I can verify that they're operating in the
expected scope.


You will neeed graphic acceleration, then for displaying all the verbose
stuff ;-)


--
RMA.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4E70496C.9000003@rktmb.org">http://lists.debian.org/4E70496C.9000003@rktmb.org

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

Am Mittwoch, 14. September 2011 schrieb Andrew Reid:
> > On Wed, Sep 7, 2011 at 1:00 PM, Bob Proulx <bob@proulx.com> wrote:
> > > jacques wrote:
> > >> by error most of the binaries in /usr are erased (killing rm :-(
> > >
> > > Everyone has made that mistake at some point. I know I have!
> >
> > Not me! Though I did chmod -R /usr once. I noticed it immediately
> > and cancelled. Most of the commands were broken, though luckily tar
> > and scp still worked, so I copied over a backup and untarred it.
> > This anecdote is brought up whenever anyone suggests skipping /usr
> > /bin in backups is a good idea because the data doesn't change and
> > would be recovered by OS reinstall anyway (yes I've heard that
> > argument).
> >
> > Which brings me to another fun question. What's your worst
> > administration mistake and how did you recover?
>
> I once tried to change the ownership of all the files in a user
> directory by doing something like "chown -R <newuser> .*" from within
> the directory -- I've forgotten what exactly I typed, but my motive was
> to get all the "." files included in the scope of the command.
>
> Unfortunately, ".*" includes "..", so the chown command hopped
> up to /home, and started switching the whole file system over
> to be owned by the new user.
>
> I caught it after it was taking a suspiciously long time, and
> after a minute or two, I figured out what had happened.

In my Linux/UNIX basics training I have a slide about exactly that.

And another one about escaping. Consider rm * and rm "*". Or mkdir "A
directory with space" versus mkdir A directory with spaces.

And a recommendation to put echo in front of the command if unsure what
the shell does make out of the command line.

Cause I think I do not need to let my course participants run into data
loss experiences.

And yes, I did rm -rf .* once ;).

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201109140954.02316.Martin@lichtvoll.de">http://lists.debian.org/201109140954.02316.Martin@lichtvoll.de

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

Am Mittwoch, 14. September 2011 schrieb Mihamina Rakotomandimby:
> On 09/14/2011 01:15 AM, Bryan Irvine wrote:
> > Which brings me to another fun question. What's your worst
> > administration mistake and how did you recover?
>
> On ext4 resizing
>
> - I did not put a swap partition on my laptop, just / and /home
> - I decided to shrink /home to put a swap at the end
> - fdisk
> -- del & new smaller partition for /home, with same beginning
> -- new partition at the end for swap (label 82)
> - reboot
> - fsck failed, resize2fs failed, all failed
> - I forgot to backup during the week end
> - I lost important documents...
>
> There is a need of absolute "ext4 resizing tutorial"... :-P

Most important rule is the ordering:

1) On enlarging first enlarge the volume / partition, then grow the
filesystem

2) On shrinking do it the *other* way around when you are interested in
your data. So first shrink your filesystem and then shrink the volume /
partition, but not below the new size of the filesystem.

But I guess you know that by now ;)

--
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 201109140957.44874.Martin@lichtvoll.de">http://lists.debian.org/201109140957.44874.Martin@lichtvoll.de

Worst Admin Mistake? was --> /usr broken, will the machine reboot ?
 

On Wed, Sep 14, 2011 at 3:45 AM, Bryan Irvine <sparctacus@gmail.com> wrote:

> Which brings me to another fun question. *What's your worst
> administration mistake and how did you recover?
>
> -Bryan

drop database Live_database;

Restored the previous day's backup and blamed it on a bad power supply :P

--
The mysteries of the Universe are revealed when you break stuff.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: CAM8yCh8ABoFWT8hfynVAK1xPYyddr_Nsh4VBH2Xbim4vKaspw Q@mail.gmail.com">http://lists.debian.org/CAM8yCh8ABoFWT8hfynVAK1xPYyddr_Nsh4VBH2Xbim4vKaspw Q@mail.gmail.com