So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji
is in a sort of funky state. There's a section of the filesystem that is
pretty borked. Running fsck.ext3 on it causes a segfault. The core dump
is 2G right now. This is likely from the short few hours we ran /mnt/koji
from a RHEL4 box (lesson learned there...)

Steps to proceed:

1) Snapshots - I've been doing all of my tests from snapshots and will
continue to do so. The share itself doesn't seem to be catastrophically
broken, just that one section.

2) Contact some fs experts with the dump and see what is going on.

3) Fix the filesystem.

4) Also, for the last 4 nights at around 3 am my time the filesystem has
hit this part of the disk causing it to get remounted ro. I'd like to
setup a script to auto correct this issue until we can come up with a
permanent fix. Just saves me from having to get up in the middle of the
night.

Is anyone against this? I'm mostly worried about step3 then anything.
Before we do that I want to make sure we have a proper backup and do
plenty of tests from snapshots. It might be best to do 3) after the
release.

-Mike

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

On Mon, 2008-04-21 at 09:04 -0500, Mike McGrath wrote:
> So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji
> is in a sort of funky state. There's a section of the filesystem that is
> pretty borked. Running fsck.ext3 on it causes a segfault. The core dump
> is 2G right now. This is likely from the short few hours we ran /mnt/koji
> from a RHEL4 box (lesson learned there...)
>
> Steps to proceed:
>
> 1) Snapshots - I've been doing all of my tests from snapshots and will
> continue to do so. The share itself doesn't seem to be catastrophically
> broken, just that one section.
>
> 2) Contact some fs experts with the dump and see what is going on.
>
> 3) Fix the filesystem.
>
> 4) Also, for the last 4 nights at around 3 am my time the filesystem has
> hit this part of the disk causing it to get remounted ro. I'd like to
> setup a script to auto correct this issue until we can come up with a
> permanent fix. Just saves me from having to get up in the middle of the
> night.
>
> Is anyone against this? I'm mostly worried about step3 then anything.
> Before we do that I want to make sure we have a proper backup and do
> plenty of tests from snapshots. It might be best to do 3) after the
> release.
>

This gets a +1 from me. Must keep that thing stable, and you sane.

--
Jesse Keating
Fedora -- All my bits are free, are yours?
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

2008/4/21 Jesse Keating <jkeating@redhat.com>:
> On Mon, 2008-04-21 at 09:04 -0500, Mike McGrath wrote:
> > So we're in a bit of trouble. Right now the ext3 filesystem on /mnt/koji
> > is in a sort of funky state. There's a section of the filesystem that is
> > pretty borked. Running fsck.ext3 on it causes a segfault. The core dump
> > is 2G right now. This is likely from the short few hours we ran /mnt/koji
> > from a RHEL4 box (lesson learned there...)
> >
> > Steps to proceed:
> >
> > 1) Snapshots - I've been doing all of my tests from snapshots and will
> > continue to do so. The share itself doesn't seem to be catastrophically
> > broken, just that one section.
> >
> > 2) Contact some fs experts with the dump and see what is going on.
> >
> > 3) Fix the filesystem.
> >
> > 4) Also, for the last 4 nights at around 3 am my time the filesystem has
> > hit this part of the disk causing it to get remounted ro. I'd like to
> > setup a script to auto correct this issue until we can come up with a
> > permanent fix. Just saves me from having to get up in the middle of the
> > night.
> >
> > Is anyone against this? I'm mostly worried about step3 then anything.
> > Before we do that I want to make sure we have a proper backup and do
> > plenty of tests from snapshots. It might be best to do 3) after the
> > release.
> >
>
> This gets a +1 from me. Must keep that thing stable, and you sane.
>

Ah but Mike has viewed the depths of filesystem creation and seen the
dark Lords who live there dancing forever around the sleeping Nameless
One. Its too late.. the best we can do is wall him up in a room next
to Seth and Stephen Tweedie and let them chant their prayers to their
new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more
"O'yum rp'm" but they do go together as harmony.


--
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

On Mon, 2008-04-21 at 09:51 -0600, Stephen John Smoogen wrote:

> Ah but Mike has viewed the depths of filesystem creation and seen the
> dark Lords who live there dancing forever around the sleeping Nameless
> One. Its too late.. the best we can do is wall him up in a room next
> to Seth and Stephen Tweedie and let them chant their prayers to their
> new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more
> "O'yum rp'm" but they do go together as harmony.

Stephen,
The pills you're taking today are clearly just placebos. Tell the
doctors you'd like the 'good stuff' from now on. okay?

-sv


_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list

On Mon, Apr 21, 2008 at 10:02 AM, seth vidal <skvidal@fedoraproject.org> wrote:
> On Mon, 2008-04-21 at 09:51 -0600, Stephen John Smoogen wrote:
>
> > Ah but Mike has viewed the depths of filesystem creation and seen the
> > dark Lords who live there dancing forever around the sleeping Nameless
> > One. Its too late.. the best we can do is wall him up in a room next
> > to Seth and Stephen Tweedie and let them chant their prayers to their
> > new Mad Overlords: "Ph'sck mount'slafh Extphree" though Seth's is more
> > "O'yum rp'm" but they do go together as harmony.
>
> Stephen,
> The pills you're taking today are clearly just placebos. Tell the
> doctors you'd like the 'good stuff' from now on. okay?
>

Oh I am not falling for that trick again.. You said last week to tell
them to up the voltage.. haha very funny that hurt.

--
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list