Iain Buchanan wrote:
> On Sat, 2008-04-05 at 12:42 -0400, Steven Lembark wrote:
>>> I tried ALT + SysRq + EISUB today on my MythTV backend server which
>>> has been crashing lately. Unfortunately it's crashing so badly that
>>> even at the server's keyboard this didn't work.
>>>
>>> I guess my weekend fate of building a new server is sealed...
>> Have fun.
>>
>> Check out motherboards with watchdog capability
>> and enable it in the kernel.
>
> watchdogs are nice, and linux makes them ultra-easy to program, but of
> course if your watchdog task dies, then the machine effectively hits the
> reset button for you - no nice shutdown whatsoever! (Which is what you
> want in a hard lock-up, but not if your programming skills are the cause
> of the problem

- Have the system turn off the watchdog if the file is
closed.

- After that just open it and poke a bit out now and
then.

- Make a point of closing the file on exit.

#!/usr/bin/perl

use strict;

open my $fh, '<', '/path/to/watchdog/file'
or die "Failed opening watchdog file: $!";

# watchdog is now watching...

select $fh;

for(;
{
print "
";

sleep 1; # watchdog timeout / 2
}

my $graceful_exit
= sub
{
close $fh;

exit 0
};

for sig in ( qw( TERM QUIT INT __DIE__ ) )
{
$SIG{ $sig } = $graceful_exit;
}

for sig in ( qw( HUP ) )
{
$SIG{ $SIG } = 'IGNORE';
}

__END__

--
Steven Lembark 85-09 90th St.
Workhorse Computing Woodhaven, NY, 11421
lembark@wrkhors.com +1 888 359 3508
--
gentoo-user@lists.gentoo.org mailing list

>> Main thing
>> that speeds up the AMD box is using 320MB scsi's
>> for near-term storage. They are hugely faster than
>> [S]ATA or IDE used on most equipment these days.
>
> what R/W speeds can you expect?

Operations on SCSI run 2-3 times faster for large-ish
file transfers (say 1MB or more). For example GIMP
runs without noticeable pauses on most operations with
my 2GHz opteron using the SCSI's as storage and scratch
space; on the same machine with the SATA's it bogs down
to the point of annoyance, with things like saves taking
seconds to complete. The only difference being which
drives I'm using for tiles and data.

Bonnie gets upset validating the scsi space on this
box since it has 8GB of core and I don't have 16GB
free on the device so I can't give you any specific
numbers.


--
Steven Lembark 85-09 90th St.
Workhorse Computing Woodhaven, NY, 11421
lembark@wrkhors.com +1 888 359 3508
--
gentoo-user@lists.gentoo.org mailing list

On Mon, 2008-04-07 at 13:28 -0400, Steven Lembark wrote:
> Iain Buchanan wrote:

> > watchdogs are nice, and linux makes them ultra-easy to program, but of
> > course if your watchdog task dies, then the machine effectively hits the
> > reset button for you - no nice shutdown whatsoever! (Which is what you
> > want in a hard lock-up, but not if your programming skills are the cause
> > of the problem
>
> - Have the system turn off the watchdog if the file is
> closed.

maybe, maybe not I personally like setting CONFIG_WATCHDOG_NOWAYOUT
on systems with hardware watchdogs, especially remote unattended
systems. Usually your watchdog task never dies on such a system, and
when it does (be it from a nice kill or not) you want the watchdog to
fire. However if this is a semi-used system (you ssh or log-in to it in
any way to do stuff) you may not want this.

> - After that just open it and poke a bit out now and
> then.
>
> - Make a point of closing the file on exit.
>
> #!/usr/bin/perl
>
> use strict;
>
> open my $fh, '<', '/path/to/watchdog/file'

it's usually /dev/watchdog if you're using the linux kernel interface.

I agree that your script is nice and simple, and hence less prone to
errors. I coded mine in c++ because I use it not only for a machine
type watchdog, but also a task based watchdog that reboots the machine
based on certain tasks living or not. Each task has to register with
the watchdog server and continually tell the server they're alive, or
reboot! But that's a story for another thread...
--
Iain Buchanan <iaindb at netspace dot net dot au>

Linux - Where do you want to fly today?
-- Unknown source

--
gentoo-user@lists.gentoo.org mailing list

> I agree that your script is nice and simple, and hence less prone to
> errors. I coded mine in c++ because I use it not only for a machine
> type watchdog, but also a task based watchdog that reboots the machine
> based on certain tasks living or not. Each task has to register with
> the watchdog server and continually tell the server they're alive, or
> reboot! But that's a story for another thread...

#!/path/to/perl

use strict;

use Sys::Syslog;

open my $fh, '>', '/dev/watchdog'
or die "/dev/watchdog: $!";

# if any of these go away we need to notice it.
# ok... you'll notice the first one anyway.

my @watchz
= qw
(
init
ntpd
apache
/opt/sybase/ASE-12_5/bin/dataserver
);

# wd timeout / 2, or 1 for minimum sleep
# (avoid usleep: too much overhead).

my $cycle = 15;

# get the syslog handle

openlog blah blah blah
or die 'Et tu, syslog?';

CYCLE:
for(;
{
sleep ( $cycle - ( time % $cycle ) );

# split and args vary by O/S, this works on linux.

my @procz = map { split /s+/, $_, 6 )[5] } qx( ps a );

my %chechz = ();

@chechz{ @watchz } = ();

delete @chechz{ @procz };

if( %chechz )
{
# oops, current proc's don't include the
# list of processes being watched.
#
# this can happen twice in a w/d interval
# before the system goes down.

my $nastygram
= join " ", 'Missing proc's:', join " ", keys %chechz

syslog LOG_CRIT | LOG_FOO, $nastygram;

next CYCLE

# alternative here is to close $fh here and
# bounce the system immediately, the
# approach of looping allows an
# intentional restart of the service
# (in less than 1 w/d cycle) w/o bouncing the box.
}

# if the proc check got this far then the w/d
# file gets poked and we live for another loop.

print $wd "
";
}

# this isn't a module

0

__END__

--
Steven Lembark 85-09 90th St.
Workhorse Computing Woodhaven, NY, 11421
lembark@wrkhors.com +1 888 359 3508
--
gentoo-user@lists.gentoo.org mailing list