FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 09-28-2010, 05:44 AM
lrhorer
 
Default Mirror applications on Linux Server

If there is a better forum for this, let me know and I will post my
questions there.

I am building an application which needs to have high reliability. I
have two essentially identical Linux servers which can host the
application. Right now, I have the programs - a bash script and a c
binary, running on one machine every minute in a cron job. I also have
an rsync cron job running to synchronize the files on the standby
machine so the data (and binaries, of course) will be identical. What
I need to do is have the standby machine take over operations if the
applications on the primary machine quite working, for whatever reason.
Of course I can easily ping the primary to make sure the machine is up,
but what is going to be my best bet for having the standby machine wake
up and start running the apps every minute until such time as the
primary comes back online? I'm wide open on how to implement. An
external application would be great, or I could write either or both c
or shell apps to have the two machines talk to one another.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: he-dnansr_1KHTzRnZ2dnUVZ_sWdnZ2d@giganews.com">http://lists.debian.org/he-dnansr_1KHTzRnZ2dnUVZ_sWdnZ2d@giganews.com
 
Old 09-28-2010, 01:05 PM
Miles Fidelman
 
Default Mirror applications on Linux Server

lrhorer wrote:

If there is a better forum for this, let me know and I will post my
questions there.

I am building an application which needs to have high reliability. I
have two essentially identical Linux servers which can host the
application. Right now, I have the programs - a bash script and a c
binary, running on one machine every minute in a cron job. I also have
an rsync cron job running to synchronize the files on the standby
machine so the data (and binaries, of course) will be identical. What
I need to do is have the standby machine take over operations if the
applications on the primary machine quite working, for whatever reason.
Of course I can easily ping the primary to make sure the machine is up,
but what is going to be my best bet for having the standby machine wake
up and start running the apps every minute until such time as the
primary comes back online? I'm wide open on how to implement. An
external application would be great, or I could write either or both c
or shell apps to have the two machines talk to one another.

It may be overkill, but take a look at Pacemaker and the Linux-HA
Project - http://www.linux-ha.org - it's specifically intended for such
applications.


Also look at DRBD - www.drbd.org - which mirrors a disk (or partition),
in realtime across two machines.


The combination gives you automated fail-over capability.

Now, if you want to get really fancy, you can run your application in a
virtual machine, and use pacemaker and DRBD to fail-over the entire VM.


Be warned, it takes a while to get all of these working properly - both
individually and in combination. You could also take a look at ganeti -
http://code.google.com/p/ganeti/ - which pulls a bunch of the pieces
together.



--
In theory, there is no difference between theory and practice.
In<fnord> practice, there is. .... Yogi Berra



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4CA1E804.4020407@meetinghouse.net">http://lists.debian.org/4CA1E804.4020407@meetinghouse.net
 
Old 09-28-2010, 01:11 PM
Michal
 
Default Mirror applications on Linux Server

On 28/09/10 14:05, Miles Fidelman wrote:

lrhorer wrote:

If there is a better forum for this, let me know and I will post my
questions there.

I am building an application which needs to have high reliability. I
have two essentially identical Linux servers which can host the
application. Right now, I have the programs - a bash script and a c
binary, running on one machine every minute in a cron job. I also have
an rsync cron job running to synchronize the files on the standby
machine so the data (and binaries, of course) will be identical. What
I need to do is have the standby machine take over operations if the
applications on the primary machine quite working, for whatever reason.
Of course I can easily ping the primary to make sure the machine is up,
but what is going to be my best bet for having the standby machine wake
up and start running the apps every minute until such time as the
primary comes back online? I'm wide open on how to implement. An
external application would be great, or I could write either or both c
or shell apps to have the two machines talk to one another.
It may be overkill, but take a look at Pacemaker and the Linux-HA
Project - http://www.linux-ha.org - it's specifically intended for
such applications.


Also look at DRBD - www.drbd.org - which mirrors a disk (or
partition), in realtime across two machines.


The combination gives you automated fail-over capability.

Now, if you want to get really fancy, you can run your application in
a virtual machine, and use pacemaker and DRBD to fail-over the entire VM.


Be warned, it takes a while to get all of these working properly -
both individually and in combination. You could also take a look at
ganeti - http://code.google.com/p/ganeti/ - which pulls a bunch of the
pieces together.



DRBD and soforth might be overkill, or it might not. If you think it is
and are happy with the rsync you can stick to that and use heartbeat to
monitor the application. Easy to setup as well



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4CA1E977.2020707@sharescope.co.uk">http://lists.debian.org/4CA1E977.2020707@sharescope.co.uk
 
Old 09-29-2010, 12:47 AM
lrhorer
 
Default Mirror applications on Linux Server

Michal wrote:

> On 28/09/10 14:05, Miles Fidelman wrote:
>> lrhorer wrote:
>>> If there is a better forum for this, let me know and I will post my
>>> questions there.
>>>
>>> I am building an application which needs to have high reliability.
>>> I have two essentially identical Linux servers which can host the
>>> application. Right now, I have the programs - a bash script and a c
>>> binary, running on one machine every minute in a cron job. I also
>>> have an rsync cron job running to synchronize the files on the
>>> standby
>>> machine so the data (and binaries, of course) will be identical.
>>> What I need to do is have the standby machine take over operations
>>> if the applications on the primary machine quite working, for
>>> whatever reason. Of course I can easily ping the primary to make
>>> sure the machine is up, but what is going to be my best bet for
>>> having the standby machine wake up and start running the apps every
>>> minute until such time as the
>>> primary comes back online? I'm wide open on how to implement. An
>>> external application would be great, or I could write either or both
>>> c or shell apps to have the two machines talk to one another.
>> It may be overkill, but take a look at Pacemaker and the Linux-HA
>> Project - http://www.linux-ha.org - it's specifically intended for
>> such applications.
>>
>> Also look at DRBD - www.drbd.org - which mirrors a disk (or
>> partition), in realtime across two machines.
>>
>> The combination gives you automated fail-over capability.
>>
>> Now, if you want to get really fancy, you can run your application in
>> a virtual machine, and use pacemaker and DRBD to fail-over the entire
>> VM.
>>
>> Be warned, it takes a while to get all of these working properly -
>> both individually and in combination. You could also take a look at
>> ganeti - http://code.google.com/p/ganeti/ - which pulls a bunch of
>> the pieces together.
>>
>>
> DRBD and soforth might be overkill, or it might not. If you think it
> is and are happy with the rsync you can stick to that and use
> heartbeat to monitor the application. Easy to setup as well

Thanks. I skimmed the intro, and I'm not sure heartbeat is really what
I need. These aren't server applications that run full-time on the
machine. They are relatively simple programs that only take a few
seconds to run, and then terminate (hopefully with a return value of
0), to be run again in 60 seconds. I don't need the inter-machine
service to cause something to happen on the standby system if the apps
are no longer running, or at least not immediately so. Rather, I need
the standby machine to take over if:

1. The primary machine is no longer on the network (clearly the
heartbeat app can do this).

2. The applications being run by cron fail to run, say, 5 or 6 times in
a row.

3. One or more of the applications fails to terminate (hangs).

4. One or more of the applications terminates with other than a 0
status.

#4 may be the trickiest, because one of the apps may be spawned as a
detached process from the other. Can the heartbeat app handle this?
It sounded to me like it will pass the notification to the standby
system a few milliseconds after one the monitored processes ends. What
I need it to do is monitor the termination status of the programs.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: If6dnXovIOQ8ET_RnZ2dnUVZ_rOdnZ2d@giganews.com">htt p://lists.debian.org/If6dnXovIOQ8ET_RnZ2dnUVZ_rOdnZ2d@giganews.com
 
Old 09-29-2010, 01:10 AM
lrhorer
 
Default Mirror applications on Linux Server

Miles Fidelman wrote:

> lrhorer wrote:
>> If there is a better forum for this, let me know and I will post my
>> questions there.
>>
>> I am building an application which needs to have high reliability. I
>> have two essentially identical Linux servers which can host the
>> application. Right now, I have the programs - a bash script and a c
>> binary, running on one machine every minute in a cron job. I also
>> have an rsync cron job running to synchronize the files on the
>> standby
>> machine so the data (and binaries, of course) will be identical.
>> What I need to do is have the standby machine take over operations if
>> the applications on the primary machine quite working, for whatever
>> reason. Of course I can easily ping the primary to make sure the
>> machine is up, but what is going to be my best bet for having the
>> standby machine wake up and start running the apps every minute until
>> such time as the
>> primary comes back online? I'm wide open on how to implement. An
>> external application would be great, or I could write either or both
>> c or shell apps to have the two machines talk to one another.
>>
> It may be overkill, but take a look at Pacemaker and the Linux-HA
> Project - http://www.linux-ha.org - it's specifically intended for
> such applications.
>
> Also look at DRBD - www.drbd.org - which mirrors a disk (or
> partition), in realtime across two machines.
>
> The combination gives you automated fail-over capability.

No, definitely not necessary. The binaries and scripts, of course,
won't change very often, at all, after I am done with development. The
whole application suite - data and all - is also quite small.
Excluding the log files, the entire directory set is less than 300KB.
There are three sets of data files. One is user data, generated by CGI
scripts running on the web server. These only get modified at most
once or twice daily. They are not critical, and any daily changes
evaporate after at most a few hours, anyway. The second set is
schedule data, 9 small files which will usually only be changed
extremely rarely - once or twice a year at most. The third set is
real-time data, all of which is overwritten every minute, except for
some non-critical log files. It is critical the applications run
regularly, but it is not absolutely required the apps have perfectly
current data.

These applications run an HVAC system. Failure to turn on or off the
air handlers for 15 minutes or so won't create a big problem, but being
down for several hours will most definitely cause a major issue.

> Now, if you want to get really fancy, you can run your application in
> a virtual machine, and use pacemaker and DRBD to fail-over the entire
> VM.

No, I really don't want to get fancy. KISS is the order of the day,
here.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: If6dnXUvIORwDD_RnZ2dnUVZ_rMAAAAA@giganews.com">htt p://lists.debian.org/If6dnXUvIORwDD_RnZ2dnUVZ_rMAAAAA@giganews.com
 
Old 09-30-2010, 09:38 PM
Rob Owens
 
Default Mirror applications on Linux Server

On Tue, Sep 28, 2010 at 07:47:29PM -0500, lrhorer wrote:
> Michal wrote:
>
> > On 28/09/10 14:05, Miles Fidelman wrote:
> >> lrhorer wrote:
> >>> If there is a better forum for this, let me know and I will post my
> >>> questions there.
> >>>
> >>> I am building an application which needs to have high reliability.
> >>> I have two essentially identical Linux servers which can host the
> >>> application. Right now, I have the programs - a bash script and a c
> >>> binary, running on one machine every minute in a cron job. I also
> >>> have an rsync cron job running to synchronize the files on the
> >>> standby
> >>> machine so the data (and binaries, of course) will be identical.
> >>> What I need to do is have the standby machine take over operations
> >>> if the applications on the primary machine quite working, for
> >>> whatever reason. Of course I can easily ping the primary to make
> >>> sure the machine is up, but what is going to be my best bet for
> >>> having the standby machine wake up and start running the apps every
> >>> minute until such time as the
> >>> primary comes back online? I'm wide open on how to implement. An
> >>> external application would be great, or I could write either or both
> >>> c or shell apps to have the two machines talk to one another.
> >> It may be overkill, but take a look at Pacemaker and the Linux-HA
> >> Project - http://www.linux-ha.org - it's specifically intended for
> >> such applications.
> >>
> >> Also look at DRBD - www.drbd.org - which mirrors a disk (or
> >> partition), in realtime across two machines.
> >>
> >> The combination gives you automated fail-over capability.
> >>
> >> Now, if you want to get really fancy, you can run your application in
> >> a virtual machine, and use pacemaker and DRBD to fail-over the entire
> >> VM.
> >>
> >> Be warned, it takes a while to get all of these working properly -
> >> both individually and in combination. You could also take a look at
> >> ganeti - http://code.google.com/p/ganeti/ - which pulls a bunch of
> >> the pieces together.
> >>
> >>
> > DRBD and soforth might be overkill, or it might not. If you think it
> > is and are happy with the rsync you can stick to that and use
> > heartbeat to monitor the application. Easy to setup as well
>
> Thanks. I skimmed the intro, and I'm not sure heartbeat is really what
> I need. These aren't server applications that run full-time on the
> machine. They are relatively simple programs that only take a few
> seconds to run, and then terminate (hopefully with a return value of
> 0), to be run again in 60 seconds. I don't need the inter-machine
> service to cause something to happen on the standby system if the apps
> are no longer running, or at least not immediately so. Rather, I need
> the standby machine to take over if:
>
> 1. The primary machine is no longer on the network (clearly the
> heartbeat app can do this).
>
> 2. The applications being run by cron fail to run, say, 5 or 6 times in
> a row.
>
> 3. One or more of the applications fails to terminate (hangs).
>
> 4. One or more of the applications terminates with other than a 0
> status.
>
How about this:

Every time Server A does its thing, it signals success to Server B.

Server B runs a timer script that resets every time it receives the
"success" signal from Server A. If it doesn't receive the signal, it
"does its thing" and signals success to Server A.

Server A is also running a timer script, and it resets when it gets the
success signal from Server B.

So now Server B will act as "primary", and Server A will just wait until
Server B fails to send a success signal. ...Unless you adjust the
timing of the script and the cron job. For instance:

Server A could be set to do its thing every 5 minutes, assuming it has
not received a success signal from Server B for the last 4 minutes.
This gives Server A the opportunity to become primary again after Server
B had been doing its thing.

Server B would be set to do its thing every 5 minutes, assuming it has
not received a success signal from Server A for the last 5 minutes (or 6
minutes, or whatever seems reasonable).

Sorry I can't help w/ the timer script. But I think something like "ssh
serverA touch timerfile" would be sufficient to reset the timer. Then
your script could check for the timestamp on "timerfile".

-Rob


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20100930213758.GA3080@aurora.owens.net">http://lists.debian.org/20100930213758.GA3080@aurora.owens.net
 
Old 09-30-2010, 10:11 PM
Miles Fidelman
 
Default Mirror applications on Linux Server

lrhorer wrote:

I am building an application which needs to have high reliability.
I have two essentially identical Linux servers which can host the
application. Right now, I have the programs - a bash script and a c
binary, running on one machine every minute in a cron job. I also
have an rsync cron job running to synchronize the files on the
standby machine so the data (and binaries, of course) will be identical.

Thanks. I skimmed the intro, and I'm not sure heartbeat is really what
I need. These aren't server applications that run full-time on the
machine. They are relatively simple programs that only take a few
seconds to run, and then terminate (hopefully with a return value of
0), to be run again in 60 seconds. I don't need the inter-machine
service to cause something to happen on the standby system if the apps
are no longer running, or at least not immediately so. Rather, I need
the standby machine to take over if:


This might work:

1. Use DRBD to keep your disks (or specific volumes) in sync. Run it in
it's standard primary-secondary mode.


2. Run the job on both machines. When it starts, have it check whether
the volume is in primary or secondary mode; if in secondary mode, abort.


Of course that only works if your backup machine doesn't need to access
the disk volume for other things.




--
In theory, there is no difference between theory and practice.
In<fnord> practice, there is. .... Yogi Berra



--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Archive: 4CA50B1E.50605@meetinghouse.net">http://lists.debian.org/4CA50B1E.50605@meetinghouse.net
 
Old 10-02-2010, 04:55 AM
lrhorer
 
Default Mirror applications on Linux Server

Rob Owens wrote:

> On Tue, Sep 28, 2010 at 07:47:29PM -0500, lrhorer wrote:
>> Michal wrote:
>>
>> > On 28/09/10 14:05, Miles Fidelman wrote:
>> >> lrhorer wrote:
>> >>> If there is a better forum for this, let me know and I will post
>> >>> my questions there.
>> >>>
>> >>> I am building an application which needs to have high
>> >>> reliability. I have two essentially identical Linux servers which
>> >>> can host the
>> >>> application. Right now, I have the programs - a bash script and
>> >>> a c
>> >>> binary, running on one machine every minute in a cron job. I
>> >>> also have an rsync cron job running to synchronize the files on
>> >>> the standby
>> >>> machine so the data (and binaries, of course) will be identical.
>> >>> What I need to do is have the standby machine take over
>> >>> operations if the applications on the primary machine quite
>> >>> working, for whatever reason. Of course I can easily ping the
>> >>> primary to make sure the machine is up, but what is going to be
>> >>> my best bet for having the standby machine wake up and start
>> >>> running the apps every minute until such time as the
>> >>> primary comes back online? I'm wide open on how to implement.
>> >>> An external application would be great, or I could write either
>> >>> or both c or shell apps to have the two machines talk to one
>> >>> another.
>> >> It may be overkill, but take a look at Pacemaker and the Linux-HA
>> >> Project - http://www.linux-ha.org - it's specifically intended for
>> >> such applications.
>> >>
>> >> Also look at DRBD - www.drbd.org - which mirrors a disk (or
>> >> partition), in realtime across two machines.
>> >>
>> >> The combination gives you automated fail-over capability.
>> >>
>> >> Now, if you want to get really fancy, you can run your application
>> >> in a virtual machine, and use pacemaker and DRBD to fail-over the
>> >> entire VM.
>> >>
>> >> Be warned, it takes a while to get all of these working properly -
>> >> both individually and in combination. You could also take a look
>> >> at ganeti - http://code.google.com/p/ganeti/ - which pulls a bunch
>> >> of the pieces together.
>> >>
>> >>
>> > DRBD and soforth might be overkill, or it might not. If you think
>> > it is and are happy with the rsync you can stick to that and use
>> > heartbeat to monitor the application. Easy to setup as well
>>
>> Thanks. I skimmed the intro, and I'm not sure heartbeat is really
>> what
>> I need. These aren't server applications that run full-time on the
>> machine. They are relatively simple programs that only take a few
>> seconds to run, and then terminate (hopefully with a return value of
>> 0), to be run again in 60 seconds. I don't need the inter-machine
>> service to cause something to happen on the standby system if the
>> apps
>> are no longer running, or at least not immediately so. Rather, I
>> need the standby machine to take over if:
>>
>> 1. The primary machine is no longer on the network (clearly the
>> heartbeat app can do this).
>>
>> 2. The applications being run by cron fail to run, say, 5 or 6 times
>> in a row.
>>
>> 3. One or more of the applications fails to terminate (hangs).
>>
>> 4. One or more of the applications terminates with other than a 0
>> status.
>>
> How about this:
>
> Every time Server A does its thing, it signals success to Server B.
>
> Server B runs a timer script that resets every time it receives the
> "success" signal from Server A. If it doesn't receive the signal, it
> "does its thing" and signals success to Server A.

Yeah, I was thinking of writing a simple client / server pair in c that
opens a UDP socket and sends a status from the client on B to the
server on A.

> Server A is also running a timer script, and it resets when it gets
> the success signal from Server B.

'Not needed. Server B will be trying to run the process, regardless. If
it works, it signals A, which is inhibited from doing anything. If it
fails for whatever reason, then A will take over when it doesn't get an
update. If server A croaks completely, B doesn't really care. It will
still be happily running the process. If any aspect of server B's
processes fail, A does its bit.

> So now Server B will act as "primary", and Server A will just wait
> until
> Server B fails to send a success signal. ...Unless you adjust the
> timing of the script and the cron job. For instance:

The process doesn't care in the least which server is actually running,
so there is no need to "flip-flop" server rolls. There are certainly
cases where one server must actually assume the mantle of the other in
order to take over, particularly, for example, if external processes
are seeking to initiate conversations with the server, but that is not
the case, here. The active server initiates all conversations with the
subordinate host. Indeed, for this application, it would mostly not
really hurt if both machines ran the processes, as long as both did not
attempt to run them at the exact same time. There are a few
scenarios, however, where running the process on both machines a few
seconds apart could produce some undesirable results, so I am going to
inhibit machine A from doing anything as long as machine B is alive and
well.

> Server A could be set to do its thing every 5 minutes, assuming it has
> not received a success signal from Server B for the last 4 minutes.
> This gives Server A the opportunity to become primary again after
> Server B had been doing its thing.
>
> Server B would be set to do its thing every 5 minutes, assuming it has
> not received a success signal from Server A for the last 5 minutes (or
> 6 minutes, or whatever seems reasonable).
>
> Sorry I can't help w/ the timer script. But I think something like
> "ssh
> serverA touch timerfile" would be sufficient to reset the timer. Then
> your script could check for the timestamp on "timerfile".

Yeah, you know, that would work. Creating a timer is no big deal. If I
use your ssh example, I could simply do something like

find -cmin +5 -name timerfile -exec <Process> ;

Since both processes are running every minute under cron, nothing more
is needed.

As long as server B is alive and talking, timerfile will never be older
than 1 minute. If B goes down for 4+ minutes, <Process> will run every
minute on A. Once B comes back, timerfile will once again be young.
As long as I add a little delay on A or B so the two processes don't
collide endlessly, it should work.

Hmm.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: xumdnZVRHcbFJjvRnZ2dnUVZ_uSdnZ2d@giganews.com">htt p://lists.debian.org/xumdnZVRHcbFJjvRnZ2dnUVZ_uSdnZ2d@giganews.com
 

Thread Tools




All times are GMT. The time now is 12:16 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org