FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor


 
 
LinkBack Thread Tools
 
Old 02-20-2009, 08:44 PM
David Teigland
 
Default unfencing

Fencing devices that do not reboot a node, but just cut off storage have
always required the impractical step of re-enabling storage access after the
node has been reset. We've never provided a mechanism to automate this
unfencing.

Below is an outline of how we might automate unfencing with some simple
extensions to the existing fencing library, config scheme and agents. It does
not involve the fencing daemon (fenced). Nodes would unfence themselves when
they start up. We might also consider a scheme where a node is unfenced by
*other* nodes when it starts up, if that has any advantage over
self-unfencing.

cluster3 is the context, but a similar thing would apply to a next generation
unified fencing system, e.g.
https://www.redhat.com/archives/cluster-devel/2008-October/msg00005.html

init.d/cman would run:
cman_tool join
fence_node -U <ourname>
qdiskd
groupd
fenced
dlm_controld
gfs_controld
fence_tool join

The new step fence_node -U <name> would call libfence:fence_node_undo(name).
[fence_node <name> currently calls libfence:fence_node(name) to fence a node.]

libfence:fence_node_undo(node_name) logic:
for each device_name under given node_name,
if an unfencedevice exists with name=device_name, then
run the unfencedevice agent with first arg of "undo"
and other args the normal combination of node and device args
(any agent used with unfencing must recognize/support "undo")

[logic derived from cluster.conf structure and similar to fence_node logic]

Example 1:

<clusternode name="foo" nodeid="3">
<fence>
<method="1">
<device name="san" node="foo"/>
</method>
</fence>
</clusternode>

<fencedevices>
<fencedevice name="san" agent="fence_scsi"/>
</fencedevices>

<unfencedevices>
<unfencedevice name="san" agent="fence_scsi"/>
</unfencedevices>

fence_node_undo("foo") would:
- fork fence_scsi
- pass arg string: undo node="foo" agent="fence_scsi"

[Note: we've talked about fence_scsi getting a device list from
/etc/cluster/fence_scsi.conf instead of from clvm. It would require
more user configuration, but would create fewer problems and should
be more robust.]

Example 2:

<clusternode name="bar" nodeid="4">
<fence>
<method="1">
<device name="switch1" port="4"/>
<device name="switch2" port="6"/>
</method>
<method="2">
<device name="apc" port="4"/>
</method>
</fence>
</clusternode>

<fencedevices>
<fencedevice name="switch1" agent="fence_brocade" ipaddr="1.1.1.1"/>
<fencedevice name="switch2" agent="fence_brocade" ipaddr="2.2.2.2"/>
<fencedevice name="apc" agent="fence_apc" ipaddr="3.3.3.3"/>
</fencedevices>

<unfencedevices>
<unfencedevice name="switch1" agent="fence_brocade" ipaddr="1.1.1.1"/>
<unfencedevice name="switch2" agent="fence_brocade" ipaddr="2.2.2.2"/>
</unfencedevices>

fence_node_undo("bar") would:
- fork fence_brocade
- pass arg string: undo port="4" agent="fence_brocade" ipaddr="1.1.1.1"
- fork fence_brocade
- pass arg string: undo port="6" agent="fence_brocade" ipaddr="2.2.2.2"
- ignore device "apc" because it's not found under <unfencedevices>
 
Old 02-23-2009, 05:27 AM
"Fabio M. Di Nitto"
 
Default unfencing

Hi David,

On Fri, 2009-02-20 at 15:44 -0600, David Teigland wrote:
> Fencing devices that do not reboot a node, but just cut off storage have
> always required the impractical step of re-enabling storage access after the
> node has been reset. We've never provided a mechanism to automate this
> unfencing.
>
> Below is an outline of how we might automate unfencing with some simple
> extensions to the existing fencing library, config scheme and agents. It does
> not involve the fencing daemon (fenced). Nodes would unfence themselves when
> they start up. We might also consider a scheme where a node is unfenced by
> *other* nodes when it starts up, if that has any advantage over
> self-unfencing.

Use case where we need remote unfencing is to recover nodes that boot
from the shared storage and those are not that uncommon.

I personally don't like the idea of exposing a -U option to users. It's
a short cut that could be easily misused in an attempt to recover a node
and make more damage than anything else, but I can't see another
solution either.

> cluster3 is the context, but a similar thing would apply to a next generation
> unified fencing system, e.g.
> https://www.redhat.com/archives/cluster-devel/2008-October/msg00005.html
>
> init.d/cman would run:
> cman_tool join
> fence_node -U <ourname>
> qdiskd
> groupd
> fenced
> dlm_controld
> gfs_controld
> fence_tool join
>
> The new step fence_node -U <name> would call libfence:fence_node_undo(name).
> [fence_node <name> currently calls libfence:fence_node(name) to fence a node.]
>
> libfence:fence_node_undo(node_name) logic:
> for each device_name under given node_name,
> if an unfencedevice exists with name=device_name, then
> run the unfencedevice agent with first arg of "undo"
> and other args the normal combination of node and device args
> (any agent used with unfencing must recognize/support "undo")

All our agents already support on/off enable/disable operations. It's
probably best to align them to have the same config options rather than
adding a new one across the board.

>
> [logic derived from cluster.conf structure and similar to fence_node logic]
>
> Example 1:
>
> <clusternode name="foo" nodeid="3">
> <fence>
> <method="1">
> <device name="san" node="foo"/>
> </method>
> </fence>
> </clusternode>
>
> <fencedevices>
> <fencedevice name="san" agent="fence_scsi"/>
> </fencedevices>
>
> <unfencedevices>
> <unfencedevice name="san" agent="fence_scsi"/>
> </unfencedevices>

I think that we can avoid the whole <unfence* structure either by
overriding the default action="" for that fence method or possibly
consider unfencing a special case method. The idea is to contain the
whole fence config for the node within the <clusternode> object rather
than spreading it even more.

For e.g.:

<method name="1">
<device name="san" node="foo"/>
</method>
<method name="unfence">
...
</method>

OR

<method name="1">
<device name="san" node="foo"/>
</method>
<method name="2" operation="unfence">
...
</method>

(clearly names and format are up for discussion)


>
> [Note: we've talked about fence_scsi getting a device list from
> /etc/cluster/fence_scsi.conf instead of from clvm. It would require
> more user configuration, but would create fewer problems and should
> be more robust.]

I think we should really consider firing up a separate thread for this.
It seems to be a more and more often recurring issue.

Fabio
 
Old 02-23-2009, 05:15 PM
David Teigland
 
Default unfencing

On Mon, Feb 23, 2009 at 07:27:20AM +0100, Fabio M. Di Nitto wrote:
> > libfence:fence_node_undo(node_name) logic:
> > for each device_name under given node_name,
> > if an unfencedevice exists with name=device_name, then
> > run the unfencedevice agent with first arg of "undo"
> > and other args the normal combination of node and device args
> > (any agent used with unfencing must recognize/support "undo")
>
> All our agents already support on/off enable/disable operations. It's
> probably best to align them to have the same config options rather than
> adding a new one across the board.

Yes, I have those options in mind, and would prefer to use them as well.
We'll have to wait and see during the implementation phase; for the time being
they complicate things, so I'm using "undo" to avoid those details.

(I did reuse those options back in my first unfencing attempt which I
eventually removed:
http://git.fedorahosted.org/git/cluster.git?p=cluster.git;a=commitdiff;h=c781fbb6d f57f9780cdadf42126cdcc9a2ff3878)


> > <clusternode name="foo" nodeid="3">
> > <fence>
> > <method="1">
> > <device name="san" node="foo"/>
> > </method>
> > </fence>
> > </clusternode>
> >
> > <fencedevices>
> > <fencedevice name="san" agent="fence_scsi"/>
> > </fencedevices>
> >
> > <unfencedevices>
> > <unfencedevice name="san" agent="fence_scsi"/>
> > </unfencedevices>
>
> I think that we can avoid the whole <unfence* structure either by
> overriding the default action="" for that fence method or possibly
> consider unfencing a special case method. The idea is to contain the
> whole fence config for the node within the <clusternode> object rather
> than spreading it even more.
>
> For e.g.:
>
> <method name="1">
> <device name="san" node="foo"/>
> </method>
> <method name="unfence">
> ...
> </method>
>
> OR
>
> <method name="1">
> <device name="san" node="foo"/>
> </method>
> <method name="2" operation="unfence">
> ...
> </method>
>
> (clearly names and format are up for discussion)

The meanings of those fencing structures have never changed since being
introduced many years ago, and both of those fundamentally change it. It
would be very unfortunate to redefine them.

A good alternative to <unfencedevices> would be an <unfence> section within
the node setions (it would not require a method level).... Now that I've
thought more about it, it seems a better choice than "unfencedevices". It
defines explicitly what should be done, rather than depending on the implicit
effects of matching names between fencedevice/unfencedevice.

<clusternode name="foo" nodeid="3">
<fence>
<method="1">
<device name="san" node="foo"/>
</method>
</fence>

<unfence>
<device name="san" node="foo"/>
</unfence>
</clusternode>

<fencedevices>
<fencedevice name="san" agent="fence_scsi"/>
</fencedevices>

and

<clusternode name="bar" nodeid="4">
<fence>
<method="1">
<device name="switch1" port="4"/>
<device name="switch2" port="6"/>
</method>
<method="2">
<device name="apc" port="4"/>
</method>
</fence>

<unfence>
<device name="switch1" port="4"/>
<device name="switch1" port="6"/>
</unfence>
</clusternode>

<fencedevices>
<fencedevice name="switch1" agent="fence_brocade" ipaddr="1.1.1.1"/>
<fencedevice name="switch2" agent="fence_brocade" ipaddr="2.2.2.2"/>
<fencedevice name="apc" agent="fence_apc" ipaddr="3.3.3.3"/>
</fencedevices>

The key thing I've realized since the previous attempt in 2004, is that we
need to explicitly configure what unfencing should happen, rather than just
trying to apply the normal fencing config in reverse.

Dave
 
Old 02-23-2009, 05:31 PM
"Fabio M. Di Nitto"
 
Default unfencing

On Mon, 2009-02-23 at 12:15 -0600, David Teigland wrote:
> On Mon, Feb 23, 2009 at 07:27:20AM +0100, Fabio M. Di Nitto wrote:
> > > libfence:fence_node_undo(node_name) logic:
> > > for each device_name under given node_name,
> > > if an unfencedevice exists with name=device_name, then
> > > run the unfencedevice agent with first arg of "undo"
> > > and other args the normal combination of node and device args
> > > (any agent used with unfencing must recognize/support "undo")
> >
> > All our agents already support on/off enable/disable operations. It's
> > probably best to align them to have the same config options rather than
> > adding a new one across the board.
>
> Yes, I have those options in mind, and would prefer to use them as well.
> We'll have to wait and see during the implementation phase; for the time being
> they complicate things, so I'm using "undo" to avoid those details.
>

I know Marek is about to start a "matrix" to map fence agents features
and options. It might be a good thing to talk to him soon'ish. We were
discussing it only a few hours ago.

> The meanings of those fencing structures have never changed since being
> introduced many years ago, and both of those fundamentally change it. It
> would be very unfortunate to redefine them.

I agree. it's a good point.

>
> A good alternative to <unfencedevices> would be an <unfence> section within
> the node setions (it would not require a method level).... Now that I've
> thought more about it, it seems a better choice than "unfencedevices". It
> defines explicitly what should be done, rather than depending on the implicit
> effects of matching names between fencedevice/unfencedevice.

Agreed.

>
> <clusternode name="foo" nodeid="3">
> <fence>
> <method="1">
> <device name="san" node="foo"/>
> </method>
> </fence>
>
> <unfence>
> <device name="san" node="foo"/>
> </unfence>
> </clusternode>
>
> <fencedevices>
> <fencedevice name="san" agent="fence_scsi"/>
> </fencedevices>
>
> and
>
> <clusternode name="bar" nodeid="4">
> <fence>
> <method="1">
> <device name="switch1" port="4"/>
> <device name="switch2" port="6"/>
> </method>
> <method="2">
> <device name="apc" port="4"/>
> </method>
> </fence>
>
> <unfence>
> <device name="switch1" port="4"/>
> <device name="switch1" port="6"/>
> </unfence>
> </clusternode>
>
> <fencedevices>
> <fencedevice name="switch1" agent="fence_brocade" ipaddr="1.1.1.1"/>
> <fencedevice name="switch2" agent="fence_brocade" ipaddr="2.2.2.2"/>
> <fencedevice name="apc" agent="fence_apc" ipaddr="3.3.3.3"/>
> </fencedevices>
>
> The key thing I've realized since the previous attempt in 2004, is that we
> need to explicitly configure what unfencing should happen, rather than just
> trying to apply the normal fencing config in reverse.

I think I was trying to apply this same logic and stalled at some point
in the apc+brocade example. With more than one fence agent the amount of
combinations to achieve fencing and then safely unfence node simply
grows exponentially..

Given this last example, a reasonable unfence operation would be to try
to poweron via apc too.

There is no guarantee that it was only method="1" fencing the node and
the node could be powered off.

if we succeed in enabling the switch port, we still don't guarantee that
the node will come back because of lack of power..

How do we protect a node that failed to be fenced, from being unfenced?

Example 2:
both method="1" and method="2" fail to fence node X.
At this point any unfence operation is extremely dangerous.

Fabio
 
Old 02-23-2009, 05:40 PM
David Teigland
 
Default unfencing

On Mon, Feb 23, 2009 at 07:31:29PM +0100, Fabio M. Di Nitto wrote:
> Given this last example, a reasonable unfence operation would be to try
> to poweron via apc too.
>
> There is no guarantee that it was only method="1" fencing the node and
> the node could be powered off.
>
> if we succeed in enabling the switch port, we still don't guarantee that
> the node will come back because of lack of power..
>
> How do we protect a node that failed to be fenced, from being unfenced?
>
> Example 2:
> both method="1" and method="2" fail to fence node X.
> At this point any unfence operation is extremely dangerous.

A node unfences *itself* when it boots up. As such, power-unfencing doesn't
make sense; unfencing is only meant to reverse storage fencing.

Dave
 
Old 02-23-2009, 05:52 PM
"Fabio M. Di Nitto"
 
Default unfencing

On Mon, 2009-02-23 at 12:40 -0600, David Teigland wrote:
> On Mon, Feb 23, 2009 at 07:31:29PM +0100, Fabio M. Di Nitto wrote:
> > Given this last example, a reasonable unfence operation would be to try
> > to poweron via apc too.
> >
> > There is no guarantee that it was only method="1" fencing the node and
> > the node could be powered off.
> >
> > if we succeed in enabling the switch port, we still don't guarantee that
> > the node will come back because of lack of power..
> >
> > How do we protect a node that failed to be fenced, from being unfenced?
> >
> > Example 2:
> > both method="1" and method="2" fail to fence node X.
> > At this point any unfence operation is extremely dangerous.
>
> A node unfences *itself* when it boots up. As such, power-unfencing doesn't
> make sense; unfencing is only meant to reverse storage fencing.

What can stop a user to run fence_node -U from another node to do remote
(un)fencing?

How do we address the problem of nodes booting from that same shared
storage?

Fabio
 
Old 02-23-2009, 06:09 PM
David Teigland
 
Default unfencing

On Mon, Feb 23, 2009 at 07:52:55PM +0100, Fabio M. Di Nitto wrote:
> > A node unfences *itself* when it boots up. As such, power-unfencing doesn't
> > make sense; unfencing is only meant to reverse storage fencing.
>
> What can stop a user to run fence_node -U from another node to do remote
> (un)fencing?

It would work. Users can do anything they like, that's beside the point.

The point is to make storage fencing more practical by automating storage
unfencing. Otherwise, users have to invent ad hoc methods of doing it
themselves, often manually. And, we end up solving the problem in painful,
one-off cases like scsi_reserve/fence_scsi, which cry out for a better
approach.

> How do we address the problem of nodes booting from that same shared
> storage?

Use power fencing (that's not the problem I'm trying to solve.)

Dave
 
Old 02-23-2009, 06:22 PM
"Ryan O'Hara"
 
Default unfencing

On Mon, Feb 23, 2009 at 01:09:58PM -0600, David Teigland wrote:
> On Mon, Feb 23, 2009 at 07:52:55PM +0100, Fabio M. Di Nitto wrote:
> > > A node unfences *itself* when it boots up. As such, power-unfencing doesn't
> > > make sense; unfencing is only meant to reverse storage fencing.
> >
> > What can stop a user to run fence_node -U from another node to do remote
> > (un)fencing?
>
> It would work. Users can do anything they like, that's beside the point.
>
> The point is to make storage fencing more practical by automating storage
> unfencing. Otherwise, users have to invent ad hoc methods of doing it
> themselves, often manually. And, we end up solving the problem in painful,
> one-off cases like scsi_reserve/fence_scsi, which cry out for a better
> approach.

I was going to ask about this. From the sounds of it, we could
elimiate the need for scsi_reserve completly. At least I can't think
of any reason that it could not me eliminated at this point.
 
Old 02-23-2009, 06:27 PM
David Teigland
 
Default unfencing

On Mon, Feb 23, 2009 at 01:22:26PM -0600, Ryan O'Hara wrote:
> On Mon, Feb 23, 2009 at 01:09:58PM -0600, David Teigland wrote:
> > On Mon, Feb 23, 2009 at 07:52:55PM +0100, Fabio M. Di Nitto wrote:
> > > > A node unfences *itself* when it boots up. As such, power-unfencing doesn't
> > > > make sense; unfencing is only meant to reverse storage fencing.
> > >
> > > What can stop a user to run fence_node -U from another node to do remote
> > > (un)fencing?
> >
> > It would work. Users can do anything they like, that's beside the point.
> >
> > The point is to make storage fencing more practical by automating storage
> > unfencing. Otherwise, users have to invent ad hoc methods of doing it
> > themselves, often manually. And, we end up solving the problem in painful,
> > one-off cases like scsi_reserve/fence_scsi, which cry out for a better
> > approach.
>
> I was going to ask about this. From the sounds of it, we could
> elimiate the need for scsi_reserve completly. At least I can't think
> of any reason that it could not me eliminated at this point.

Yep, that's the idea.
 
Old 02-23-2009, 06:36 PM
"Ryan O'Hara"
 
Default unfencing

On Fri, Feb 20, 2009 at 03:44:32PM -0600, David Teigland wrote:

> [Note: we've talked about fence_scsi getting a device list from
> /etc/cluster/fence_scsi.conf instead of from clvm. It would require
> more user configuration, but would create fewer problems and should
> be more robust.]

Agreed. The "discovery" concept in scsi_reserve/fence_scsi limits its
use to clvm volumes. How to configure devices for use with scsi
reservations is a problem that will need to be addressed.

> fence_node_undo("bar") would:
> - fork fence_brocade
> - pass arg string: undo port="4" agent="fence_brocade" ipaddr="1.1.1.1"
> - fork fence_brocade
> - pass arg string: undo port="6" agent="fence_brocade" ipaddr="2.2.2.2"
> - ignore device "apc" because it's not found under <unfencedevices>

What happens if unfencing fails? Is it safe to say that a node that
fails to unfence itself will be prohibited from joining the fence
domain? This is important for fence_scsi, since unfencing is
equivalient to re-registering with the scsi devices. Failure to
unfence (ie. register) precludes that node from being able to fence
other nodes. I'm not sure if other fencing methods have this type of
requirement.
 

Thread Tools




All times are GMT. The time now is 11:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org