FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 02-09-2012, 10:54 PM
Bob Hoffman
 
Default oops, or how to bring a datacenter router down with one setting

so I gave up on bonding.
I found about 300 posts showing eth0 and eth1 both pointing to br0 (bridge)
as interfaces.
I followed them correctly, or so I thought.
I pointed both ethx to the bridge, restarted network and bam...!!!

entire ip block went out.

when I called datacenter they told me the router was under attack and I
was like 'uh oh' and told them to just shut off my computer I would be
there to fix it. They did not believe me.
An hour later I was there and deleted the eth1 point to the br0 and all
was fine.
Meanwhile they were all around the router trying to stop the attack.
(it was just the router for me and others in that room....oops)

I wonder if they will boot me from the center now?
How is it possible that it did that so quickly?
Such an easy way to bring down routers, wow, a hacker could have a field
day.

Apparently there is more to making to eth ports go to the same bridge
than a simple point.
I have since tried bridge_ports command as listed online, however that
must be deprecated.
I think I am just gonna stay with multiple bridges with one eth on each
for a while until
I can test this stuff in a safe environ.

I never had a chance to recover, the second the network came up I lost
all contact with my ip block.
The ratelimit number got this high by the time I got there.



Feb 9 04:22:41 main kernel: __ratelimit: 100807 callbacks suppressed
Feb 9 04:22:41 main kernel: eth1: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth1: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth1: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth1: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
Feb 9 04:22:41 main kernel: eth0: received packet with own address as
source address
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 02-10-2012, 09:18 AM
 
Default oops, or how to bring a datacenter router down with one setting

In article <4F345CD3.4060604@bobhoffman.com>,
Bob Hoffman <bob@bobhoffman.com> wrote:
> so I gave up on bonding.
> I found about 300 posts showing eth0 and eth1 both pointing to br0 (bridge)
> as interfaces.
> I followed them correctly, or so I thought.
> I pointed both ethx to the bridge, restarted network and bam...!!!
>
> entire ip block went out.
>
> [...]
>
> Feb 9 04:22:41 main kernel: __ratelimit: 100807 callbacks suppressed
> Feb 9 04:22:41 main kernel: eth1: received packet with own address as
> source address

I think to do this you also need to be connected to a managed switch
which supports interface bonding. You would have to tell it that the two
switch ports are bonded to the same machine. That should prevent it from
forwarding packets received on one of the ports out via the other port.

The key phrase to look for appears to be "IEEE 802.3ad Dynamic Link
Aggregation".

Cheers
Tony
--
Tony Mountifield
Work: tony@softins.co.uk - http://www.softins.co.uk
Play: tony@mountifield.org - http://tony.mountifield.org
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 02-10-2012, 01:02 PM
Janez Kosmrlj
 
Default oops, or how to bring a datacenter router down with one setting

i have several centos 5.x servers with bonding enabled. And none of them
have any problems.

I used this tutorial:
http://www.howtoforge.com/network_card_bonding_centos

I use mode=6.

On Fri, Feb 10, 2012 at 2:54 PM, Bob Hoffman <bob@bobhoffman.com> wrote:

>
> ---------------------------------------------------------
> Dennis Jacobfeuerborn wrote
> /Fri Feb 10 06:47:22 EST 2012/
>
> On 02/10/2012 12:54 AM, Bob Hoffman wrote:
> >/ so I gave up on bonding.
> />/ I found about 300 posts showing eth0 and eth1 both pointing to br0
> (bridge)
> />/ as interfaces.
> />/ I followed them correctly, or so I thought.
> />/ I pointed both ethx to the bridge, restarted network and bam...!!!
> /
> Bonding and bridging are completely different things. If you want to start
> bonding then you should first start with simply bonding the two interfaces
> and only once you got that going add the bridge and then add the bond0
> device to it.
>
> Regards,
> Dennis
>
> -----------------------------------------------------------
>
> Yea, I gave up on bonding, ended up just using eth1. But every tutorial
> I found had added eth0 and eth1 as interfaces to br0, thus sharing the
> bridge so to speak.
> All the tutorials were for debian though, all the centos ones ended up
> pointing each eth to a different cridge (br0 and br1)
> So I tried it....bam, took down router in less than a second.
>
> I did not add a domain= setting in the bridge though. With network
> manager off completely I thought I would not need too.
> Looking at the resolv.conf it was overwritten anyway and since no domain
> was listed, it said
> "search belkin"
> search belkin
>
> I assume that was the datacenters router....
>
> I was not bonding at this time. I am wondering though why the network
> manager overwrites resolv.conf if NM is off, all ifcfg files say
> nm_controlled=no, and chkconfig NetworkManager off was run.
>
> It is not that way on my 5.x, but I guess things change. I wonder if
> that was messing my bond experiment up too without me knowing it.
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 02-10-2012, 06:49 PM
Les Mikesell
 
Default oops, or how to bring a datacenter router down with one setting

On Fri, Feb 10, 2012 at 9:25 AM, Bob Hoffman <bob@bobhoffman.com> wrote:

>
> Nothing at all to do with bonding. Not at all.
> eth1 to br0 , eth0 to br0....that's all.
> If that is possible, I see no reason for a bond at all.
> I just want to make sure if an NIC fails, the other one is still working
> while I am asleep and not a care in the world.
>

I suppose it is possible for a NIC to fail, but I can't recall actually
ever seeing it. I've seen lots of complicated failover schemes introduce
new problems and their own failure modes though, including a bad cable that
kept flipping the primary/backup links at approximately the same rate that
spanning-tree would let them switch.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 02-10-2012, 07:01 PM
Lamar Owen
 
Default oops, or how to bring a datacenter router down with one setting

On Feb 9, 2012, at 6:54 PM, Bob Hoffman wrote:

entire ip block went out.

when I called datacenter they told me the router was under attack
and I

was like 'uh oh' and told them to just shut off my computer I would be
there to fix it. They did not believe me.
An hour later I was there and deleted the eth1 point to the br0 and
all

was fine.
Meanwhile they were all around the router trying to stop the attack.
(it was just the router for me and others in that room....oops)

I wonder if they will boot me from the center now?
How is it possible that it did that so quickly?
Such an easy way to bring down routers, wow, a hacker could have a
field

day.


If you weren't running a spanning-tree on your Linux bridge, and their
switch ports aren't sending you BPDU's for STP, then you found out
what happens when you activate a bridging (from the point of view of
the switch, not the Linux bridging) loop. Been there, done that.
Most monitoring tools are written to track layer-3 happenings, and
this is happening at layer 2. And it will take down that whole layer
2 broadcast domain, that's for sure.


And since many, if not most, tools are working at layer 3 and dealing
with IP flows and not actual ethernet traffic, none of the typical
layer 3 tools will give any indication why the network just bogged
down to a halt; you just about have to have a network probe (like
wireshark) on a SPAN port to catch it, unless you know some of the
telltale signs. On a gigabit switch a fully saturating bridge loop
can form in less than a second, and bring things close to a halt.


Most datacenter switches have configurable parameters to guard against
loops (Cisco even has a feature called, appropriately enough,
loopguard, but this may or may not fix this case).


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 02-10-2012, 08:58 PM
Les Mikesell
 
Default oops, or how to bring a datacenter router down with one setting

On Fri, Feb 10, 2012 at 3:40 PM, <m.roth@5-cent.us> wrote:

> Devin Reade wrote:
> <snip>
> > I do have clusters where bonding is in use but those have helped not so
> > much in avoiding NIC failures as they do in allowing the machines
> > to continue operating as the network team brings down part of the
> > redundant switch network for maintenance (or to replace a failed switch,
> > or when some fool decides that they can unplug a network cable
> > briefly so that they can move other cables around).
> >
> Now wait a minute - I would dearly love to disconnect some cables we have
> in a shared rack downstairs in the datacenter - it's a rats' nest, and
> more than half ain't ours, and every single time I have to do something in
> the back, I'm deathly afraid I'm going to pull out somebody's power,
> or....
>

Do you really want to double the size of the mess to make it a little safer
to move one thing? Redundant power connections normally do work with only
a little attention to grounding and that the connections really do go to
separate circuits/UPSs. But with NICs, you have to be very careful that
the switch ports are configured to match so you are even more likely to
break things by moving them around. It's not impossible, but rarely
worthwhile if you don't need the combined bandwidth. But the real lesson
here is to not do something for the first time in a place where mistakes
will cause big trouble.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 10:39 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org