FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 05-08-2010, 02:23 AM
Ralph Angenendt
 
Default Not firewall, but what?

Am 06.05.10 20:43, schrieb Paul Heinlein:
> A while back, I remember there was a problem with TCP window scaling
> that would impact only some clients in a way that you describe:
>
> http://lwn.net/Articles/92727/

Thank you, I was searching for that a few weeks ago, but didn't find it.
Bookmarked.

Ralph

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 06:01 AM
Gordon Messmer
 
Default Not firewall, but what?

On 05/07/2010 07:26 AM, Jussi Hirvi wrote:
>
> [root@farm1 log]# ip route show
> 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67
> 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
> 169.254.0.0/16 dev eth1 scope link
> default via 62.220.237.126 dev eth1

Yeah, so you have two interfaces on different IP networks. When someone
connects to 62.236.221.67, the reply packets will still head out through
62.220.237.126 on eth1. That router probably filters the reply packets
since they're from a non-local IP network.

I'm not sure if there's a simpler way to do this: When I have
multi-homed servers I usually just use Shorewall to create two routing
tables: one with a default route through each outbound router. Packets
are marked based on their source address and routed based on those marks.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 06:11 AM
John R Pierce
 
Default Not firewall, but what?

Gordon Messmer wrote:

On 05/07/2010 07:26 AM, Jussi Hirvi wrote:


[root@farm1 log]# ip route show
62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67
62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
169.254.0.0/16 dev eth1 scope link
default via 62.220.237.126 dev eth1



Yeah, so you have two interfaces on different IP networks. When someone
connects to 62.236.221.67, the reply packets will still head out through
62.220.237.126 on eth1. That router probably filters the reply packets
since they're from a non-local IP network.

I'm not sure if there's a simpler way to do this: When I have
multi-homed servers I usually just use Shorewall to create two routing
tables: one with a default route through each outbound router. Packets
are marked based on their source address and routed based on those marks.






that can be done with `ip route` and `ip rule` and a few iptables
commands, you dont need shorewall to do it for you.* you got it exactly
right, you mark sockets that are 'to' the 2nd interface to use the
alternate routing table, whihc specifies the alternate default gateway.



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 07:38 AM
Jussi Hirvi
 
Default Not firewall, but what?

On 8.5.2010 4.31, Kahlil Hodgson wrote:
> Hmmm have you got more than one bridge on your network? If so you need
> to make sure you have STP turned ON on all your bridges.
> If you have any services that require network at start up (nfs), you'll
> need set you network start up delay to more than 10 seconds
> as well, so STP has some time to settle.
>
> I encountered similar problems when I plugged a _second_ virtualisation
> host into my network.

Turning on stp sounds promising (I have to confess that I never heard
about stp before). Stp is indeed off for both bridges:

[root@farm1 scripts]# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes
xenbr0 8000.feffffffffff no vif5.0
vif4.0
peth0
vif0.0
xenbr1 8000.feffffffffff no vif5.1
vif3.0
vif2.0
peth1
vif0.1


How can I turn stp on? In my /etc/xen/scripts/xen-network-common.sh
there is a section:

# Don't create the bridge if it already exists.
if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then
brctl addbr ${bridge}
brctl stp ${bridge} off
brctl setfd ${bridge} 0
sysctl -w "net.bridge.bridge-nf-call-arptables=0"
sysctl -w "net.bridge.bridge-nf-call-ip6tables=0"
sysctl -w "net.bridge.bridge-nf-call-iptables=0"
ip link set ${bridge} arp off
ip link set ${bridge} multicast off
fi

Is if safe to turn stp "on" there (instead of "off"? (Requires xend
restart at least, I suppose.) Or is there a better way to turn stp on
permanently?

The box has 2 physical if cards, and both of them are used for bridges
(xenbr0 and xenbr1).


- Jussi

--
Jussi Hirvi * Green Spot
Topeliuksenkatu 15 C * 00250 Helsinki * Finland
Tel. +358 9 493 981 * Mobile +358 40 771 2098 (only sms)
jussi.hirvi@greenspot.fi * http://www.greenspot.fi
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 08:56 AM
Kahlil Hodgson
 
Default Not firewall, but what?

On 05/08/2010 05:38 PM, Jussi Hirvi wrote:
> How can I turn stp on? In my /etc/xen/scripts/xen-network-common.sh
> there is a section:
>
> # Don't create the bridge if it already exists.
> if [ ! -e "/sys/class/net/${bridge}/bridge" ]; then
> brctl addbr ${bridge}
> brctl stp ${bridge} off
> brctl setfd ${bridge} 0
> sysctl -w "net.bridge.bridge-nf-call-arptables=0"
> sysctl -w "net.bridge.bridge-nf-call-ip6tables=0"
> sysctl -w "net.bridge.bridge-nf-call-iptables=0"
> ip link set ${bridge} arp off
> ip link set ${bridge} multicast off
> fi
>
> Is if safe to turn stp "on" there (instead of "off"? (Requires xend
> restart at least, I suppose.) Or is there a better way to turn stp on
> permanently?

STP is safe to turn on, but there is a small start up and tiny
performance hit - that's why its off by default. All the bridges on
your network have to establish relationships with each other, which can
take 10-15 seconds depending on you network. Also, its not just the
bridges on that box that you have to worry about: any other bridges on
other boxes that are on the same network also need STP turned on. Your
old Fedora box may be a potential culprit.

I've never used Xen, so I can't give any firm advice.
That looks like the place where the bridge is created, so at a guess,
that's where you want to turn it on. Not to sure about turning ARP or
MULTICAST off though -- that might interfere with STP.

> The box has 2 physical if cards, and both of them are used for bridges
> (xenbr0 and xenbr1).

Yeah. Thinking you definitely need STP. You can turn it on temporarily
with

brctl stp xenbr0 on
brctl stp xenbr1 on

wait a few seconds and run

brctrl showstp xenbr0

to see what's going on, and also see if it fixes your problem.

Hope this helps

Kal
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 12:00 PM
Jussi Hirvi
 
Default Not firewall, but what?

On 8.5.2010 11.56, Kahlil Hodgson wrote:
>> Is if safe to turn stp "on" there (instead of "off"? (Requires xend
>> restart at least, I suppose.) Or is there a better way to turn stp on
>> permanently?
>
> STP is safe to turn on, but there is a small start up and tiny
> performance hit - that's why its off by default. All the bridges on
> your network have to establish relationships with each other, which can
> take 10-15 seconds depending on you network. Also, its not just the
> bridges on that box that you have to worry about: any other bridges on
> other boxes that are on the same network also need STP turned on. Your
> old Fedora box may be a potential culprit.
>
> I've never used Xen, so I can't give any firm advice.
> That looks like the place where the bridge is created, so at a guess,
> that's where you want to turn it on. Not to sure about turning ARP or
> MULTICAST off though -- that might interfere with STP.
>
>> The box has 2 physical if cards, and both of them are used for bridges
>> (xenbr0 and xenbr1).
>
> Yeah. Thinking you definitely need STP. You can turn it on temporarily
> with
>
> brctl stp xenbr0 on
> brctl stp xenbr1 on
>
> wait a few seconds and run
>
> brctrl showstp xenbr0
>
> to see what's going on, and also see if it fixes your problem.
>
> Hope this helps
>
> Kal

Thanks, it does (though the problem still persists).

I turned stp on (for both bridges). I found another virbridge on another
machine which has 2 if-cards: "virbr0", created by CentOS 5 by default I
guess, for dhcp network, which I never even thought of. I brought this
bridge down with icfonfig - btw, how can I disable it so that it stays
off through reboots?

So far the problem persists - I guess that I will have to start
modifying routing tables.

I guess it's natural that this kind of problem is weird. :-)

For example, it is kind of natural that I can access these problematic
62.236.221.xx addresses (on the xen box) from other boxes in the same
62.236.221.xx network segment.

But I can *also* access those ip addresses from the network
62.220.237.xx. Why? No idea. (the other if-card on the xen box is
configured to this network segment, but I don't see why this would
explain this.)

Also seen from my home computer at 84.20.154.60 everything seems normal
- no idea why!

These (62.236.221.xx, 62.220.237.xx, 84.20.154.58/xx) are the only known
clients from which the problematic addresses (62.236.221.67,
62.236.221.71) on the xen box are visible. :-/

- Jussi

--
Jussi Hirvi * Green Spot
Topeliuksenkatu 15 C * 00250 Helsinki * Finland
Tel. +358 9 493 981 * Mobile +358 40 771 2098 (only sms)
jussi.hirvi@greenspot.fi * http://www.greenspot.fi
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 01:28 PM
JohnS
 
Default Not firewall, but what?

On Sat, 2010-05-08 at 15:00 +0300, Jussi Hirvi wrote:

>
> But I can *also* access those ip addresses from the network
> 62.220.237.xx. Why? No idea. (the other if-card on the xen box is
> configured to this network segment, but I don't see why this would
> explain this.)
>
> Also seen from my home computer at 84.20.154.60 everything seems normal
> - no idea why!
>
> These (62.236.221.xx, 62.220.237.xx, 84.20.154.58/xx) are the only known
> clients from which the problematic addresses (62.236.221.67,
> 62.236.221.71) on the xen box are visible. :-/
---

You only see them from your home pc because your on the same address
block/dom/ip carrier! Look at your routing and dns. Sporadic dns
issues and routing? BTW some addys get blocked from certain countries
also. If I were you I would start from scratch and go step by step and
set it up.

John

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 06:28 PM
Jussi Hirvi
 
Default Not firewall, but what?

On 8.5.2010 16.28, JohnS wrote:
> You only see them from your home pc because your on the same address
> block/dom/ip carrier! Look at your routing and dns. Sporadic dns
> issues and routing? BTW some addys get blocked from certain countries
> also. If I were you I would start from scratch and go step by step and
> set it up.

Just for the record: dns has been ruled out as a cause of these errors,
since the error persists even when connecting by ip address.

The subject of the thread still seems appropriate: it *seems* just like
firewall issue (my home, and the local network segments get access,
everybody else not), but it isn't (iptables stop does not help).
Instead, the cause of the malady is most probably some routing tangle in
the xen box itself.

- Jussi

--
Jussi Hirvi * Green Spot
Topeliuksenkatu 15 C * 00250 Helsinki * Finland
Tel. +358 9 493 981 * Mobile +358 40 771 2098 (only sms)
jussi.hirvi@greenspot.fi * http://www.greenspot.fi
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-08-2010, 07:16 PM
JohnS
 
Default Not firewall, but what?

On Sat, 2010-05-08 at 21:28 +0300, Jussi Hirvi wrote:
> On 8.5.2010 16.28, JohnS wrote:
> > You only see them from your home pc because your on the same address
> > block/dom/ip carrier! Look at your routing and dns. Sporadic dns
> > issues and routing? BTW some addys get blocked from certain countries
> > also. If I were you I would start from scratch and go step by step and
> > set it up.
>
> Just for the record: dns has been ruled out as a cause of these errors,
> since the error persists even when connecting by ip address.
>
> The subject of the thread still seems appropriate: it *seems* just like
> firewall issue (my home, and the local network segments get access,
> everybody else not), but it isn't (iptables stop does not help).
> Instead, the cause of the malady is most probably some routing tangle in
> the xen box itself.
---
Since you really only have access on the address blocks allocated to
(your name here) then maybe talk to the Up Stream provider from yours
since you have access with in the *greenspot domain allocation. For the
record I cant ping any one of the two IPs from my home or any where
else.

Do you have inbound access to other servers on the same block and
subnet? Do you have the right sub net mask for the ip block your using?
62.236.221.64 - 62.236.221.79

Out of that block I can only access these if that helps you any. This
should give you a big clue to what is going on.

62.236.221.70:
https://webmail.greenspot.fi/cgi-bin/openwebmail/openwebmail.pl

http://62.236.221.73/

62.236.221.76
http://www.danskecapital.fi/


John

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2010, 11:03 AM
Kahlil Hodgson
 
Default Not firewall, but what?

On 05/08/2010 11:28 PM, JohnS wrote:
> If I were you I would start from scratch and go step by step and set
> it up.
>
> John

I'm in agreement with John here. Your set up looks complex and may be
starting from scratch is the way to go. Looking back though the thread,
your set up might also be unnecessarily complex. I'm not entirely sure
why you have two xen bridges and two vlans on the xen host, but that may
just be the way the xen setup scripts work, and, well, I've never used
xen. (I use libvirt with KVM, plus virtio to give me decent performance,
and I hardwire my bridges in my network-scripts so I can ensure they are
up and running before anything tries to use them).

I think the problem is either ARP or routing (or both) or an overly
complex setup (or all three) ;-) Lets start with ARP.

ARP is pretty reliable and usually 'just works' on most networks -- so
much so that it rarely causes any problems and is easy to forget about.
The one situation that is guaranteed to cause a problem is having two
host with the same IP address on the same network. (I had a situation
once where a technician reset a problematic UPS to its defaults --
giving it the same IP address as one of our core routers and causing us
no end of problems till we figured it out). This situation is tricky to
troubleshoot because some parts of the network may be quite stable
depending on the layout of bridges. (I had another situation once where
a wireless router was reset to defaults and took on the IP address of
the main gateway -- one bunch of Linux machines behind one switch was
stable whereas another bunch of Windows machines behind a different
switch was unstable with arp entries flip-flopping every 10 minutes or
so). I mention this case in detail because: 1) you're migrating from an
existing setup and there is a small chance that the old host is still on
the network and has the problematic IP address, or there is some static
APR entry lying around somewhere, and double checking may save you some
sanity; and 2) you're using virtualization and bridges which means ARP
has to flow and (in complex set ups) is why turning on STP can help.

For completeness (and the archives) detecting the 'multiple hosts with
the same IP' case is pretty easy: just install arpwatch on a host that
receives traffic from all hosts on the network (router, dns server, dhcp
server). You get a nice little email every time it thinks an IP address
is flip-flopping. You also get a nice little email every time someone
plugs a new device into your network (say, the cleaner's laptop at 2am).
If you don't have arpwatch, then you could try:

while true; do arp -n >> arp.log; sleep 5; done

for 10-15 minutes (longer on a quiet network) and then do

sort -u arp.log

If the same IP address pops up twice, you have a problem. Tracking down
the culprit is a little harder, but you can use the fact that the
leading portion of the HW address is usually the same for the same NIC
manufacturer. If the offending HW address is similar to those used by,
say, known SUN machines, then that rules out any, say, Dell desktops.
The arp.log file you just created can be handy for making these
comparisons.

The above problem is rare. I've only experienced on 'real' hardware 3
times in the passed 10 years, but I've seen it a lot more since I
started playing with virtualization. A more common issue is ARP
requests not flowing around your network properly. Bridges allow you to
join networks together by passing along ARP queries. If you have
multiple bridges, you can get loops, and this can prevent the ARP query
reaching the right bridge, so you need STP to ensure bridges cooperate
appropriately. (You can get by without STP, but only if you are 100%
sure there are no loops). Firewalls can stop ARP flowing as well. On
Linux this is arptables and ebtables. I note that your xen startup
scripts turn this off, so you should be good, provided turning them off
doesn't block ARP forwarding. Not sure about that. (I notice they are on
on my systems, but I like to restrict traffic on the bridge, with the
iptables/arptable/ebtables configuration managed by shorewall).

I troubleshoot the above ARP issue the same way as for routing and
firewalls, so lets move onto that. For newbies and the archives, I'll
detail my approach.

Firstly, you want to map out whether or not traffic can pass between
various IP addresses. These flows may or may not be bi-directional (A
to B, is not necessarily the same as B to A) depending on stateful
firewalls and convoluted routing. If there are more than 2 addresses
involved, things can get complicated and its easy miss important clues,
so its good to be organised. I usually grab a scrap piece of paper and
draw up a NxN matrix of the problematic IP addresses so I can note which
flows pass and which flows fail. I also add a few good IP addresses
(your desktop, routers, dhcp server, dns server, xen host, other guests,
home desktop, are good candidates) so I can verify my methodology and
potentially spot related issues -- these may provide clues as to the
real problem. This may seem excessive, but at least you know you are
getting a complete picture.

I test flows with a combination of ping (or hping) and tcpdump. I
normally layout 4 terminals in a 2x2 grid like so:

host A: tcpdump host A: ping host B

host B: tcpdump host B: ping host A

For testing routes and ARP my tcpdump invocation is something like

tcpdump -i ethX icmp or arp

where ethX is the interface with the IP address I'm testing. For testing
firewall rules I use something like

tcpdump -i ethX port YYYY

and invoke hping like so

hping -S -p YYYY aaa.bbb.ccc.ddd

where YYYY is the port you are testing and aaa.bbb.ccc.ddd the IP
address. There are lots of other tcpdump options that you can use to
refine what you see and cut out any irrelevant traffic (like your ssh
session). Note that I only run one ping test at a time so I can watch
packets flowing from both sides. The extra terminal is just to make it
a little easier to be systematic in tracing reverse flows.

Now if you do the above tests on a pair of IP addresses where routing
and ARP is good, you should see nice little ICMP request/reply pairs and
nice little ARP who-has/reply pairs. If APR isn't working, you'll be
getting a strings of ARP who-has packets with no replies. If routing is
broken, you get strings of ICMP requests without replies (yeah
obviously) and by comparing the outputs in the three terminals, you may
be able to figure out where its broken. If you fill out your flow
matrix, you'll have a pretty clear picture of the situation and may have
some more clues as to were the problem lies. You'll also have a nice
framework for testing potential fixes.

Jussi, if this is all old news to you, please forgive my rambling
diatribe. From bitter experience, I just like to make sure everyone's on
the same page with this stuff:-)

Now lets look at some details.

> Physical if cards:
> Intel Corporation 82567LM-3 Gigabit Network Connection
> (eth0; on motherboard)

Recently I've had some onboard NICs slowly die over a period of about 2
weeks. When it happens you can get some fairly inexplicable errors.
Probably not your problem, but thought I might point out the possibility.

ip route output on the xen host:

> 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67
> 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104
> default via 62.220.237.126 dev eth1

Your 'good' interface eth1 is also the default route. I gather
62.220.237.126 is your border router. Where does your 'bad' interface
eth0 go then? The most common problem with routing on multi-homed
systems is having packets leaving one interface and responses coming
back on another.

Really need to see what 'ip route' says on each of you guests.

Hmmm... ifconfig on you xen host:

> vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
> ....
> vif5.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
> ...
> xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
> UP BROADCAST RUNNING NOARP MTU:1500 Metric:1

Normally an interface with a 'dot' suffix is on a specific VLAN. So
you've got two VLANs running here? This complicated things. Also your
bridge has NOARP set which does not make much sense to me.

brctl show
> bridge name bridge id STP enabled interfaces
> virbr0 8000.000000000000 yes
> xenbr0 8000.feffffffffff no vif5.0
> vif4.0
> peth0
> vif0.0
> xenbr1 8000.feffffffffff no vif5.1
> vif3.0
> vif2.0
> peth1
> vif0.1

Okay, that makes my head hurt. Why two VLANs? What's you mapping
between virtual interfaces and guests? And which guest is the bad one?
Hmmm ... I could interpret this as meaning vif5.0 and vif5.1 are two
interfaces on guest5 which you want to directly tie to to different
physical cards. Perhaps some source based routing on your xen host
would serve you better. This may be part of the overly complex setup
which is causing problems.

Are these guests all just web servers?

Sorry if that is a too long ramble. I'm a bit confused by your set up,
and as John suggested, you may need to rebuild things from scratch to
sort things out.

Hope this helps!

Kal

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 09:20 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org