This is on Centos 6 and not something I think is wrong with Centos 6
but I am looking to see if anybody else has experienced this and
if there is solution. So thanks up front for indulging me.
Because Linux makes routing decisions before SNAT it is causing
problems when trying to use FTP with two upstream providers in
a load balanced setup.
Other than ftp, things seem to work OK. Below is my setup and tcpdump
output that shows ftp packets trying to go out the wrong interface.
ip ru sh
0: from all lookup local
200: from y.y.y.174 lookup t1
201: from x.x.x.217 lookup t2
32766: from all lookup main
32767: from all lookup default
ip r s
y.y.y.129 dev eth1 scope link
172.16.0.0/29 dev gre1 proto kernel scope link src 172.16.0.1
y.y.y.128/25 dev eth1 proto kernel scope link src y.y.y.174
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.90
192.168.198.0/24 dev eth0 proto kernel scope link src 192.168.198.92
x.x.x.0/24 dev eth2 proto kernel scope link src x.x.x.217
10.0.128.0/17 dev eth0 proto kernel scope link src 10.0.129.88
default
nexthop via y.y.y.129 dev eth1 weight 1
nexthop via x.x.x.1 dev eth2 weight 1
ip r s tab t1
default via y.y.y.129 dev eth1 src y.y.y.174
ip r s tab t2
default via x.x.x.1 dev eth2 src x.x.x.217
Chain PREROUTING (policy ACCEPT 1050K packets, 128M bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 423K packets, 35M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * eth1 10.0.1.0/24 10.0.0.0/8
0 0 ACCEPT all -- * eth1 10.0.1.0/24 172.16.0.0/12
0 0 ACCEPT all -- * eth1 10.0.1.0/24 192.168.0.0/16
58 3480 SNAT all -- * eth1 10.0.1.0/24 0.0.0.0/0
to:y.y.y.174
4 240 SNAT all -- * eth2 10.0.1.0/24 0.0.0.0/0
to:x.x.x.217
connection starts out eth2 - but then subsequent packets that should be
routed out eth2 are being sent out eth1 see below.
eth2 x.x.x.217
tcpdump -nli eth2 host 131.247.254.5
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 02:16 AM
Gordon Messmer
Routing issue
On 09/26/2012 09:15 AM, Steve Clark wrote:
> Is there a way to make this work correctly?
Shorewall will generate a proper configuration if you specify the
"track" option in the "providers" file. It might be a good idea to use
that to generate your configs rather than building them by hand.
I believe that you need to mark your connections and use the marks to
select the routing table, in addition to using the "from" rules that you
posted. Otherwise, nothing binds the connection to a fixed
route/interface in a load balanced configuration.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 03:57 AM
Manish Kathuria
Routing issue
On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
> On 09/26/2012 09:15 AM, Steve Clark wrote:
>> Is there a way to make this work correctly?
>
> Shorewall will generate a proper configuration if you specify the
> "track" option in the "providers" file. It might be a good idea to use
> that to generate your configs rather than building them by hand.
>
> I believe that you need to mark your connections and use the marks to
> select the routing table, in addition to using the "from" rules that you
> posted. Otherwise, nothing binds the connection to a fixed
> route/interface in a load balanced configuration.
In addition, you should ideally applying the following patches for
Static, Alternative Routes, Dead Gateway Detection & NAT and recompile
the kernel:
http://www.ssi.bg/~ja/#routes
Thanks,
--
Manish Kathuria
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 01:34 PM
Steve Clark
Routing issue
On 09/26/2012 11:57 PM, Manish Kathuria wrote:
> On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>> On 09/26/2012 09:15 AM, Steve Clark wrote:
>>> Is there a way to make this work correctly?
> In addition, you should ideally applying the following patches for
> Static, Alternative Routes, Dead Gateway Detection & NAT and recompile
> the kernel:
>
> http://www.ssi.bg/~ja/#routes
Hmmm... not being a kernel guru, correct me if I am wrong but isn't the route patch used to detect dead nexthops?
I am already doing that from userspace.
The second set looks like is calls the routing logic after the SNAT, is that correct? This could solve the problem.
Why aren't these patches in the kernel?
Thanks,
--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.clark@netwolves.com
http://www.netwolves.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 01:36 PM
Steve Clark
Routing issue
On 09/26/2012 10:16 PM, Gordon Messmer wrote:
> On 09/26/2012 09:15 AM, Steve Clark wrote:
>> Is there a way to make this work correctly?
> Shorewall will generate a proper configuration if you specify the
> "track" option in the "providers" file. It might be a good idea to use
> that to generate your configs rather than building them by hand.
>
> I believe that you need to mark your connections and use the marks to
> select the routing table, in addition to using the "from" rules that you
> posted. Otherwise, nothing binds the connection to a fixed
> route/interface in a load balanced configuration.
I was trying to figure out what criteria to use to mark the connection. FTP is such a
braindead application, using to channels and active and passive mode. What really
needs to happen is someway to tell the kernel to recheck the routing after SNAT.
--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.clark@netwolves.com
http://www.netwolves.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 03:01 PM
Manish Kathuria
Routing issue
On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark@netwolves.com> wrote:
> On 09/26/2012 11:57 PM, Manish Kathuria wrote:
>
> On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>
> On 09/26/2012 09:15 AM, Steve Clark wrote:
>
> Is there a way to make this work correctly?
>
> In addition, you should ideally applying the following patches for
> Static, Alternative Routes, Dead Gateway Detection & NAT and recompile
> the kernel:
>
> http://www.ssi.bg/~ja/#routes
>
>
> Hmmm... not being a kernel guru, correct me if I am wrong but isn't the
> route patch used to detect dead nexthops?
> I am already doing that from userspace.
>
> The second set looks like is calls the routing logic after the SNAT, is that
> correct? This could solve the problem.
> Why aren't these patches in the kernel?
>
> Thanks,
>
The routes-x.y-z.diff is a unified patch containing different parts
which include support for Dead Gateway Detection as well. However,
since that is limited to the first hop, it is preferable to have a
userspace script as you are doing. I also use a script to check the
accessibility of a remote popular site from each of the ISPs and based
upon the response the links are treated alive or dead and the default
gateway is changed. However, the routing problem as described by you
will only be solved after applying this patch (routes-x.y-z.diff).
As for marking the incoming packets to ensure that they go out from
the same interface they came from, you could do something like the
following:
Using iptables mark the incoming traffic from external interfaces
/sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
/sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
/sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1
/sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark
/sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2
/sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark
Add the following rules to your existing ones for policy routing
Thanks,
--
Manish Kathuria
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 03:24 PM
Gordon Messmer
Routing issue
On 09/27/2012 06:36 AM, Steve Clark wrote:
> I was trying to figure out what criteria to use to mark the connection.
> FTP is such a
> braindead application, using to channels and active and passive mode.
> What really
> needs to happen is someway to tell the kernel to recheck the routing
> after SNAT.
I'm mostly sure that if you mark the *connection* to the FTP server, the
related data will follow its path.
Again, multipath routing is complex, and Shorewall will do it properly.
At the very least, I recommend building a working configuration with
Shorewall and then reading the rules that it compiles to understand why
it handles routing the way that it does.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-27-2012, 03:25 PM
Steve Clark
Routing issue
On 09/27/2012 11:01 AM, Manish Kathuria wrote:
> On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark@netwolves.com> wrote:
>> On 09/26/2012 11:57 PM, Manish Kathuria wrote:
>>
>> On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>>
>> On 09/26/2012 09:15 AM, Steve Clark wrote:
> The routes-x.y-z.diff is a unified patch containing different parts
> which include support for Dead Gateway Detection as well. However,
> since that is limited to the first hop, it is preferable to have a
> userspace script as you are doing. I also use a script to check the
> accessibility of a remote popular site from each of the ISPs and based
> upon the response the links are treated alive or dead and the default
> gateway is changed. However, the routing problem as described by you
> will only be solved after applying this patch (routes-x.y-z.diff).
>
> As for marking the incoming packets to ensure that they go out from
> the same interface they came from, you could do something like the
> following:
>
> Using iptables mark the incoming traffic from external interfaces
>
> /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1
> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark
> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2
> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark
>
> Add the following rules to your existing ones for policy routing
>
> /sbin/ip rule add fwmark 1 table T1
> /sbin/ip rule add fwmark 2 table T2
Hi Manish,
Thanks for the info. The one question I have is about
/sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
I thought the OUTPUT chain was only for packets originating locally. I am only concerned
with clients behind my Linux router, do I still need this?
Again, thanks much for responding.
--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.clark@netwolves.com
http://www.netwolves.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-28-2012, 01:47 AM
Manish Kathuria
Routing issue
On Thu, Sep 27, 2012 at 8:55 PM, Steve Clark <sclark@netwolves.com> wrote:
> On 09/27/2012 11:01 AM, Manish Kathuria wrote:
>
> On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark@netwolves.com> wrote:
>
> On 09/26/2012 11:57 PM, Manish Kathuria wrote:
>
> On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>
> On 09/26/2012 09:15 AM, Steve Clark wrote:
>
> The routes-x.y-z.diff is a unified patch containing different parts
> which include support for Dead Gateway Detection as well. However,
> since that is limited to the first hop, it is preferable to have a
> userspace script as you are doing. I also use a script to check the
> accessibility of a remote popular site from each of the ISPs and based
> upon the response the links are treated alive or dead and the default
> gateway is changed. However, the routing problem as described by you
> will only be solved after applying this patch (routes-x.y-z.diff).
>
> As for marking the incoming packets to ensure that they go out from
> the same interface they came from, you could do something like the
> following:
>
> Using iptables mark the incoming traffic from external interfaces
>
> /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1
> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark
> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2
> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark
>
> Add the following rules to your existing ones for policy routing
>
> /sbin/ip rule add fwmark 1 table T1
> /sbin/ip rule add fwmark 2 table T2
>
> Hi Manish,
>
> Thanks for the info. The one question I have is about
> /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
>
> I thought the OUTPUT chain was only for packets originating locally. I am
> only concerned
> with clients behind my Linux router, do I still need this?
Yes you are right but in case if you have any services running on the
linux router itself (for example sshd) and accessible from the
internet, it would help.
>
> Again, thanks much for responding.
>
> --
> Stephen Clark
> NetWolves
> Director of Technology
> Phone: 813-579-3200
> Fax: 813-882-0209
> Email: steve.clark@netwolves.com
> http://www.netwolves.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
09-28-2012, 10:52 AM
Steve Clark
Routing issue
On 09/27/2012 09:47 PM, Manish Kathuria wrote:
> On Thu, Sep 27, 2012 at 8:55 PM, Steve Clark <sclark@netwolves.com> wrote:
>> On 09/27/2012 11:01 AM, Manish Kathuria wrote:
>>
>> On Thu, Sep 27, 2012 at 7:04 PM, Steve Clark <sclark@netwolves.com> wrote:
>>
>> On 09/26/2012 11:57 PM, Manish Kathuria wrote:
>>
>> On Thu, Sep 27, 2012 at 7:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>>
>> On 09/26/2012 09:15 AM, Steve Clark wrote:
>>
>> The routes-x.y-z.diff is a unified patch containing different parts
>> which include support for Dead Gateway Detection as well. However,
>> since that is limited to the first hop, it is preferable to have a
>> userspace script as you are doing. I also use a script to check the
>> accessibility of a remote popular site from each of the ISPs and based
>> upon the response the links are treated alive or dead and the default
>> gateway is changed. However, the routing problem as described by you
>> will only be solved after applying this patch (routes-x.y-z.diff).
>>
>> As for marking the incoming packets to ensure that they go out from
>> the same interface they came from, you could do something like the
>> following:
>>
>> Using iptables mark the incoming traffic from external interfaces
>>
>> /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
>> /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
>> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j MARK --set-mark 1
>> /sbin/iptables -t mangle -A PREROUTING -i eth1 -j CONNMARK --save-mark
>> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j MARK --set-mark 2
>> /sbin/iptables -t mangle -A PREROUTING -i eth2 -j CONNMARK --save-mark
>>
>> Add the following rules to your existing ones for policy routing
>>
>> /sbin/ip rule add fwmark 1 table T1
>> /sbin/ip rule add fwmark 2 table T2
>>
>> Hi Manish,
>>
>> Thanks for the info. The one question I have is about
>> /sbin/iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
>>
>> I thought the OUTPUT chain was only for packets originating locally. I am
>> only concerned
>> with clients behind my Linux router, do I still need this?
> Yes you are right but in case if you have any services running on the
> linux router itself (for example sshd) and accessible from the
> internet, it would help.
>
>
Hi Manish,
The above rules appear to be for clients coming into the router from external. They
don't solve the problem for clients inside the router going out thru the load balanced
interfaces.
I have done much googling and testing without much luck. At this point in time I would
be satisfied with just being able to have a client inside the router do FTP over just one
of the outbound interfaces without any load balancing for FTP. I have this working but only
for active mode FTP by using the following:
/sbin/iptables -t mangle -A PREROUTING -p tcp --dport 20:21 -j MARK --set-mark 1
But it doesn't work for passive because you don't know what ports are going to be used.
Regards,
--
Stephen Clark
*NetWolves*
Director of Technology
Phone: 813-579-3200
Fax: 813-882-0209
Email: steve.clark@netwolves.com
http://www.netwolves.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Routing issue
