FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Infrastructure

 
 
LinkBack Thread Tools
 
Old 09-16-2011, 06:58 PM
Kevin Fenzi
 
Default Freeze break request: Switch back to bastion02 for now

We've continued to have issues with bastion03 and this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=725332

It's been requiring a reboot every day or two, resulting in 5-10min of
downtime and about 90 pages. ;(

I hate changes on fridays and more so during a freeze, but I think we
need to switch back to bastion02 for now to avoid this issue until we
can get a fix.

So, I would like to:

- commit the following patch.
- puppet update nameservers to get the new info.
- puppet update bastion02/03 to get openvpn running on 02 and stopped
on 03
- Make sure everything reconnects.

Unfortunately this will result in a small outage, but no worse than the
bastion03 ones have been. If we don't want to do it now, I can wait
until the next time bastion03 freaks out and just change it then, since
it should be all prepped below:

diff --git a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp b/manifests/nodes/bastion02.phx2.fe
index 4018ec9..1a0ee7c 100644
--- a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
@@ -1,6 +1,5 @@
node bastion02{
- # Moving openvpn over to bastion03
- $enable_openvpn = false
+ $enable_openvpn = true
include phx
$syncFasAliases = true
include gateway
diff --git a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp b/manifests/nodes/bastion03.phx2.fe
index 8c5fca9..b7b0f32 100644
--- a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
@@ -3,7 +3,7 @@ node bastion03{
# comment out the line below when bastion02 is down or going to be down.
# Under normal situations, only one bastion host should be running openvpn
# or we'll end up with a split-brain problem in the network
- #$enable_openvpn = false
+ $enable_openvpn = false
include phx
$syncFasAliases = true
include gateway
diff --git a/modules/bind/files/master/fedoraproject.org b/modules/bind/files/master/fedoraproject.o
index 5b72f2d..f3dc836 100644
--- a/modules/bind/files/master/fedoraproject.org
+++ b/modules/bind/files/master/fedoraproject.org
@@ -1,6 +1,6 @@
$TTL 3600
@ IN SOA ns01.fedoraproject.org. hostmaster.fedoraproject.org. (
- 2011091301 ; Serial
+ 2011091601 ; Serial
1H ; refresh
10M ; retry
4W ; expire
@@ -85,7 +85,7 @@ autoqa-stg IN A 209.132.181.9
; need rhit to fix.
bastion01 IN A 209.132.181.2
bastion02 IN A 209.132.181.3
-bastion IN A 209.132.181.2
+bastion IN A 209.132.181.3
bastion-comm01 IN A 209.132.181.13
backup02 IN A 152.19.134.140
blogs IN CNAME wildcard
diff --git a/modules/bind/files/master/phx2.fedoraproject.org b/modules/bind/files/master/phx2.fedor
index b8caea3..7c9eed7 100644
--- a/modules/bind/files/master/phx2.fedoraproject.org
+++ b/modules/bind/files/master/phx2.fedoraproject.org
@@ -28,7 +28,7 @@ bapp01 IN A 10.5.126.38
bapp1 IN CNAME bapp01
bapp02 IN A 10.5.126.39
bapp2 IN CNAME bapp02
-bastion IN CNAME bastion03
+bastion IN CNAME bastion02
;bastion01 IN A 10.5.126.13
;bastion1 IN CNAME bastion01
bastion02 IN A 10.5.126.11
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 
Old 09-16-2011, 07:05 PM
Kevin Fenzi
 
Default Freeze break request: Switch back to bastion02 for now

To followup on myself and after an excellent suggestion from Seth...

Another way to do this is to just move vpn over to bastion02, don't
change dns or email. 'gateway' and 'bastion' stay pointed at bastion03.

This means a change to the vpn client.conf on all machines, but we can
push that out. It also means no DNS changes, which is good.
We can also change back by just changing which machine is running
openvpn server on it.

Here's the change for that:

diff --git a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
b/manifests/nodes/bastion02.phx2.fe index 4018ec9..1a0ee7c 100644
--- a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
@@ -1,6 +1,5 @@
node bastion02{
- # Moving openvpn over to bastion03
- $enable_openvpn = false
+ $enable_openvpn = true
include phx
$syncFasAliases = true
include gateway
diff --git a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
b/manifests/nodes/bastion03.phx2.fe index 8c5fca9..b7b0f32 100644
--- a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
+++ b/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
@@ -3,7 +3,7 @@ node bastion03{
# comment out the line below when bastion02 is down or going to be
down. # Under normal situations, only one bastion host should be
running openvpn # or we'll end up with a split-brain problem in the
network
- #$enable_openvpn = false
+ $enable_openvpn = false
include phx
$syncFasAliases = true
include gateway
diff --git a/modules/openvpn/files/client.conf
b/modules/openvpn/files/client.conf index b1b2d95..d274e72 100644
--- a/modules/openvpn/files/client.conf
+++ b/modules/openvpn/files/client.conf
@@ -6,7 +6,7 @@ proto udp

# Specify multiple vpn servers here
remote gateway
-remote bastion01
+remote bastion02

resolv-retry infinite

_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 
Old 09-16-2011, 07:25 PM
Stephen John Smoogen
 
Default Freeze break request: Switch back to bastion02 for now

On Fri, Sep 16, 2011 at 13:05, Kevin Fenzi <kevin@scrye.com> wrote:
> To followup on myself and after an excellent suggestion from Seth...
>
> Another way to do this is to just move vpn over to bastion02, don't
> change dns or email. 'gateway' and 'bastion' stay pointed at bastion03.

+1

>
--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 
Old 09-16-2011, 07:28 PM
Ricky Zhou
 
Default Freeze break request: Switch back to bastion02 for now

On 2011-09-16 01:05:02 PM, Kevin Fenzi wrote:
> To followup on myself and after an excellent suggestion from Seth...
>
> Another way to do this is to just move vpn over to bastion02, don't
> change dns or email. 'gateway' and 'bastion' stay pointed at bastion03.
>
> This means a change to the vpn client.conf on all machines, but we can
> push that out. It also means no DNS changes, which is good.
> We can also change back by just changing which machine is running
> openvpn server on it.
+1, but be careful with making sure that no VPN clients end up stranded
on the old one, as discussed on IRC.

Thanks,
Ricky
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 
Old 09-16-2011, 07:30 PM
Dennis Gilmore
 
Default Freeze break request: Switch back to bastion02 for now

On Friday, September 16, 2011 02:05:02 PM Kevin Fenzi wrote:
> To followup on myself and after an excellent suggestion from Seth...
>
> Another way to do this is to just move vpn over to bastion02, don't
> change dns or email. 'gateway' and 'bastion' stay pointed at bastion03.
>
> This means a change to the vpn client.conf on all machines, but we can
> push that out. It also means no DNS changes, which is good.
> We can also change back by just changing which machine is running
> openvpn server on it.
>
> Here's the change for that:
>
> diff --git a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
> b/manifests/nodes/bastion02.phx2.fe index 4018ec9..1a0ee7c 100644
> --- a/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
> +++ b/manifests/nodes/bastion02.phx2.fedoraproject.org.pp
> @@ -1,6 +1,5 @@
> node bastion02{
> - # Moving openvpn over to bastion03
> - $enable_openvpn = false
> + $enable_openvpn = true
> include phx
> $syncFasAliases = true
> include gateway
> diff --git a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
> b/manifests/nodes/bastion03.phx2.fe index 8c5fca9..b7b0f32 100644
> --- a/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
> +++ b/manifests/nodes/bastion03.phx2.fedoraproject.org.pp
> @@ -3,7 +3,7 @@ node bastion03{
> # comment out the line below when bastion02 is down or going to be
> down. # Under normal situations, only one bastion host should be
> running openvpn # or we'll end up with a split-brain problem in the
> network
> - #$enable_openvpn = false
> + $enable_openvpn = false
> include phx
> $syncFasAliases = true
> include gateway
> diff --git a/modules/openvpn/files/client.conf
> b/modules/openvpn/files/client.conf index b1b2d95..d274e72 100644
> --- a/modules/openvpn/files/client.conf
> +++ b/modules/openvpn/files/client.conf
> @@ -6,7 +6,7 @@ proto udp
>
> # Specify multiple vpn servers here
> remote gateway
> -remote bastion01
> +remote bastion02
>
> resolv-retry infinite
+1
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure
 

Thread Tools




All times are GMT. The time now is 04:03 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org