Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Infrastructure (http://www.linux-archive.org/fedora-infrastructure/)
-   -   Freeze break: puppet update (http://www.linux-archive.org/fedora-infrastructure/656615-freeze-break-puppet-update.html)

Stephen John Smoogen 04-16-2012 04:46 PM

Freeze break: puppet update
 
+1 From smooge. Test plan looks best for our setup.


--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Years ago my mother used to say to me,... Elwood, you must be oh
so smart or oh so pleasant. Well, for years I was smart. I
recommend pleasant. You may quote me." Â*—James Stewart as Elwood P. Dowd
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Toshio Kuratomi 04-16-2012 04:56 PM

Freeze break: puppet update
 
On Mon, Apr 16, 2012 at 10:27:52AM -0600, Kevin Fenzi wrote:
> Greetings
>
> There's a new puppet release that has a number of security fixes to it.
>
> I'd like to do the following:
>
> - Update puppet in our stg machines and wait an hour for it to run
> twice ok on those machines. This may fail due to the older master, we
> will need to watch the output here.
>
> - Update on lockbox01 (our puppetmaster) and wait for a cycle to make
> sure all machines process ok.
>
Is the new puppet server supposed to work with old puppet clients and vice
versa? By doing this one piece at a time, like this, we're going to
experience the following scenarios:

* Old server, new client (stg at step 1)
* New server, old client (prod at step 2)
* New server, new client (stg at step 2, prod at the end phase)

So we're going to go through a more combinations than we're worried
about servicing in the end and we're not actually going to be testing our
intermediate production step at all....

Maybe our steps should be:

1 Update puppet on our stg machines. Make sure that runs twice ok
2 Update puppet on our production client machines. Make sure those run
twice okay.
3 Update the puppet server with the option to revert.

That way, step 1 and 2 are tested. Only step 3 is untested by us (which
should be the most tested step by upstream/other sites). And if we have to
revert step 3 we'll only be reverting the puppetmaster on lockbox. (It
kinda sucks that we won't be testing the final configuration in this
scenario. But it avoids the stage where, for the whole time we're testing
the final configuration in stg we're running an untested configuration in
production).

> - Finally, update puppet on all the other boxes.
>
> We should be able to back out to the current version at any of these
> points if it fails.
>
Do we know that the new puppet on lockbox won't create new data structures
that the old puppet won't understand? If we don't, perhaps we want to take
an lvm snapshot before we update to the new puppet on the server (and tell
people not to make puppet git commits until we know we're not going to
back out).

After looking at the security issues this solves, I think we do want to go
ahead with the update. But we should: 1) make sure we understand what it
takes to back out of this. 2) Alert the other groups that we think this is
important enough that we have to do it now but if we have difficulties it
could impact getting the beta release out on time.

-Toshio
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Toshio Kuratomi 04-16-2012 05:04 PM

Freeze break: puppet update
 
On Mon, Apr 16, 2012 at 10:46:25AM -0600, Stephen John Smoogen wrote:
> +1 From smooge. Test plan looks best for our setup.
>
If not obvious from my last message:

+1 from me to the update whichever way we decide is best to do it.

-Toshio
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Kevin Fenzi 04-16-2012 05:26 PM

Freeze break: puppet update
 
On Mon, 16 Apr 2012 09:56:25 -0700
Toshio Kuratomi <a.badger@gmail.com> wrote:

> Is the new puppet server supposed to work with old puppet clients and
> vice versa? By doing this one piece at a time, like this, we're
> going to experience the following scenarios:
>
> * Old server, new client (stg at step 1)
> * New server, old client (prod at step 2)
> * New server, new client (stg at step 2, prod at the end phase)
>
> So we're going to go through a more combinations than we're worried
> about servicing in the end and we're not actually going to be testing
> our intermediate production step at all....

True. Yes, I think either client/server should work.

> Maybe our steps should be:
>
> 1 Update puppet on our stg machines. Make sure that runs twice ok
> 2 Update puppet on our production client machines. Make sure those
> run twice okay.
> 3 Update the puppet server with the option to revert.
>
> That way, step 1 and 2 are tested. Only step 3 is untested by us
> (which should be the most tested step by upstream/other sites). And
> if we have to revert step 3 we'll only be reverting the puppetmaster
> on lockbox. (It kinda sucks that we won't be testing the final
> configuration in this scenario. But it avoids the stage where, for
> the whole time we're testing the final configuration in stg we're
> running an untested configuration in production).

Yeah, I like that better... easier/faster to revert.

> > - Finally, update puppet on all the other boxes.
> >
> > We should be able to back out to the current version at any of these
> > points if it fails.
> >
> Do we know that the new puppet on lockbox won't create new data
> structures that the old puppet won't understand? If we don't,
> perhaps we want to take an lvm snapshot before we update to the new
> puppet on the server (and tell people not to make puppet git commits
> until we know we're not going to back out).

It's not supposed to.

I can make a backup of the puppet files tho just to be sure.

> After looking at the security issues this solves, I think we do want
> to go ahead with the update. But we should: 1) make sure we
> understand what it takes to back out of this. 2) Alert the other
> groups that we think this is important enough that we have to do it
> now but if we have difficulties it could impact getting the beta
> release out on time.

i can also lock commits by changing perms on the git repo while we are
updating.

kevin


_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure


All times are GMT. The time now is 02:31 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.