Hosted plans
 

Greetings.

I thought I would throw out some plans for hosted moving forward and
see if we could hash out a plan and some timetable for implementing
things.

== Current status/setup/problems ==

Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
There is an hourly rsync of data from 01->02. 02 is completely standby,
it takes none of the load or requests, it's simply there in case
hosted01 dies.

Hosted01 has the following items on it:

* apache/httpd
gitweb
other scm web
trac
mailman archive http access
source downloads (tar.gz, etc)
loggerhead
reviewboard

* rsync
* scm checkouts
* scm checkins
* lists.fedorahosted.org mailman
* spamassassin for lists.

Issues with current setup:

* hosted01 is under heavy load much of the time. Needs to be more
responsive.
* hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
* If hosted01 dies we could loose up to 1 hour of data with switching
to hosted02.
* hosted01/02 are at serverbeach. If they are off line, hosted is off
line.

=== Short term plan ===

* Create a hosted-lists01/02 (or whatever we want to call them)
sync all lists and list data to it.
make sure dns is happy on it.
Move all lists.fedorahosted.org over to these new machines.

This will move some of the load off the existing hosted01/02 setup,
and should be a pretty easy thing to do. We will just setup them like
collab01/02 and hosted01/02 are with a live machine and hot spare
(unless we can use some shared fs somehow?)

Questions:

- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 23GB of mailman archives on hosted01 currently).

* Create a hosted03/04
Setup rhel6 versions of hosted01/02
Setup in similar way, except with no lists (see above).

Ask hosted project owners to 'opt-in' to moving to the new machines.
Get a small pool of things at first, then move more as any issues are
fixed. Once everything looks stable, announce a cut over date, and just
move all the rest over then.

Questions:
- Should these be at serverbeach? Or one there and one somewhere else?
(There is approx 90GB of data on hosted01 currently).

* Look at some caching. Could we add memcached on hosted03/04 easily?
What about varnish?

Questions:
- What of the things we have would use caching well?

=== Long term plan ===

We kicked around some plans here in several of our meetings, but did a
poor job of recording that anywhere. ;)

Some things we talked about:

* A caching frontend of some kind. Could cache web requests and take
load off the main machines.
* Some way to distribute the data, so if serverbeach were off line we
could still switch to and use another machine.
* Adding more/different services.
IRC commit bot
redmine (still not in fedora, but people are working on it)
your idea here.

So, what do folks think? Should we try and come up with more concrete
longer term plans before acting on the short term ones?
Are there other paths forward that make sense here?

kevin
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 8 Aug 2011, Kevin Fenzi wrote:

> Greetings.
>
> I thought I would throw out some plans for hosted moving forward and
> see if we could hash out a plan and some timetable for implementing
> things.
>
> == Current status/setup/problems ==
>
> Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
> There is an hourly rsync of data from 01->02. 02 is completely standby,
> it takes none of the load or requests, it's simply there in case
> hosted01 dies.
>
> Hosted01 has the following items on it:
>
> * apache/httpd
> gitweb
> other scm web
> trac
> mailman archive http access
> source downloads (tar.gz, etc)
> loggerhead
> reviewboard
>
> * rsync
> * scm checkouts
> * scm checkins
> * lists.fedorahosted.org mailman
> * spamassassin for lists.
>
> Issues with current setup:
>
> * hosted01 is under heavy load much of the time. Needs to be more
> responsive.
> * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
> * If hosted01 dies we could loose up to 1 hour of data with switching
> to hosted02.
> * hosted01/02 are at serverbeach. If they are off line, hosted is off
> line.
>
> === Short term plan ===
>
> * Create a hosted-lists01/02 (or whatever we want to call them)
> sync all lists and list data to it.
> make sure dns is happy on it.
> Move all lists.fedorahosted.org over to these new machines.
>
> This will move some of the load off the existing hosted01/02 setup,
> and should be a pretty easy thing to do. We will just setup them like
> collab01/02 and hosted01/02 are with a live machine and hot spare
> (unless we can use some shared fs somehow?)
>
> Questions:
>
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 23GB of mailman archives on hosted01 currently).
>
> * Create a hosted03/04
> Setup rhel6 versions of hosted01/02
> Setup in similar way, except with no lists (see above).
>
> Ask hosted project owners to 'opt-in' to moving to the new machines.
> Get a small pool of things at first, then move more as any issues are
> fixed. Once everything looks stable, announce a cut over date, and just
> move all the rest over then.
>
> Questions:
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 90GB of data on hosted01 currently).
>
> * Look at some caching. Could we add memcached on hosted03/04 easily?
> What about varnish?
>

My understanding of the load issues has always been that the trac git
plugin is very inefficient. It'd probably be worth it to see if the
load problems magically vanish by updating trac and the git plugin before
committing to any archtectural changes related to performance.

-Mike
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 8 Aug 2011 12:01:33 -0500 (CDT)
Mike McGrath <mmcgrath@redhat.com> wrote:

> My understanding of the load issues has always been that the trac git
> plugin is very inefficient. It'd probably be worth it to see if the
> load problems magically vanish by updating trac and the git plugin
> before committing to any archtectural changes related to performance.

Yeah, that could well be the case. I know it leaves around defunct
processes all the time. ;)

However, I looking today, the issue seems to be I/O.

There's a mailman archive process:
mailman 1933 17.8 1.3 98980 92432 ? R Aug01 1821:31 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
and the rsync to hosted02:
root 5955 4.9 0.4 63772 34224 ? Ds 15:53 3:53 rsync --daemon

sucking up most of the I/O. httpd is trying to squeeze in there. ;)

(As a side note, we should get apache-status working on there, might
tell us more).

kevin
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

> plugin is very inefficient.

That seems like a logical place to start.

It seems like a shared FS solution might fit but that doesn't
necessarily protect against datacenter failure. The two extra hosts in
another location would probably help prevent that situation.
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Aug 8, 2011, at 10:01, Mike McGrath <mmcgrath@redhat.com> wrote:

> On Mon, 8 Aug 2011, Kevin Fenzi wrote:
>
>> Greetings.
>>
>> I thought I would throw out some plans for hosted moving forward and
>> see if we could hash out a plan and some timetable for implementing
>> things.
>>
>> == Current status/setup/problems ==
>>
>> Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
>> There is an hourly rsync of data from 01->02. 02 is completely standby,
>> it takes none of the load or requests, it's simply there in case
>> hosted01 dies.
>>
>> Hosted01 has the following items on it:
>>
>> * apache/httpd
>> gitweb
>> other scm web
>> trac
>> mailman archive http access
>> source downloads (tar.gz, etc)
>> loggerhead
>> reviewboard
>>
>> * rsync
>> * scm checkouts
>> * scm checkins
>> * lists.fedorahosted.org mailman
>> * spamassassin for lists.
>>
>> Issues with current setup:
>>
>> * hosted01 is under heavy load much of the time. Needs to be more
>> responsive.
>> * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
>> * If hosted01 dies we could loose up to 1 hour of data with switching
>> to hosted02.
>> * hosted01/02 are at serverbeach. If they are off line, hosted is off
>> line.
>>
>> === Short term plan ===
>>
>> * Create a hosted-lists01/02 (or whatever we want to call them)
>> sync all lists and list data to it.
>> make sure dns is happy on it.
>> Move all lists.fedorahosted.org over to these new machines.
>>
>> This will move some of the load off the existing hosted01/02 setup,
>> and should be a pretty easy thing to do. We will just setup them like
>> collab01/02 and hosted01/02 are with a live machine and hot spare
>> (unless we can use some shared fs somehow?)
>>
>> Questions:
>>
>> - Should these be at serverbeach? Or one there and one somewhere else?
>> (There is approx 23GB of mailman archives on hosted01 currently).
>>
>> * Create a hosted03/04
>> Setup rhel6 versions of hosted01/02
>> Setup in similar way, except with no lists (see above).
>>
>> Ask hosted project owners to 'opt-in' to moving to the new machines.
>> Get a small pool of things at first, then move more as any issues are
>> fixed. Once everything looks stable, announce a cut over date, and just
>> move all the rest over then.
>>
>> Questions:
>> - Should these be at serverbeach? Or one there and one somewhere else?
>> (There is approx 90GB of data on hosted01 currently).
>>
>> * Look at some caching. Could we add memcached on hosted03/04 easily?
>> What about varnish?
>>
>
> My understanding of the load issues has always been that the trac git
> plugin is very inefficient. It'd probably be worth it to see if the
> load problems magically vanish by updating trac and the git plugin before
> committing to any archtectural changes related to performance.
>
> -Mike

In my experience the git plugin has been the major performance hit on memory strapped systems, even with the devel branches.
Just throwing this out there but getting people to use gitweb over trac browser might help performance although is not ideal.

--Brennan Ashton
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
> Hosted01 has the following items on it:
>
> * apache/httpd
> gitweb
> other scm web
> trac
> mailman archive http access
> source downloads (tar.gz, etc)
> loggerhead
> reviewboard

I've been talking with Mike McGrath about ReviewBoard. I really want to
get ReviewBoard off of Hosted, as the performance is incredibly poor and
the FAS integration causes problems that I have not had a chance to
identify.

Furthermore, starting with ReviewBoard 1.5.6 (being released upstream
soon), I've submitted patches to make it possible to use ReviewBoard
against any FedoraHosted git repository remotely. The main reason that
ReviewBoard was located on FedoraHosted to begin with is because it
needed direct access to the git repositories. So I'd like to move
ReviewBoard to one of the app servers or into an OpenShift instance.

Of course, we still have issues regarding the FAS integration. For
reasons I've still not been able to nail down, it causes us to lose
access to the server. I was hoping to switch over to using OpenID with
the release of ReviewBoard 1.6, but unfortunately they've deferred that
feature until 1.7.

So I'm proposing the following options:

1) Move our existing ReviewBoard instance to one of the app servers.
This will significantly improve the performance and responsiveness, but
we'll still have no email notification support (due to as-yet-unknown
negative interaction with FAS integration)
2) Move ReviewBoard to an app server and drop integration with FAS and
allow standard enrollment for users, be they Fedora users or not. This
will solve the performance and email issues, but results in a server
running on Fedora systems that is not using Fedora accounts. Also I'm
not sure we can maintain the existing review histories for the few
projects currently using the system.
3) Turn ReviewBoard into a turnkey OpenShift virtual instance and allow
any Fedora Hosted project to spin one up. This instance would use
standard enrollment (rather than FAS integration, which is impossible
outside the Infra firewall). Each project could have its own complete
instance to maintain on its own. Upsides: less work for Fedora Admins,
support for email and better performance. Downsides: no
centrally-managed user accounts and projects need to do more of the
maintaining of the system themselves.

I'm all ears for a fourth (or fifth...) option.
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 8 Aug 2011, Kevin Fenzi wrote:

> On Mon, 8 Aug 2011 12:01:33 -0500 (CDT)
> Mike McGrath <mmcgrath@redhat.com> wrote:
>
> > My understanding of the load issues has always been that the trac git
> > plugin is very inefficient. It'd probably be worth it to see if the
> > load problems magically vanish by updating trac and the git plugin
> > before committing to any archtectural changes related to performance.
>
> Yeah, that could well be the case. I know it leaves around defunct
> processes all the time. ;)
>
> However, I looking today, the issue seems to be I/O.
>
> There's a mailman archive process:
> mailman 1933 17.8 1.3 98980 92432 ? R Aug01 1821:31 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
> and the rsync to hosted02:
> root 5955 4.9 0.4 63772 34224 ? Ds 15:53 3:53 rsync --daemon
>
> sucking up most of the I/O. httpd is trying to squeeze in there. ;)
>
> (As a side note, we should get apache-status working on there, might
> tell us more).
>

https://admin.fedoraproject.org/status/hosted01

We've got that. The IO is what I saw with git as well. It doesn't
properly cache so all page loads have to craw the related git repo.
Compounding this issue is web crawlers but I think we have/had something
in place to help limit this.

-Mike
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 8 Aug 2011, Stephen Gallagher wrote:

> On Mon, 2011-08-08 at 10:54 -0600, Kevin Fenzi wrote:
> > Hosted01 has the following items on it:
> >
> > * apache/httpd
> > gitweb
> > other scm web
> > trac
> > mailman archive http access
> > source downloads (tar.gz, etc)
> > loggerhead
> > reviewboard
>
> I've been talking with Mike McGrath about ReviewBoard. I really want to
> get ReviewBoard off of Hosted, as the performance is incredibly poor and
> the FAS integration causes problems that I have not had a chance to
> identify.
>
> Furthermore, starting with ReviewBoard 1.5.6 (being released upstream
> soon), I've submitted patches to make it possible to use ReviewBoard
> against any FedoraHosted git repository remotely. The main reason that
> ReviewBoard was located on FedoraHosted to begin with is because it
> needed direct access to the git repositories. So I'd like to move
> ReviewBoard to one of the app servers or into an OpenShift instance.
>
> Of course, we still have issues regarding the FAS integration. For
> reasons I've still not been able to nail down, it causes us to lose
> access to the server. I was hoping to switch over to using OpenID with
> the release of ReviewBoard 1.6, but unfortunately they've deferred that
> feature until 1.7.
>
> So I'm proposing the following options:
>
> 1) Move our existing ReviewBoard instance to one of the app servers.
> This will significantly improve the performance and responsiveness, but
> we'll still have no email notification support (due to as-yet-unknown
> negative interaction with FAS integration)
> 2) Move ReviewBoard to an app server and drop integration with FAS and
> allow standard enrollment for users, be they Fedora users or not. This
> will solve the performance and email issues, but results in a server
> running on Fedora systems that is not using Fedora accounts. Also I'm
> not sure we can maintain the existing review histories for the few
> projects currently using the system.
> 3) Turn ReviewBoard into a turnkey OpenShift virtual instance and allow
> any Fedora Hosted project to spin one up. This instance would use
> standard enrollment (rather than FAS integration, which is impossible
> outside the Infra firewall). Each project could have its own complete
> instance to maintain on its own. Upsides: less work for Fedora Admins,
> support for email and better performance. Downsides: no
> centrally-managed user accounts and projects need to do more of the
> maintaining of the system themselves.
>

FYI, just in the last hour or so we've got full support for this in
OpenShift. Here's a howto on running reviewboard:

https://github.com/openshift/reviewboard-example

-Mike
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Monday, August 08, 2011 11:54:30 AM Kevin Fenzi wrote:
> Greetings.
>
> I thought I would throw out some plans for hosted moving forward and
> see if we could hash out a plan and some timetable for implementing
> things.
>
> == Current status/setup/problems ==
>
> Hosted is 2 machines. One primary (hosted01) and one spare (hosted02).
> There is an hourly rsync of data from 01->02. 02 is completely standby,
> it takes none of the load or requests, it's simply there in case
> hosted01 dies.
>
> Hosted01 has the following items on it:
>
> * apache/httpd
> gitweb
> other scm web
> trac
> mailman archive http access
> source downloads (tar.gz, etc)
> loggerhead
> reviewboard
>
> * rsync
> * scm checkouts
> * scm checkins
> * lists.fedorahosted.org mailman
> * spamassassin for lists.
>
> Issues with current setup:
>
> * hosted01 is under heavy load much of the time. Needs to be more
> responsive.
> * hosted01 is rhel5 and very old trac. rhel6 has a much newer trac.
> * If hosted01 dies we could loose up to 1 hour of data with switching
> to hosted02.
> * hosted01/02 are at serverbeach. If they are off line, hosted is off
> line.
>
> === Short term plan ===
>
> * Create a hosted-lists01/02 (or whatever we want to call them)
> sync all lists and list data to it.
> make sure dns is happy on it.
> Move all lists.fedorahosted.org over to these new machines.
>
> This will move some of the load off the existing hosted01/02 setup,
> and should be a pretty easy thing to do. We will just setup them like
> collab01/02 and hosted01/02 are with a live machine and hot spare
> (unless we can use some shared fs somehow?)
>
> Questions:
>
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 23GB of mailman archives on hosted01 currently).
>
> * Create a hosted03/04
> Setup rhel6 versions of hosted01/02
> Setup in similar way, except with no lists (see above).
>
> Ask hosted project owners to 'opt-in' to moving to the new machines.
> Get a small pool of things at first, then move more as any issues are
> fixed. Once everything looks stable, announce a cut over date, and just
> move all the rest over then.
>
> Questions:
> - Should these be at serverbeach? Or one there and one somewhere else?
> (There is approx 90GB of data on hosted01 currently).
>
> * Look at some caching. Could we add memcached on hosted03/04 easily?
> What about varnish?
>
> Questions:
> - What of the things we have would use caching well?
>
> === Long term plan ===
>
> We kicked around some plans here in several of our meetings, but did a
> poor job of recording that anywhere. ;)
>
> Some things we talked about:
>
> * A caching frontend of some kind. Could cache web requests and take
> load off the main machines.
> * Some way to distribute the data, so if serverbeach were off line we
> could still switch to and use another machine.
> * Adding more/different services.
> IRC commit bot
> redmine (still not in fedora, but people are working on it)
> your idea here.
>
> So, what do folks think? Should we try and come up with more concrete
> longer term plans before acting on the short term ones?
> Are there other paths forward that make sense here?
>
> kevin

another thing ive wanted to test and look into is moving to using something
like glusterfs as the backend storage so that we can scale beyond more than
one box without users needing to know what box there proect is stored on.

Dennis
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure

Hosted plans
 

On Mon, 2011-08-08 at 15:23 -0500, Dennis Gilmore wrote:

> another thing ive wanted to test and look into is moving to using something
> like glusterfs as the backend storage so that we can scale beyond more than
> one box without users needing to know what box there proect is stored on.
>

I looked at glusterfs and cloudfs(now hekafs) for this and the answer is
'maybe' if we're on a lan - not on a wan and
'maybe' if we can cope with the groups/acls/selinux limitations and/or
wait for patches to be upstreamed.

-sv


_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure