Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Development (http://www.linux-archive.org/fedora-development/)
-   -   Outage: Fedora related services down 2010-12-15 22:00 UTC (http://www.linux-archive.org/fedora-development/466240-outage-fedora-related-services-down-2010-12-15-22-00-utc.html)

"Nick Bebout" 12-15-2010 11:47 PM

Outage: Fedora related services down 2010-12-15 22:00 UTC
 
Due to problems with some NFS in our PHX2 facility we are experiencing
diminished capacity to several of our services. We are working with
our provider and engineers on how to deal with this issue soon.

To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:

date -d '2010-12-15 22:00 UTC'

Reason for outage:
NFS operations with filer are peaking below expected rates causing
hangs on NFS clients.

Affected Services:

BFO - http://boot.fedoraproject.org/
Buildsystem - http://koji.fedoraproject.org/
CVS / Source Control
Main Website - http://fedoraproject.org/
Mirror List - https://mirrors.fedoraproject.org/
Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
Package Database - https://admin.fedoraproject.org/pkgdb/


Unaffected Services:

Bodhi - https://admin.fedoraproject.org/updates/
DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
Docs - http://docs.fedoraproject.org/
Email system
Fedora Account System - https://admin.fedoraproject.org/accounts/
Fedora Community - https://admin.fedoraproject.org/community/
Fedora Hosted - https://fedorahosted.org/
Fedora People - http://fedorapeople.org/
Fedora Talk - http://talk.fedoraproject.org/
Smolt - http://smolts.org/
Spins - http://spins.fedoraproject.org/
Start - http://start.fedoraproject.org/
Torrent - http://torrent.fedoraproject.org/
Translation Services - http://translate.fedoraproject.org/
Wiki - http://fedoraproject.org/wiki/


Ticket Link:

https://fedorahosted.org/fedora-infrastructure/ticket/2516

Contact Information:

Please join #fedora-admin in irc.freenode.net or respond to this email to
track the status of this outage.

--
Nick Bebout
Fedora Project Infrastructure Team
nb@fedoraproject.org


_______________________________________________
devel-announce mailing list
devel-announce@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel-announce
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Stephen John Smoogen 12-17-2010 10:38 PM

Outage: Fedora related services down 2010-12-15 22:00 UTC
 
This outage is considered finished and closed at the moment.

Due to some change in network, DNS traffic was not being properly
moving between Fedora network and Red Hat filer. It appears that in
doing so the filer's NFS permission caches were being flooded with bad
lookups thus increasing CPU and lowering throughput.


On Wed, Dec 15, 2010 at 15:42, Stephen John Smoogen <smooge@gmail.com> wrote:
> Due to problems with some NFS in our PHX2 facility we are experiencing
> diminished capacity to several of our services. We are working with
> our provider and engineers on how to deal with this issue soon.
>
> To convert UTC to your local time, take a look at
> http://fedoraproject.org/wiki/Infrastructure/UTCHowto
> or run:
>
> date -d '2010-12-15 22:00 UTC'
>
> Reason for outage:
> NFS operations with filer are peaking below expected rates causing
> hangs on NFS clients.
>
> Affected Services:
>
> BFO - http://boot.fedoraproject.org/
> Buildsystem - http://koji.fedoraproject.org/
> CVS / Source Control
> Main Website - http://fedoraproject.org/
> Mirror List - https://mirrors.fedoraproject.org/
> Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
> Package Database - https://admin.fedoraproject.org/pkgdb/
>
>
> Unaffected Services:
>
> Bodhi - https://admin.fedoraproject.org/updates/
> DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
> Docs - http://docs.fedoraproject.org/
> Email system
> Fedora Account System - https://admin.fedoraproject.org/accounts/
> Fedora Community - https://admin.fedoraproject.org/community/
> Fedora Hosted - https://fedorahosted.org/
> Fedora People - http://fedorapeople.org/
> Fedora Talk - http://talk.fedoraproject.org/
> Smolt - http://smolts.org/
> Spins - http://spins.fedoraproject.org/
> Start - http://start.fedoraproject.org/
> Torrent - http://torrent.fedoraproject.org/
> Translation Services - http://translate.fedoraproject.org/
> Wiki - http://fedoraproject.org/wiki/
>
>
> Ticket Link:
>
> https://fedorahosted.org/fedora-infrastructure/ticket/2516
>
> Contact Information:
>
> Please join #fedora-admin in irc.freenode.net or respond to this email to
> track the status of this outage.
>
>
> --
> Stephen J Smoogen.
> "The core skill of innovators is error recovery, not failure avoidance."
> Randy Nelson, President of Pixar University.
> "Let us be kind, one to another, for most of us are fighting a hard
> battle." -- Ian MacLaren
>



--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Stephen John Smoogen 12-17-2010 10:38 PM

Outage: Fedora related services down 2010-12-15 22:00 UTC
 
This outage is considered finished and closed at the moment.

Due to some change in network, DNS traffic was not being properly
moving between Fedora network and Red Hat filer. It appears that in
doing so the filer's NFS permission caches were being flooded with bad
lookups thus increasing CPU and lowering throughput.


On Wed, Dec 15, 2010 at 15:42, Stephen John Smoogen <smooge@gmail.com> wrote:
> Due to problems with some NFS in our PHX2 facility we are experiencing
> diminished capacity to several of our services. We are working with
> our provider and engineers on how to deal with this issue soon.
>
> To convert UTC to your local time, take a look at
> http://fedoraproject.org/wiki/Infrastructure/UTCHowto
> or run:
>
> date -d '2010-12-15 22:00 UTC'
>
> Reason for outage:
> NFS operations with filer are peaking below expected rates causing
> hangs on NFS clients.
>
> Affected Services:
>
> BFO - http://boot.fedoraproject.org/
> Buildsystem - http://koji.fedoraproject.org/
> CVS / Source Control
> Main Website - http://fedoraproject.org/
> Mirror List - https://mirrors.fedoraproject.org/
> Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
> Package Database - https://admin.fedoraproject.org/pkgdb/
>
>
> Unaffected Services:
>
> Bodhi - https://admin.fedoraproject.org/updates/
> DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
> Docs - http://docs.fedoraproject.org/
> Email system
> Fedora Account System - https://admin.fedoraproject.org/accounts/
> Fedora Community - https://admin.fedoraproject.org/community/
> Fedora Hosted - https://fedorahosted.org/
> Fedora People - http://fedorapeople.org/
> Fedora Talk - http://talk.fedoraproject.org/
> Smolt - http://smolts.org/
> Spins - http://spins.fedoraproject.org/
> Start - http://start.fedoraproject.org/
> Torrent - http://torrent.fedoraproject.org/
> Translation Services - http://translate.fedoraproject.org/
> Wiki - http://fedoraproject.org/wiki/
>
>
> Ticket Link:
>
> https://fedorahosted.org/fedora-infrastructure/ticket/2516
>
> Contact Information:
>
> Please join #fedora-admin in irc.freenode.net or respond to this email to
> track the status of this outage.
>
>
> --
> Stephen J Smoogen.
> "The core skill of innovators is error recovery, not failure avoidance."
> Randy Nelson, President of Pixar University.
> "Let us be kind, one to another, for most of us are fighting a hard
> battle." -- Ian MacLaren
>



--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
_______________________________________________
infrastructure mailing list
infrastructure@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/infrastructure


All times are GMT. The time now is 05:49 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.