FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.

» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Infrastructure

LinkBack Thread Tools
Old 07-31-2008, 09:16 PM
Mike McGrath
Default Meeting this week

2008-07-31T15:59:39 *** mmcgrath changes topic to "Infrastructure -- Who's here?"
2008-07-31T15:59:43 <mmcgrath> Who's around?
2008-07-31T15:59:45 <ianweller> there are too many meetings on wednesdays
2008-07-31T15:59:50 <G> me me me!
2008-07-31T15:59:50 * ianweller
2008-07-31T15:59:58 * wakko6661 is here, but waiting for his other session to time out before reclaiming his /nick.
2008-07-31T16:00:18 <londo> here
2008-07-31T16:00:20 <ianweller> wakko6661: /msg nickserv ghost
2008-07-31T16:00:29 <fchiulli> fchiulli here
2008-07-31T16:00:50 *** wakko6661 is now known as wakko666
2008-07-31T16:00:52 <mmcgrath> ianweller: today's thursday
2008-07-31T16:00:57 <wakko666> ianweller: thanks.
2008-07-31T16:01:04 <ianweller> mmcgrath: today too!
2008-07-31T16:01:17 <ianweller> why can i never remember what day of the week it is
2008-07-31T16:01:24 <ianweller> this is going to be a problem when school rolls around
2008-07-31T16:01:27 <mmcgrath> Ok, lets get started
2008-07-31T16:01:31 <mmcgrath> ianweller: because schools out ;-)
2008-07-31T16:01:33 <mmcgrath> https://fedorahosted.org/fedora-infrastructure/query?status=new&status=assigned&status=reopened&g roup=milestone&keywords=%7EMeeting&order=priority
2008-07-31T16:01:34 <zodbot> <http://tinyurl.com/2hyyz6> (at fedorahosted.org)
2008-07-31T16:01:40 <mmcgrath> .ticket 671
2008-07-31T16:01:45 <mmcgrath> anyone from in.fedoraproject.org around?
2008-07-31T16:01:49 <mmcgrath> grr zodbot
2008-07-31T16:01:51 <SmootherFrOgZ> hello
2008-07-31T16:01:54 <mmcgrath> https://fedorahosted.org/fedora-infrastructure/ticket/671
2008-07-31T16:02:13 <ianweller> mmcgrath: since you're not using it anyway you can comment out the python-fedora fetch line in __init__ i think
2008-07-31T16:02:45 <kschreyack> present
2008-07-31T16:02:48 *** ianweller changes topic to "Infrastructure -- Tickets"
2008-07-31T16:03:01 <mmcgrath> .load Fedora
2008-07-31T16:03:01 <zodbot> mmcgrath: The operation succeeded.
2008-07-31T16:03:11 <mmcgrath> Ok, I'm removing ticket 671 if no one is around.
2008-07-31T16:03:13 *** mcgarnicle has joined #fedora-meeting
2008-07-31T16:03:21 <mmcgrath> err removing the "Meeting" flag
2008-07-31T16:03:27 <abadger1999> BTW, that method interacts truly horribly with our privacy stuff... going to have to figure something else out for it.
2008-07-31T16:03:28 <mmcgrath> we're skipping 732 for now (end of meeting)
2008-07-31T16:03:29 <G> skip it
2008-07-31T16:03:34 <mmcgrath> abadger1999: solid
2008-07-31T16:03:37 <mmcgrath> .ticket 395
2008-07-31T16:03:38 <zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/395
2008-07-31T16:03:39 <zodbot> mmcgrath: #395 (Audio Streaming of Fedora Board Conference Calls) - Fedora Infrastructure - Trac - http://tinyurl.com/5qb3gv
2008-07-31T16:03:43 <mmcgrath> jcollie: any update?
2008-07-31T16:04:07 <yingbull> mmcgrath: I'm here too, btw, just missed the start.
2008-07-31T16:05:10 <ivazquez> As am I.
2008-07-31T16:05:13 <G> jcollie said he was busy today
2008-07-31T16:05:15 *** buggbot has quit IRC
2008-07-31T16:05:16 <mmcgrath> yingbull: yo
2008-07-31T16:05:18 <mmcgrath> thats fine
2008-07-31T16:05:21 <mmcgrath> we'll skip that ticket for now.
2008-07-31T16:05:28 <mmcgrath> .ticket 446
2008-07-31T16:05:29 <zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/446
2008-07-31T16:05:30 *** buggbot has joined #fedora-meeting
2008-07-31T16:05:31 <zodbot> mmcgrath: #446 (Possibility to add external links on spins page) - Fedora Infrastructure - Trac - http://tinyurl.com/3w4uwn
2008-07-31T16:05:31 <ianweller> daaang, buggbot's having issues
2008-07-31T16:05:33 <mmcgrath> dgilmore: ping?
2008-07-31T16:05:48 *** ricky has joined #fedora-meeting
2008-07-31T16:05:55 * ricky is here
2008-07-31T16:05:58 <mmcgrath> ricky in the house
2008-07-31T16:06:07 <ricky> (ISP had a loooong outage)
2008-07-31T16:06:11 <ianweller> ricky: ouch
2008-07-31T16:06:51 *** roguedaemon has left #fedora-meeting
2008-07-31T16:07:18 <mmcgrath> ok, we'll skip that ticket as well.
2008-07-31T16:07:21 <mmcgrath> .ticket 576
2008-07-31T16:07:22 <zodbot> mmcgrath: #576 (Infrastructure Contact Information) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/576
2008-07-31T16:07:24 <zodbot> mmcgrath: #576 (Infrastructure Contact Information) - Fedora Infrastructure - Trac - http://tinyurl.com/4kevcb
2008-07-31T16:07:42 <G> mmcgrath: I seem to remember you having a proposal for that which sounds reasonable
2008-07-31T16:07:51 <mmcgrath> G: so should I just setup a simple script + encfs to get that share up? Its handy because it allows for rsync local copies as well.
2008-07-31T16:07:59 <mmcgrath> G: the only problem I haven't figured out is... shared password.
2008-07-31T16:08:04 <mmcgrath> but thats not terrible.
2008-07-31T16:08:25 <mmcgrath> we can try it out and see what we think, if its horrible we can always change it
2008-07-31T16:08:35 <G> yeah, I think thats the best option
2008-07-31T16:08:37 <ricky> Cool
2008-07-31T16:08:38 <mmcgrath> Ok, so thats it for the tickets. I had a couple of other things to dicsuss before going back to 732
2008-07-31T16:08:46 *** mmcgrath changes topic to "Infrastructure -- Me gone (again)"
2008-07-31T16:08:57 <mmcgrath> So I'll be gone... again On Monday and Tuesday.
2008-07-31T16:09:04 <ricky> Heh.
2008-07-31T16:09:06 <mmcgrath> that should be my last vacation of the year so woo hoo
2008-07-31T16:09:07 <G> mmcgrath: where is thou going?
2008-07-31T16:09:07 <kschreyack> nice
2008-07-31T16:09:23 <ianweller> mmcgrath: i don't think i've ever seen anyone happy about not having any more vacations
2008-07-31T16:09:24 <mmcgrath> G: http://www.cedarpoint.com/
2008-07-31T16:09:35 <mmcgrath> ianweller: its hard to get stuff done when you're camping
2008-07-31T16:09:44 <ricky> Ooh, fun
2008-07-31T16:09:58 <ianweller> ah, fedora. the job everyone (literally) lives to have
2008-07-31T16:09:58 <G> mmcgrath: ooh nice
2008-07-31T16:10:02 <SmootherFrOgZ> he he
2008-07-31T16:10:04 <mmcgrath> Ok, so thats that.
2008-07-31T16:10:10 *** mmcgrath changes topic to "Infrastructure -- Outage last night"
2008-07-31T16:10:16 <mmcgrath> really, wtf was that all about?
2008-07-31T16:10:22 <ricky> What about the network switch?
2008-07-31T16:10:23 <mmcgrath> Its the first time thats ever happened to us.
2008-07-31T16:10:27 <ricky> As in, does that happen while you're here?
2008-07-31T16:10:31 <mmcgrath> ricky: I haven't heard any updates on that
2008-07-31T16:10:44 <ricky> Ah, OK.
2008-07-31T16:10:46 * mmcgrath summons the notting
2008-07-31T16:11:05 <fchiulli> What was the cause of the problem?
2008-07-31T16:11:20 <mmcgrath> fchiulli: so here's what we know.
2008-07-31T16:11:25 <mmcgrath> the netapp crashed.
2008-07-31T16:11:27 <mmcgrath> not sure why
2008-07-31T16:11:33 <mmcgrath> they're still looking into it.
2008-07-31T16:11:41 <mmcgrath> but I do know that our current iptables rules got in the way of the fix.
2008-07-31T16:11:51 <mmcgrath> right now we can't mount the netapp with iptables enabled.
2008-07-31T16:11:57 <mmcgrath> after the mount is done, we can re-enable iptables.
2008-07-31T16:12:05 <mmcgrath> I looked briefly this morning but haven't figured out wtf is going on.
2008-07-31T16:12:08 <mmcgrath> I'll look more this afternoon.
2008-07-31T16:12:27 <G> and as a result it brought down all the services that had data on the netapp, i.e. db2 - fas and all that depends on it, and non-cached wiki data
2008-07-31T16:12:28 <ricky> At least, we can hopefully narrow down the tables rule
2008-07-31T16:12:34 <ricky> Did it use to work with iptables on before?
2008-07-31T16:12:36 <mmcgrath> correct.
2008-07-31T16:12:41 <mmcgrath> ricky: yeah
2008-07-31T16:12:54 <ricky> All right, so it's something we changed :-)
2008-07-31T16:12:57 <mmcgrath> I made the last change that I could find and it seemed pretty harmless.
2008-07-31T16:12:59 <londo> do the rpc services have fixed ports in the netapp?
2008-07-31T16:13:03 <mmcgrath> ricky: or its related to the rhel 5.2 update.
2008-07-31T16:13:08 <ricky> Ah
2008-07-31T16:13:09 <mmcgrath> londo: not sure.
2008-07-31T16:13:19 *** buggbot has quit IRC
2008-07-31T16:13:30 *** buggbot has joined #fedora-meeting
2008-07-31T16:13:44 <G> mmcgrath: maybe we need to say "pretty pretty please can you update our firmware"
2008-07-31T16:13:46 <ricky> Does NFS require any incoming connections?
2008-07-31T16:14:03 <mmcgrath> G: a good possibility.
2008-07-31T16:14:13 <mmcgrath> ricky: as in, does the client need to send something to the server?
2008-07-31T16:14:28 <ricky> Yeah.
2008-07-31T16:14:29 <ricky> Er.
2008-07-31T16:14:33 <ricky> The other way around
2008-07-31T16:14:43 <mmcgrath> for nfslock, I believe so yes.
2008-07-31T16:14:45 <mmcgrath> its udp based.
2008-07-31T16:14:48 <ricky> Ah
2008-07-31T16:14:55 <mmcgrath> its just something we need to look at.
2008-07-31T16:14:59 <mmcgrath> moving on...
2008-07-31T16:15:09 *** mmcgrath changes topic to "Infrastructure -- Switch upgrade."
2008-07-31T16:15:15 <mmcgrath> well, we're supposed to have a switch upgraded this week or next.
2008-07-31T16:15:24 <mmcgrath> not heard any ETA or anything about the nature of the upgrade, nothing.
2008-07-31T16:15:39 * mmcgrath sends an email
2008-07-31T16:15:48 <G> heh, classic!
2008-07-31T16:15:54 <mmcgrath> email sent.
2008-07-31T16:16:05 <mmcgrath> next
2008-07-31T16:16:10 *** mmcgrath changes topic to "Infrastructure -- Torrent"
2008-07-31T16:16:15 *** wakko666 has quit IRC
2008-07-31T16:16:17 <mmcgrath> Soooooooo. Server Beach isn't going to work out for Torrent.
2008-07-31T16:16:18 <ricky> Boo serverbeach :-(
2008-07-31T16:16:23 <mmcgrath> at least we don't think so.
2008-07-31T16:16:23 *** wakko666 has joined #fedora-meeting
2008-07-31T16:16:32 <mmcgrath> The last request I sent hasn't been answered yet so we're waiting to see.
2008-07-31T16:16:41 <mmcgrath> but there's some port blockage issues that we're seeing.
2008-07-31T16:16:41 <G> mmcgrath: so whats the current problems with SB?
2008-07-31T16:16:49 <ricky> G: They block port 6969
2008-07-31T16:16:57 <mmcgrath> G: nmap -p 6969 serverbeach1.fedoraproject.org
2008-07-31T16:16:59 <ianweller> lol w h y
2008-07-31T16:17:04 <G> 6969 = bittorrent right?
2008-07-31T16:17:11 <ianweller> can we ask them to disable that?
2008-07-31T16:17:14 <ricky> Because bittorrents are always illegal, of course.
2008-07-31T16:17:14 <mmcgrath> ianweller: ill-informed network staff I suspect.
2008-07-31T16:17:17 <mmcgrath> ianweller: we did
2008-07-31T16:17:36 <mmcgrath> I'm hoping they will move us to a different network.
2008-07-31T16:17:43 <ianweller> can we set up a gateway that funnels traffic through something if it doesn't work out?
2008-07-31T16:17:46 <mmcgrath> But there's problems....
2008-07-31T16:17:56 <mmcgrath> 1) our current torrent server died earlier this week. Its REALLY old.
2008-07-31T16:18:02 <mmcgrath> and will likely be down for the count soon.
2008-07-31T16:18:04 <ricky> Wha? It did?
2008-07-31T16:18:05 <ricky> :-(
2008-07-31T16:18:06 <G> mmcgrath: well to be honest, bit torrent can be a bit of a pain, and it might not be sb's fault, it could be the datacenter's fault
2008-07-31T16:18:26 <mmcgrath> G: its actually their parent companies fault Peer 1. Policy of the edge router
2008-07-31T16:18:42 <mmcgrath> so in the meantime seth's been hard at work and we're keeping a backup of torrent on fedorapeople.
2008-07-31T16:18:45 <skvidal> ricky: not died, dead, but it oopsed in a bad place
2008-07-31T16:18:51 <ricky> Ah.
2008-07-31T16:19:02 <skvidal> and we have another copy of the data on sb1, too
2008-07-31T16:19:05 <G> yeah exactly, AtlantaNAP filter IRC at the datacenter level
2008-07-31T16:19:25 *** sharkcz has quit IRC
2008-07-31T16:19:47 <mmcgrath> so I'm just keeping everyone in the know on that. Like I said, its not a complete no at this point, they're still looking into it and we thank them for that and sponsoring our project.
2008-07-31T16:20:03 <skvidal> does anyone here have any contacts at any other hosting providers?
2008-07-31T16:20:08 *** xaeth has joined #Fedora-meeting
2008-07-31T16:20:34 <G> skvidal: not really, but...
2008-07-31T16:20:48 <mmcgrath> I'm going to open the floor for a few minutes, then move to ticket 732.
2008-07-31T16:20:54 <mmcgrath> G: ?
2008-07-31T16:20:58 <G> I've seen people from TP (The Planet) hanging around in -admin every now and then
2008-07-31T16:21:27 <mmcgrath> might be worth a cattle call, see who answers.
2008-07-31T16:21:30 <mmcgrath> anyone have anything else on that?
2008-07-31T16:21:38 <G> They own what used to be Ev1 etc, and I know Linode have quite a few servers at TP and seem happy
2008-07-31T16:22:11 <ianweller> is it possible to load balance a bt server or is that crazy talk
2008-07-31T16:22:30 <ricky> Is there a need to?
2008-07-31T16:22:38 * ianweller is just wondering
2008-07-31T16:22:39 * ricky has no idea how intensive bt stuff is
2008-07-31T16:22:41 <ianweller> i don't see a need.
2008-07-31T16:22:43 <ianweller> yet, at least
2008-07-31T16:22:52 <skvidal> ianweller: no real need
2008-07-31T16:23:08 <mmcgrath> ianweller: thats possible yet, not much in our use case right now since we're having problems finding a single place to host a torrent server, much less multiple torrent servers
2008-07-31T16:23:15 <mmcgrath> err possible yes,
2008-07-31T16:23:15 <ianweller> i'm just thinking, an automatic way to switch to a backup tracker in case one goes *splat*
2008-07-31T16:23:29 <skvidal> ianweller: we'd need to swap out dns entry
2008-07-31T16:23:30 <ricky> ianweller: We might do it with DNS
2008-07-31T16:23:36 <SmootherFrOgZ> skvidal: i suppose you mean *us* hosting providers ?
2008-07-31T16:23:50 <skvidal> SmootherFrOgZ: anyone, really
2008-07-31T16:24:01 <mmcgrath> SmootherFrOgZ: doesn't have to be but it'd have to be fast enough for releng to copy large files to it.
2008-07-31T16:24:32 <skvidal> very very large files
2008-07-31T16:24:38 <SmootherFrOgZ> well, i'll have a talk with our hosting section on tommorow then
2008-07-31T16:25:25 <ricky> Cool
2008-07-31T16:25:30 <ricky> Just curious: Where are you?
2008-07-31T16:25:33 <G> I think really, if we were able to do GeoDNS etc a proxy server in the APAC area might be handy too
2008-07-31T16:26:01 <mmcgrath> SmootherFrOgZ: thanks, even if not for torrent we can use another proxy / cache server there.
2008-07-31T16:26:09 *** fchiulli_ has joined #fedora-meeting
2008-07-31T16:26:17 <SmootherFrOgZ> k
2008-07-31T16:26:43 *** cassmodiah has quit IRC
2008-07-31T16:26:51 <mmcgrath> Ok, moving on
2008-07-31T16:26:59 *** mmcgrath changes topic to "Infrastructure -- Open Floor (before monitoring stuff)"
2008-07-31T16:27:03 <mmcgrath> anyone have anything to discuss?
2008-07-31T16:27:09 <ianweller> wiki slowness?
2008-07-31T16:27:23 *** fchiulli has quit IRC
2008-07-31T16:27:25 <ricky> I'll be looking into alternative session storage son
2008-07-31T16:27:26 <ricky> **soon
2008-07-31T16:27:27 <ianweller> gregdek had some issues with the wiki during our marketing meeting an hour ago, apparently session problems again
2008-07-31T16:27:30 <ianweller> ricky: sweet
2008-07-31T16:27:35 <ricky> Namely, memcached
2008-07-31T16:27:35 *** buggbot has quit IRC
2008-07-31T16:28:03 <mmcgrath> ricky: yeah I ment to ask you and G if anything changed recently.
2008-07-31T16:28:07 *** buggbot has joined #fedora-meeting
2008-07-31T16:28:09 *** fchiulli_ is now known as fchiulli
2008-07-31T16:28:10 <mmcgrath> we've got a bunch of empty sessions again. I'd fixed that a while back.
2008-07-31T16:28:19 * dgilmore is here
2008-07-31T16:28:35 <G> mmcgrath: I hadn't changed anything with production recently
2008-07-31T16:29:05 <G> mmcgrath: might be something to do with the netapp weirdness etc
2008-07-31T16:29:10 <mmcgrath> possibly.
2008-07-31T16:29:14 <mmcgrath> something we'll want to look at.
2008-07-31T16:29:40 <mmcgrath> Ok, if there's nothing else, on to the ticket.
2008-07-31T16:29:54 <mmcgrath> .ticket 732
2008-07-31T16:29:55 <zodbot> mmcgrath: #732 (Investigate New System Monitoring Methods) - Fedora Infrastructure - Trac - https://fedorahosted.org/projects/fedora-infrastructure/ticket/732
2008-07-31T16:29:56 <zodbot> mmcgrath: #732 (Investigate New System Monitoring Methods) - Fedora Infrastructure - Trac - http://tinyurl.com/69whk6
2008-07-31T16:29:58 <G> okay, so I think this is my part right?
2008-07-31T16:30:01 <mmcgrath> G's got lead on this so have at it.
2008-07-31T16:30:19 *** G changes topic to "Infrastructure -- Server Monitoring"
2008-07-31T16:31:11 <G> So to recap, the last few days we have been talking about the possibility of switching from Nagios to another solution, after a bit of poking around, it seems Zabbix and Zenoss are the better of the others
2008-07-31T16:31:37 * dgilmore stays out of this
2008-07-31T16:31:47 <skvidal> not zope. please
2008-07-31T16:32:05 <G> As zenoss suffers from requiring Python 2.4 (zope) and we can't guarrantee that it'll work with RHEL6, it seems Zabbix is our primary option
2008-07-31T16:32:30 <G> skvidal: I was getting to that
2008-07-31T16:32:39 <mmcgrath> also zenoss isn't in Fedora.
2008-07-31T16:32:40 <fchiulli> Zabbix uses PHP correct?
2008-07-31T16:32:47 <G> fchiulli: it's a mix
2008-07-31T16:33:01 <fchiulli> G: of what?
2008-07-31T16:33:05 <G> The front end is fedora, but everything else I think is compiled C
2008-07-31T16:33:07 <ricky> PHP and c, it seems
2008-07-31T16:33:08 <G> errr
2008-07-31T16:33:12 <G> the Front end is PHP
2008-07-31T16:33:14 <ricky> The PHP looks very C-like too :-)
2008-07-31T16:33:48 <G> fchiulli: it's split up into 3 components, Frontend, Server and Agents
2008-07-31T16:34:46 <G> wakko666 and jcollie have put their hands up as DAYJOB users and have offered to help us out if we wish to implement it
2008-07-31T16:35:02 *** stickster is now known as stickster_mtg
2008-07-31T16:35:47 <mmcgrath> G: I got zabbix setup at home. Its got a little learning curve. I'd say not as hard as nagios though.
2008-07-31T16:36:05 <mmcgrath> Its important to get the different terms known. Event, Action, Trigger, Application, etc.
2008-07-31T16:36:19 <G> Mike proposed that it might be an idea to setup an instance alongside nagios for now, monitoring 10 or so machines in our infrastructure (I can see koji, a builder, a couple of app servers etc as primary candidates)
2008-07-31T16:36:23 <ricky> Can we get a quick rundown of this stuff if we're going to start testing in production at some point?
2008-07-31T16:37:01 <mmcgrath> ricky: well, unlike nagios, zabbix does trending so we'll be monitoring the stuff we are now, and then some.
2008-07-31T16:37:07 <mmcgrath> but we won't, for example, get notified about all of it.
2008-07-31T16:37:33 <G> okay the important bits are:
2008-07-31T16:38:01 <G> a trigger is set-off when an item been monitored say apache, goes down
2008-07-31T16:38:12 <G> in that situation the trigger is said to be TRUE
2008-07-31T16:38:56 <mmcgrath> right, in nagios I always tended to think of it as monitoring when the host was up and notifying us when its down. zabbix tends to only think of things as being down.
2008-07-31T16:39:02 <mmcgrath> IE, False == good. True == bad
2008-07-31T16:39:11 *** Sonar_Guy has joined #fedora-meeting
2008-07-31T16:39:23 <G> an event is a nice way of interpreting the triggers and a combination of triggers
2008-07-31T16:39:37 <ianweller> mmcgrath: the truth hurts
2008-07-31T16:40:10 <G> an action is performed when a trigger/action returns TRUE/ON and is something like, remote commands, e-mails, jabber messages etc
2008-07-31T16:40:17 *** buggbot has quit IRC
2008-07-31T16:40:46 *** buggbot has joined #fedora-meeting
2008-07-31T16:40:57 <fchiulli> G: Can an action perform multiple tasks?
2008-07-31T16:41:05 <G> fchiulli: yeah
2008-07-31T16:41:16 <mmcgrath> fchiulli: an action can be a script so it could do whatever you want.
2008-07-31T16:41:40 <G> http://publictest3.fedoraproject.org/zabbix/events.php for instance shows us that pt2 got rebooted recently
2008-07-31T16:42:03 <mmcgrath> The manual is actually quite good. if you're going to be using zabbix you need to give it a look because for items and triggers you need to know the pretty small zabbix language.
2008-07-31T16:42:14 <G> mmcgrath: yeah
2008-07-31T16:42:15 <mmcgrath> So, for example, {desktop:vm.memory.size[free].last(0)}<10000
2008-07-31T16:42:26 <ricky> +1 to the documentation
2008-07-31T16:42:42 <G> hmmm and pt2 was rebooted recently ):
2008-07-31T16:42:44 <G> *
2008-07-31T16:42:49 <ianweller> that is very interesting.
2008-07-31T16:42:52 <mmcgrath> thats on my zabbix machine at home, its for my desktop machine, its looking at vm free, its using the current value (of last(0)) and is checking that its not less then 10000
2008-07-31T16:43:07 <ricky> So looking at http://publictest3.fedoraproject.org/zabbix/actionconf.php, right now, there the only thing that has notifications now is apache on pt9?
2008-07-31T16:43:25 <G> correct, and that only goes to me atm
2008-07-31T16:43:37 *** tc141516 has quit IRC
2008-07-31T16:43:41 <G> it's not actually properly setup, it was just the first thing I did when I first setup the server
2008-07-31T16:43:46 <fchiulli> mmcgrath: Is the time from the host (pt2) or the Zabbix server?
2008-07-31T16:43:59 *** Standby has joined #fedora-meeting
2008-07-31T16:43:59 <mmcgrath> fchiulli: no idea, G?
2008-07-31T16:44:00 <G> fchiulli: UTC
2008-07-31T16:44:02 <mcgarnicle> I'm just lurking here, but I'd like to point out Zenoss does have "stack" installers that include all deps, so maybe the python version isn't as be of an issue for the future
2008-07-31T16:44:21 <mmcgrath> mcgarnicle: cool, unfortunately for us its not in Fedora so we can't use it
2008-07-31T16:44:21 <ricky> mcgarnicle: We usually like to work with stuff that can be packaged
2008-07-31T16:44:25 <ricky> As in, that's a packaging no-no
2008-07-31T16:44:38 <wakko666> mcgarnicle: zenoss uses rpath, which makes it a huge pain to maintain
2008-07-31T16:44:45 <G> mcgarnicle: yeah, the funny thing is, I tried that and it blew up on me
2008-07-31T16:44:45 * ianweller sees a graphs tab
2008-07-31T16:44:59 <G> ianweller: thats the trending
2008-07-31T16:45:00 <mmcgrath> G: I even got a map setup today, took about 5 minutes for my first time through
2008-07-31T16:45:08 <G> http://publictest3.fedoraproject.org/zabbix/charts.php?groupid=0&hostid=0&graphid=28
2008-07-31T16:45:09 <zodbot> <http://tinyurl.com/62t2ns> (at publictest3.fedoraproject.org)
2008-07-31T16:45:10 <ianweller> G: so nothing like cacti?
2008-07-31T16:45:35 <mmcgrath> ianweller: I wouldn't say nothing like cacti, but very different in a number of key architectural ways.
2008-07-31T16:45:41 <mmcgrath> 1) cacti uses rrd and zabbix does not
2008-07-31T16:45:47 <wakko666> ianweller: the graphs are IMO, better than cacti.
2008-07-31T16:45:48 <ianweller> i noticed that
2008-07-31T16:45:49 <G> imo it's better than cacti, it can graph anything that it monitors
2008-07-31T16:45:55 <mmcgrath> 2) adhoc graphs are much easier.
2008-07-31T16:45:57 <ianweller> can it to snmp?
2008-07-31T16:46:03 <G> yes it can
2008-07-31T16:46:03 <wakko666> ianweller: yup
2008-07-31T16:46:06 <ricky> In terms of "best practice," should actions be as fine-grained as possible?
2008-07-31T16:46:08 <mmcgrath> ianweller: yeah there's actually an snmp section in the manual but I havent' read that.
2008-07-31T16:46:11 <ianweller> rrdtool is a pain in the ass most times >.<
2008-07-31T16:46:27 <wakko666> i've done snmp stuff with zabbix against F5 LTMs and HP ProCurves
2008-07-31T16:46:37 <ricky> G: It might be a good idea to have a set of best practices for configuring this
2008-07-31T16:46:45 <mmcgrath> G: wakko666: one thing I wondered about is when would you use snmp vs just the agent?
2008-07-31T16:46:46 <ricky> Might be useful for csi, even :-)
2008-07-31T16:47:08 <wakko666> mmcgrath: my opinion is to use snmp when installing the agent isn't possible
2008-07-31T16:47:11 <G> ricky: yeah I will be
2008-07-31T16:47:20 <mmcgrath> ahh, ok. so the new switching hardware is not next week, sometime next month, he ordered it recently.
2008-07-31T16:47:21 <G> wakko666: I agree
2008-07-31T16:47:21 <wakko666> i.e appliances and such
2008-07-31T16:47:41 <ricky> Ah, OK
2008-07-31T16:47:42 *** tomspur has joined #fedora-meeting
2008-07-31T16:47:47 <G> anyway, to get this back on track for now
2008-07-31T16:48:06 <G> A) Are we happy to take this to the next level?
2008-07-31T16:48:29 <mmcgrath> A) yes.
2008-07-31T16:48:40 <G> set it up in parrallel with nagios on noc1 monitoring 10 hosts
2008-07-31T16:48:50 <mmcgrath> G: I figure certainly for this initial trial we'll have each person setup their own notifications.
2008-07-31T16:48:50 <G> okay
2008-07-31T16:49:04 <mmcgrath> I'm fine getting multiple pages and things, I'll be curious to see which ones we get first.
2008-07-31T16:49:13 <londo> the configuration is web based only?
2008-07-31T16:49:18 <wakko666> londo: correct
2008-07-31T16:49:19 <G> B) What hosts do we want to monitor
2008-07-31T16:49:20 <ricky> londo: Yes :-(
2008-07-31T16:49:49 <mmcgrath> there's an import export system so I suspect it could be scripted.
2008-07-31T16:49:49 <ricky> G: Curious: For everything listed under "Default Settings," did you configure those manually, or were they there already?
2008-07-31T16:50:01 <mmcgrath> londo: there's also a lot of auto detection auto discovery stuff.
2008-07-31T16:50:01 <wakko666> mmcgrath: correct. the import/export is in xml.
2008-07-31T16:50:06 <G> they were already there
2008-07-31T16:50:13 <ricky> All right, /me searches harder, then :-)
2008-07-31T16:50:31 <mmcgrath> ricky: yeah the setup I did at home had plenty of stuff already
2008-07-31T16:50:44 <G> btw, there is auditing, so we know who changes stuff, and I'll know who to chase with a 2x4 if they stuff it up
2008-07-31T16:50:49 <wakko666> i used the import/export heavily when extending the snmp checks to include hardware-specific MIBs
2008-07-31T16:51:10 <fchiulli> mmcgrath: Is there any plan to put the configs in puppet maybe using import/export?
2008-07-31T16:51:22 <G> I think thats an ugly method tbh
2008-07-31T16:51:25 <mmcgrath> fchiulli: naw, I don't think we really ahve that many configs.
2008-07-31T16:51:43 <mmcgrath> much of what zabbix does by default will be covered, the additional stuff we'll just do by hand and shouldn't be much.
2008-07-31T16:51:45 <wakko666> fchiulli: there's a limit to what is exported. it doesn't export everything, so if that's our restore plan, there are many things that would be lost
2008-07-31T16:51:51 <fchiulli> mmcgrath: But export for use to back it up.
2008-07-31T16:51:52 <ricky> If we test this in prod, it'll be a completely fresh install, correct?
2008-07-31T16:51:56 <G> ANYWAY, back to item B)
2008-07-31T16:52:04 <mmcgrath> fchiulli: ah, actually we'll just make db backups.
2008-07-31T16:52:17 <mmcgrath> G: hosts to monitor. I'd say at least a couple of services.
2008-07-31T16:52:18 <G> What hosts do we want to see monitored?
2008-07-31T16:52:22 <mmcgrath> hosted1, hosted2.
2008-07-31T16:52:25 <mmcgrath> app1-4
2008-07-31T16:52:28 <mmcgrath> bastion
2008-07-31T16:52:29 <ricky> nfs1?
2008-07-31T16:52:29 <G> koji1
2008-07-31T16:52:31 <mmcgrath> proxy1
2008-07-31T16:52:32 <mmcgrath> koji1
2008-07-31T16:52:34 <G> kojipkgs1
2008-07-31T16:52:42 <mmcgrath> yeah, I'd say any or all of those would be fine
2008-07-31T16:52:43 <mmcgrath> ricky: yeah nfs1
2008-07-31T16:52:44 <G> x86-1 and ppc3 maybe?
2008-07-31T16:53:00 <wakko666> do we want to do any website montioring as well as host monitoring?
2008-07-31T16:53:01 <ricky> Oh yeah, a builder would be good
2008-07-31T16:53:02 <mmcgrath> G: actually xenbuilder4 and ppc4. I think they're our busiest.
2008-07-31T16:53:08 <G> mmcgrath: k
2008-07-31T16:53:12 <mmcgrath> wakko666: yeah I'd like to.
2008-07-31T16:53:27 <xaeth> Regarding configs: For most systems and apps wouldnt the goal be to design templates so that there is minimal host specific configurations?
2008-07-31T16:53:42 <G> xaeth: thats how I'm going to lead the project yes
2008-07-31T16:53:45 <wakko666> mmcgrath: k... fp.o, koji, and fedorahosted, does that sound about right?
2008-07-31T16:53:51 <mmcgrath> yeah.
2008-07-31T16:53:53 <xaeth> G: kewl
2008-07-31T16:53:54 <mmcgrath> now here's the tricky part.
2008-07-31T16:53:58 <mmcgrath> G: where are you going to run it?
2008-07-31T16:54:00 <ricky> For web, I'd add fas as well
2008-07-31T16:54:07 <mmcgrath> hard to monitor fp.o from inside phx
2008-07-31T16:54:11 <G> oh well add fas1 too
2008-07-31T16:54:12 <ricky> And bodhi
2008-07-31T16:54:18 <wakko666> ricky: good idea.
2008-07-31T16:54:28 <G> okay, so hold on, make sure I've got my list right
2008-07-31T16:54:33 <ricky> G: app1-4 should be enough, as there are wsgi apps on those.
2008-07-31T16:54:57 <ricky> I mean the FAS/bodhi web interface, as probably goes down the most often ;-)
2008-07-31T16:55:03 <G> hosted1 hosted2 app1 app2 app3 app4 bastion nfs1 koji1 kojipkgs1 xenbuilder4 ppc4 and fas1
2008-07-31T16:55:23 <G> thats 13
2008-07-31T16:55:27 <G> oh actually...
2008-07-31T16:55:37 <G> forget fas1 add say xen7?
2008-07-31T16:56:12 <ricky> Interestingly, the xen servers are the simplest in terms of monitoring/services on them :-)
2008-07-31T16:56:17 <ricky> Could adding a db instead be useful?
2008-07-31T16:56:30 <mmcgrath> G: yeah, and for giggles lets also add the xen servers that each of those apps is on if they're on one.
2008-07-31T16:56:34 <mmcgrath> just so we can get the deps in there.
2008-07-31T16:56:40 <ricky> Oh yeah
2008-07-31T16:56:42 <G> mmcgrath: okay, I'll do that
2008-07-31T16:56:53 *** Standbye has quit IRC
2008-07-31T16:56:57 <xaeth> In my experience (and the manuals) u may see contention and slowness when running db and frontend together on 1 box w/ over 10 hosts
2008-07-31T16:57:01 *** Sparks has left #fedora-meeting
2008-07-31T16:57:02 <G> I think we'll cap it there so not to be monitoring the whole network for now
2008-07-31T16:57:20 <wakko666> G: do you mind if I take on configuring the web checks?
2008-07-31T16:57:22 <G> mmcgrath: for that, I think noc1 and db1 would be the best
2008-07-31T16:57:27 <xaeth> Will run, but u might start seeing gaps in data
2008-07-31T16:57:30 <G> wakko666: yeah
2008-07-31T16:57:56 <mmcgrath> G: I'll make that your call. If you want to go straght to db1 I'm fine with that. I did notice in the docs there's a db monitorer that will let us know when it goes down and zabbix is dead.
2008-07-31T16:58:08 <G> mmcgrath: yeah
2008-07-31T16:58:32 <mmcgrath> k
2008-07-31T16:58:35 <G> mmcgrath: as you said, db1 has plenty of power etc now so I think thats the right call
2008-07-31T16:58:59 <G> okay, D) What are we going to use as the metric of success?
2008-07-31T16:59:06 <mmcgrath> G: I agree, especially now that I know about the db monitorer.
2008-07-31T16:59:36 <mmcgrath> G: good question. I'd say a look at false positives.
2008-07-31T16:59:46 <G> sounds good to me
2008-07-31T16:59:56 <mmcgrath> as well as problems that we're monitoring that we didn't find out about.
2008-07-31T17:00:16 <G> false positives and false negatives
2008-07-31T17:00:31 <G> sounds good to moi
2008-07-31T17:00:42 <wakko666> notification latency might be another good metric. how long does it take for us to receive a notice of a problem.
2008-07-31T17:00:52 *** wfp has quit IRC
2008-07-31T17:01:12 <ricky> Based on the speed of the web interface updates, it's hopefully pretty good
2008-07-31T17:01:17 <G> wakko666: yeah, that'll be reasonable too
2008-07-31T17:01:46 <G> okay, it's past 9am here, so have we just about finished?
2008-07-31T17:02:27 <mmcgrath> G: yeah we should close the meeting now.
2008-07-31T17:02:34 *** hanthana has joined #fedora-meeting
2008-07-31T17:02:42 <G> mmcgrath: my thoughts exactly...
2008-07-31T17:02:44 <mmcgrath> anyone have anything else? we can continue the monitoring discussion in #fedora-admin
2008-07-31T17:02:47 * mmcgrath closes meeting in 30
2008-07-31T17:03:00 <fchiulli> If we find problems/want enhancements, do we know if upstream is receptive?
2008-07-31T17:03:07 <G> fchiulli: they are
2008-07-31T17:03:13 <G> and 1.6 looks pretty good for us too
2008-07-31T17:03:17 <mmcgrath> 10
2008-07-31T17:03:23 <mmcgrath> 5
2008-07-31T17:03:28 *** mmcgrath changes topic to "Infrastructure -- Meeting End"
2008-07-31T17:03:29 *** mcgarnicle has left #fedora-meeting
2008-07-31T17:03:32 <mmcgrath> thanks for coming everyone!
2008-07-31T17:03:38 <G> thank you!
2008-07-31T17:03:40 <SmootherFrOgZ> thanks
2008-07-31T17:03:48 *** mmcgrath changes topic to "Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Communicate/FedoraMeetingChannel for meeting schedule"
Fedora-infrastructure-list mailing list

Thread Tools

All times are GMT. The time now is 01:28 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org