#fedora-meeting: Infrastructure (2012-08-16)
Meeting started by nirik at 18:00:19 UTC. The full logs are available at
* Greets! (nirik, 18:00:19)
* New folks introductions and Apprentice tasks. (nirik, 18:02:16)
* Applications status / discussion (nirik, 18:07:29)
* comments welcome on websockets/busgateway plan:
* pastebin service going to staging soon. (nirik, 18:12:03)
* Sysadmin status / discussion (nirik, 18:19:04)
* Hopefully will have networking to our private cloud boxes today and
can start setting them up (nirik, 18:19:57)
* new machine at osuosl... should be setup hopefully soon. osuosl02
* should have vfilers in the next few weeks to start looking at.
* moving hosted03->01/02 next tuesday. (nirik, 18:21:11)
* new virthost should be in phx2 now, racked in next few weeks to
replace virthost13 (nirik, 18:24:31)
* we will be making a busgateway01.stg and paste01.stg soon. (nirik,
* Upcoming Tasks/Items (nirik, 18:26:32)
* 2012-08-14 to 2012-08-28 F18 Alpha Freeze (nirik, 18:26:44)
* 2012-08-21 hosted03-> hosted01/02 migration (tenative) (nirik,
* 2012-08-28 F18 Alpha release. (nirik, 18:26:44)
* 2012-08-31 end of 2nd quarter (nirik, 18:26:44)
* 2012-09-01 nag fi-apprentices. (nirik, 18:26:44)
* 2012-09-08 drop inactive fi-apprentices. (nirik, 18:26:45)
* 2012-09-10 to 2012-09-24 Skvidal is out (nirik, 18:26:47)
* 2012-09-18 to 2012-10-02 F18 Beta Freeze (nirik, 18:26:49)
* 2012-09-27 to 2012-09-28 Smooge at Puppetconf (nirik, 18:26:51)
* 2012-10-02 F18 Beta release (nirik, 18:26:55)
* Default Application Frameworks (nirik, 18:28:14)
Meeting ended at 19:00:29 UTC.
Action Items, by person
People Present (lines said)
* nirik (106)
* abadger1999 (48)
* skvidal (22)
* lmacken (19)
* tflink (11)
* jds2001 (11)
* abompard (11)
* threebean (10)
* mdomsch (8)
* smooge (6)
* zodbot (5)
* samkottler (4)
* relrod (4)
* suehle (3)
* athmane (1)
* rossdylan (1)
* ricky (0)
* dgilmore (0)
* CodeBlock (0)
18:00:19 <nirik> #startmeeting Infrastructure (2012-08-16)
18:00:19 <zodbot> Meeting started Thu Aug 16 18:00:19 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:19 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:19 <nirik> #meetingname infrastructure
18:00:19 <nirik> #topic Greets!
18:00:19 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
18:00:19 <zodbot> The meeting name has been set to 'infrastructure'
18:00:19 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
18:00:25 * skvidal is here
18:00:41 * samkottler is present
18:00:56 * athmane is around
18:01:02 * abadger1999 here
18:01:04 * abompard is here
18:01:23 * jds2001 around
18:02:00 <nirik> ok, cool.
18:02:07 <nirik> lets go ahead and dive on in.
18:02:16 <nirik> #topic New folks introductions and Apprentice tasks.
18:02:25 <smooge> here
18:02:31 <nirik> If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so in this part of the meeting. Don't be shy!
18:02:49 <abompard> I'm not sure I've been introduced around here
18:03:09 <nirik> abompard: nope... please do introduce yourself.
18:03:14 <abompard> some of you got an email from spot, but maybe not everybody
18:03:30 <abompard> I'm a new hire in the Fedora Engineering team at Red Hat
18:03:48 <nirik> Welcome abompard.
18:03:50 <abompard> I'll be working on HyperKitty first, the next Mailman 3 archiver
18:03:56 * samkottler claps
18:03:57 <abompard> thanks nirik
18:04:17 <nirik> do let us know if you need anything or are interested in working on other things too...
18:04:23 <abompard> I'm in Paris, it's UTC+2 so it's 8PM here
18:04:35 <abompard> nirik: I will, there's a lot of cool stuff to do
18:04:48 <abompard> nirik: but one thing at a time (well, not one, but not too many)
18:04:49 <nirik> yeah, we aren't lacking for stuff to work on.
18:05:16 <nirik> excellent. Look forward to working with you...
18:05:26 <abompard> I hope I'll be able to attend the Thursday meetings
18:05:34 <nirik> any other new folks? or general apprentice queries?
18:05:42 * relrod is here.
18:05:51 * rossdylan is here
18:05:57 <nirik> abompard: if you can't and you have something to bring up, feel free to send it to the list.
18:06:07 <abompard> nirik: got it
18:06:12 * mdomsch
18:06:20 * lmacken
18:06:25 <nirik> cool. welcome everyone, good turnout today.
18:06:28 <samkottler> I spoke to a few different people about our pupet setup so I'll be doing more work on that in the coming weeks
18:06:29 * threebean
18:06:37 <samkottler> No query, though
18:07:16 <nirik> samkottler: great. Thanks for updating our puppet docs too
18:07:27 <nirik> ok, moving along...
18:07:29 <nirik> #topic Applications status / discussion
18:07:39 <nirik> any application news this week or upcoming?
18:07:49 <nirik> we are in freeze, so that means not much deployment...
18:08:24 <threebean> abadger1999, fchiulli, and I are working on migrating some pieces of pkgdb into the tagger and packages apps. Still in development now, but we hope to have some chunks done by the end of the freeze.
18:08:48 <nirik> I'm going to try and help track down our pkgdb hotfixes too.
18:08:57 <nirik> so we can make sure all those are merged.
18:08:58 <abadger1999> <nod> bugz seemed almost implemented in packages already so that should be done and ready to deploy after freeze.
18:09:28 <abadger1999> other stuff may require more work as it seems packages doesn't keep all that data yet.
18:09:35 <nirik> if anyone wants to help with that, I'd be happy to divide and conquer.
18:10:42 <nirik> I updated our websockets/busgateway ticket.
18:10:50 <nirik> with a plan... if anyone has comments, please let me know soon.
18:11:03 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13
18:11:35 <nirik> also, in application news, looks like our pastebin service is ready to move to staging...
18:11:53 <nirik> #info comments welcome on websockets/busgateway plan: https://fedorahosted.org/fedora-infrastructure/ticket/3373#comment:13
18:12:03 <nirik> #info pastebin service going to staging soon.
18:12:29 <nirik> any other apps news?
18:12:41 <threebean> Anyone mind if I start monkeying with koji in stg?
18:12:54 <lmacken> new bodhi went out, last week, and I pushed out some additional bugfixes earlier this week
18:13:16 <nirik> lmacken: did those tickets get closed? we should do that if not...
18:13:27 <lmacken> fedora-packages indexer got wedged again as well.. not sure what is going on there, but I kicked it in the mean time.
18:13:33 <nirik> threebean: go for it. It has a db load in it... but I don't think it has builders or anything.
18:13:44 <lmacken> nirik: ah, I'll close them out, thanks
18:14:17 <nirik> threebean: but if you need a builder, perhaps skvidal can whip you up one on pretty short notice.
18:14:28 <skvidal> yah
18:14:29 <threebean> nirik: Cool. I'm in the dark as to how it actually works. Going to try out some fedmsg stuff.. I might end up needing a builder to test it all.
18:14:30 <skvidal> I can do things
18:14:40 <threebean> skvidal:
18:14:46 <threebean> skvidal: I will bother you, then.
18:14:54 <skvidal> threebean: as long as its easy.
18:14:57 <nirik> it would be nice for it to have a builder and also build from pkgs01.stg git, etc.
18:15:11 <nirik> because then we could actually test mock updates and such
18:15:33 <jds2001> yeah, i'd like for the vendor in the koji config to be different
18:15:45 <jds2001> such that you *know* that it came from staging without a dobut.
18:15:51 <nirik> yeah, that would be good too.
18:16:16 <nirik> and... (wait for the ponies and rainbows)...
18:16:42 <nirik> it would be cool if it ran koji-shadow and shadowed the main koji. Then we could see if new mock breaks something much more naturally
18:17:05 * jds2001 wonders where you plan on finding hte disk for that
18:17:32 <nirik> dunno. it doesn't need to follow all tags... rawhide can't be that big can it?
18:17:56 <nirik> anyhow, those are all dreams. If someone wants to make them real I will buy them a drink.
18:18:18 <jds2001> it'd take a lot more than drinks, i think
18:18:23 <nirik> ok, any other app news or shall we move on?
18:18:31 <relrod> FAS is in string freeze now
18:18:40 <nirik> relrod: cool.
18:18:52 <relrod> I'm waiting to hear back on one ticket then I'll email trans@ and they can get to work
18:19:00 <nirik> sounds good.
18:19:04 <nirik> #topic Sysadmin status / discussion
18:19:21 <nirik> so, we are in freeze so that slows down activity (hurray!
18:19:36 <nirik> however there's various bits of news:
18:19:57 <nirik> #info Hopefully will have networking to our private cloud boxes today and can start setting them up
18:20:16 <nirik> #info new machine at osuosl... should be setup hopefully soon. osuosl02
18:20:48 <nirik> #info should have vfilers in the next few weeks to start looking at.
18:21:11 <nirik> #info moving hosted03->01/02 next tuesday.
18:21:35 <nirik> Anything else pending or upcoming?
18:22:09 <nirik> skvidal wrote up a wiki page on our ansible pokings so far: https://fedoraproject.org/wiki/User:Skvidal/Ansible
18:22:46 <abadger1999> skvidal: Do we need simplejson? We have python-2.6 on all boxes except app07.
18:22:51 <skvidal> abadger1999: we don't
18:22:54 <abadger1999> Cool.
18:22:56 <skvidal> abadger1999: it's just *json(
18:23:01 <skvidal> but rhel5 needs it iirc
18:23:06 <abadger1999> Yeah. silly rhel5
18:23:16 <skvidal> I wanted to be reasonably exact
18:23:20 <skvidal> so no one nitpicked
18:23:24 <abadger1999> <nod>
18:23:31 <skvidal> pretty much @core gets it done
18:24:02 <skvidal> I'm working on breaking the repo I use for builders now
18:24:09 <skvidal> up into private and public
18:24:11 <skvidal> so I can export ours
18:24:15 <nirik> oh, another one:
18:24:31 <nirik> #info new virthost should be in phx2 now, racked in next few weeks to replace virthost13
18:24:38 <jds2001> skvidal: huh?
18:24:47 <skvidal> jds2001: our git repo for builder config
18:24:55 <jds2001> oh oh
18:25:00 <skvidal> jds2001: where I'm storing ansible playbooks and config files, etc
18:25:05 <jds2001> i thought you meant yum repo
18:25:06 <nirik> #info we will be making a busgateway01.stg and paste01.stg soon.
18:25:57 <nirik> ok, any other sysadmin type things to note?
18:26:32 <nirik> #topic Upcoming Tasks/Items
18:26:42 <nirik> here's the list I have of upcoming:
18:26:44 <nirik> #info 2012-08-14 to 2012-08-28 F18 Alpha Freeze
18:26:44 <nirik> #info 2012-08-21 hosted03-> hosted01/02 migration (tenative)
18:26:44 <nirik> #info 2012-08-28 F18 Alpha release.
18:26:44 <nirik> #info 2012-08-31 end of 2nd quarter
18:26:44 <nirik> #info 2012-09-01 nag fi-apprentices.
18:26:45 <nirik> #info 2012-09-08 drop inactive fi-apprentices.
18:26:47 <nirik> #info 2012-09-10 to 2012-09-24 Skvidal is out
18:26:49 <nirik> #info 2012-09-18 to 2012-10-02 F18 Beta Freeze
18:26:51 <nirik> #info 2012-09-27 to 2012-09-28 Smooge at Puppetconf
18:26:55 <nirik> #info 2012-10-02 F18 Beta release
18:26:57 <nirik> anyone have items they would like to add/note/remove/edit?
18:27:22 <smooge> May want ot make that the 26th so I can travel to SF
18:27:31 <nirik> oh yeah.
18:27:49 * nirik edits.
18:28:00 <nirik> ok, anything else on schedules?
18:28:14 <nirik> #topic Default Application Frameworks
18:28:24 <nirik> we didn't reach any conclusions on this the other week...
18:29:00 <nirik> anyone care to propose something? or ?
18:29:56 <tflink> I have a potentially related question about flask packages in epel6
18:30:14 <nirik> I guess I tried to propose: flask, TG2, pyramid. But abadger1999 didn't like that due to too many, and lmacken didn't like that due to too few?
18:30:22 <tflink> assuming that we do end up using Flask more, is there a version that we're trying to target?
18:30:29 <nirik> tflink: ask away... relrod and ianweller are leading that effort.
18:30:34 <abadger1999> yeah -- really.. I'd like 1 framework.
18:30:41 <tflink> I ask because there are a bunch of clask extension packages in epel6 that can't play nice together
18:30:46 <tflink> s/clask/Flask
18:30:56 <lmacken> I think instead of just trying to bless an arbitrary number of frameworks, I think we should take a step back and define the critera/policy of frameworks that we wish to allow. Maybe only 1 meets them, maybe 5?
18:30:59 <abadger1999> proposing 2 was already a compromise to what I was hoping for.
18:31:21 <tflink> it looks like the packaging effort for the extensions was started and not quite finished
18:31:44 <abadger1999> lmacken: That might be logical to think about if our concern is new development... my concern is maintainance, though.
18:32:01 <abadger1999> I think from that standpoint, the answer is: pick one and stick with that.
18:32:51 <lmacken> if maintenance of our existing apps is the main concern, then rewriting all of them completely in something that we've never had experience deploying may not be the best choice.
18:32:57 <jds2001> abadger1999: but the answer could be that we have people able/willing to maintain 5
18:33:09 <nirik> things I want: fewer parts/deps/changes, active upstream that answers bugs/security issues, not too rapid updates but updates, more developers that understand it so can fix things/track down bugs
18:33:14 <abadger1999> lmacken: then let's settle on TG2 and port everything to that.
18:33:26 <abadger1999> jds2001: right now -- but that has many problems
18:34:15 <abadger1999> Person A maintains Foo written in framework1; person B maintains Bar, Baz, and five other apps written in framework1.
18:34:25 <relrod> tflink: I don't know much about the extensions yet, I've just been focusing on getting Flask itself in EPEL for now
18:34:32 <abadger1999> Person A has done the heavy lifting of packaging, upstream interfacing etc... then they go away.
18:34:43 <lmacken> abadger1999: I'm not against porting/rewriting FAS & friends in TG2. But I'm also not against someone wanting to do it with pyramid, flask, etc.
18:34:44 <abadger1999> Person B doesn't have time to maintain the framework...
18:34:57 <tflink> relrod: I have been working on updating the specs and building locally for now, I was planning to pester the maintainer soon for an update
18:35:11 <nirik> perhaps we need to more strongly cross train people?
18:35:58 <nirik> ie, personA: you maintain bar this week and personB: you maintain foo this week?
18:35:59 <abadger1999> * New user joins infra and starts work on Baz written to framework 1. Makes changes that mean porting app Foo written to framework 2.... they have to learn both frameworks to get their change in.
18:36:50 <nirik> abadger1999: there's still some issue even if everything is the same framework...
18:36:58 <nirik> perhaps less so tho
18:37:10 <abadger1999> nirik: of course... but we partially mitigate them.
18:37:31 <jds2001> person A writes app Foo and goes away, Foo is now dead
18:37:32 <abadger1999> so we're in a better place even if we're not in a perfect place.
18:37:41 <jds2001> doesnt much matter what framework it's in
18:37:42 <tflink> I've heard the argument that a lot of the learning has to do with the application on top of the framework - learning a framework is easy. learning all the project-specific quirks and oddities is the hard part
18:38:15 <tflink> but that's developer learning curve, not maintaining tons of different frameworks
18:38:34 <abadger1999> jds2001: That can happen too. But let's say some app were to be abandoned that is critical to Fedora (packagedb, mirrormanager, bodhi, koji... one of those).
18:38:39 <smooge> my take on it more is from the non-developer sysadmin.
18:39:12 <abadger1999> jds2001: is it easier for someone else who works on one of those other apps to step in and start working if the app they currently maintain runs on that framework? Or if it's totally foreign?
18:39:47 <abadger1999> My experience contributing patches to any of those TG apps vs staring at koji code is that a common framework is huge boost to getting into the code base.
18:40:12 <nirik> is koji even using a framework?
18:40:21 <lmacken> nope
18:40:24 <abadger1999> is raw wsgi a framework? ;-)
18:40:34 <nirik> yeah, that.
18:41:08 <nirik> if we said: all applications must use flask. Would that mean someone would have to re-write koji in it? or askbot? or mediawiki?
18:41:19 <abadger1999> I could have said something like ask.fedoraproject.org (django).. same thing applies.
18:41:39 <abadger1999> nirik: those three we currently treat as third party apps
18:41:48 <abadger1999> we run them but we realy don't develop them.
18:41:49 <nirik> right.
18:41:53 <abadger1999> nirik: also --
18:42:21 <abadger1999> this applies most immediately to new app development.
18:42:29 <nirik> anyhow, are we going to reach a consensus here? or perhaps we just settle on the 3 and leave it at that until/unless someone makes a good argument for removing one or adding a new one?
18:42:31 <abadger1999> new apps should be written in our chosen framework(s)
18:42:35 <nirik> right
18:42:39 <abadger1999> old apps get ported as we have the necessity.
18:42:45 <abadger1999> TG1 is becoming a necessity.
18:42:58 <abadger1999> But TG2 , pyramid.... not so much.
18:43:12 <lmacken> if I could leave bodhi as TG1 for 5 more years, I totally would.
18:43:30 <nirik> but 2.0 needs newer framework? or ?
18:43:33 <abadger1999> lmacken: yep.... but hte TG1 stack becomes harder to support every year.
18:43:35 <lmacken> there's something to be said about a framework that we've had in production for 5 years already, with very few problems at all.
18:43:45 <abadger1999> yeah tg2 has different underpinnings.
18:43:56 <lmacken> nirik: yeah, I designed myself into a corner with the architecture of bodhi1, it wasn't TG1's fault
18:44:17 <nirik> lmacken: live and learn.
I bet that code looks fun to you now...
18:44:29 <lmacken> yeah, it's pretty horrid
18:45:14 <abadger1999> If only TG had gone with evolutionary development from tg1.0 instead of rearchitechting the stack for tg2... I think we'd all still be happily running tg1 :-)
18:45:16 <lmacken> but I think we're going to keep going through that live & learn cycle with other frameworks, over and over. We just need to keep moving forward without making the same mistakes, which will be hard if we deviate too far from where we've been
18:45:36 <abadger1999> but htey didn't... and tg1 is much less healthy upstream than the other choices.
18:46:48 <smooge> ok quick question..
18:47:16 <lmacken> if we're looking to make strong recommendations as to what frameworks we wish to use in the future, again, I think we need to decide upon critera that they must meet
18:47:55 <smooge> if lmacken or abadger won the lottery tomorrow and we didn't have them.. how tenable are our apps currently?
18:48:06 <lmacken> like, a) active upstream b) active fedora maintainers ...
18:48:12 <threebean> smooge: I've got 'em.
18:48:13 <lmacken> smooge: I came close last night
18:48:20 <nirik> a) active upstream b) active epel maintainer(s) c) something we already use d) your need here
18:49:12 <threebean> smooge: honestly, pkgdb and mirrormanager appear to be the weakest links from my PoV. The rest could be spread around.
18:49:24 <abadger1999> smooge: We wouldn't die for Fedora 18 but we'd be starting to hurt by fedora 19 release.
18:50:23 <smooge> ok
18:50:29 <abadger1999> smooge: hotfixes and new releases with those hotfixes are the two places you'd notice it first.
18:50:44 * nirik notes as tflink pointed out there are still things we need to address for flask. Likely we won't know all those until we have something deployed using it.
18:51:01 <abadger1999> I generate a lot of hotfixes for fas, python-fedora, and pkgdb. And as time permits... I even wrap those up into releases :-)
18:51:14 <tflink> nirik: sounds like I might be the guinea pig there
18:51:16 <threebean> nirik: the exact same applies for pyramid.
18:51:38 <lmacken> threebean: well, I have experience with pyramid in our env now with openhw2012
18:51:40 <nirik> yeah, true... and we are still on the early part of the curve for TG2
18:51:52 * mdomsch needs to get MM 1.4 out the door, but I've been saying that for 18 months...
18:52:02 <lmacken> threebean: and now we know that the PrefixMiddleware is crucial
18:52:02 <abadger1999> the same could apply for any framework we have... but we don't want to deploy one app in each of them to see how we like it :-)
18:52:14 <nirik> mdomsch: is that just a new bugfix release? or rewritten in tg2?
18:52:23 <nirik> abadger1999: ageed.
18:52:26 <nirik> agreed even
18:52:29 <mdomsch> nirik: bugfix/features, not rewritten
18:52:35 <tflink> abadger1999: don't we already pretty much have one in each already?
18:53:08 <lmacken> we're pretty close. I think all we need is web2py in the mix and we'll have them all :P
18:53:10 <abadger1999> tflink: not that I know of -- we have tg1 and tg2 in production.
18:53:24 <abadger1999> tflink: we have pyramid and flask being worked on (pre-stg still I think)
18:53:34 <tflink> ah, I wasn't thinking of production. just the writing of the apps
18:53:37 <abadger1999> We have django third party apps deployed.
18:53:42 <nirik> I guess from the sysadmin side this would be similar to oses... and there we are pretty much standardized. Think of "I'd like to deploy $thing, but it uses SuSE"
18:54:36 <abadger1999> But I think if we wrote our own django apps we'd have a better understanding of rhe framework and that would likely lead to changes in how we deployed them.
18:54:47 <skvidal> okay
18:54:56 <skvidal> I've been kinda quiet for a bit - I wanted to interject
18:55:01 * nirik sees time running low.
18:55:04 <nirik> by all means skvidal
18:55:11 <abadger1999> So yeah -- there are things we won't know until for real deployment -- but we don't want to really deploy to find them out.
18:55:11 <skvidal> should we stop thinking of the model of how we are currently deploying apps
18:56:09 <skvidal> and start thinking about modifying our deployments to make it less of an effort to deploy one-off stacks
18:56:42 <mdomsch> skvidal: the tricky integration is with FAS
18:56:47 <nirik> but this isn't really about one-off is it?
18:57:21 <mdomsch> at least for MM, that's the one bit outside of the framework that abadger1999 has had to make functional
18:57:34 <mdomsch> which begs the question of the future of FAS as an integration point
18:57:39 <abadger1999> yeah -- it's not really about one-off. It's about maintainance more so. I think one-off stacks would make long term maintanance even harder?
18:57:39 <skvidal> mdomsch: you're right - that is the tricky bit
18:58:03 <mdomsch> I'd be OK with one-off stacks if there were a) fewer integrations; b) more hands to help
18:58:14 <skvidal> mdomsch: +1
18:58:48 <abadger1999> since one-off stacks could be used for pegging versions.. but that means needing to maintain more overall combinations of things with those pegged versions.
18:58:52 <mdomsch> if somehow we went from python-fedora FAS integration to "some common account API" world, be it FAS of LDAP or Shibboleth or AD or ...
18:59:48 <zodbot> suehle: Error: Can't start another meeting, one is in progress.
18:59:51 <suehle> doh
19:00:02 <nirik> suehle: sorry if we are intruding on your time...
19:00:02 * suehle should learn to read topic, sorry
19:00:08 <nirik> shall we wrap up folks?
19:00:12 <abadger1999> Sure.
19:00:16 <nirik> continue in #fedora-apps?
19:00:17 * threebean nods
19:00:22 <suehle> thanks!
19:00:26 <nirik> Thanks for coming everyone!
19:00:29 <nirik> #endmeeting
infrastructure mailing list