Minutes/Summary for the 2010-06-15 FESCo meeting
#fedora-meeting: FESCO (2010-06-15)
Meeting started by mjg59 at 19:35:11 UTC. The full logs are available at
* init process (mjg59, 19:36:01)
* kylem and nirik unable to attend today (mjg59, 19:36:30)
* #351 Create a policy for updates - status report on implementation
* ACTION: mclasen to look over critpath documentation (mjg59,
* #382 Implementing Stable Release Vision (mjg59, 19:41:12)
* LINK: https://fedoraproject.org/wiki/Stable_release_updates_vision
* ACTION: fesco to continue to update
and aim to produce a comprehensive policy for the board (mjg59,
* #387 (mjg59, 19:57:18)
* #387 compile list of supported CPUs and react to recent loss of
support for Geode LX on F13 (mjg59, 19:57:24)
* AGREED: dsd_ to post a patch to disable 686 assembler optimisations
for glibc for f13 (mjg59, 20:39:30)
* AGREED: more research on the details of building i586 separately to
be carried out before deciding on f14 (mjg59, 20:40:18)
* #394 F14Feature: MeeGo 1.0
https://fedoraproject.org/wiki/Features/MeeGo_1.0 (mjg59, 20:41:18)
* AGREED: Meego 1.0 is an F14 feature (mjg59, 20:46:12)
* #395 F14Feature: Sugar 0.90
https://fedoraproject.org/wiki/Features/Sugar_0.90 (mjg59, 20:46:27)
* AGREED: Sugar 0.90 is an F14 feature (mjg59, 20:47:47)
* #396 F14Feature: systemd -
https://fedoraproject.org/wiki/Features/systemd (mjg59, 20:48:02)
* AGREED: Systemd is a feature for F14 (mjg59, 21:00:57)
* Open Floor (mjg59, 21:04:26)
* FES ticket updates -
* Open Floor (mjg59, 21:07:00)
Meeting ended at 21:08:52 UTC.
* mclasen to look over critpath documentation
* fesco to continue to update
and aim to produce a comprehensive policy for the board
19:35:11 <mjg59> #startmeeting FESCO (2010-06-15)
19:35:11 <zodbot> Meeting started Tue Jun 15 19:35:11 2010 UTC. The chair is mjg59. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:35:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:35:17 <mjg59> #meetingname fesco
19:35:17 <zodbot> The meeting name has been set to 'fesco'
19:35:36 <mjg59> #chair mjg59 mclasen notting SMParrish_mobile ajax pjones cwickert
19:35:36 <zodbot> Current chairs: SMParrish_mobile ajax cwickert mclasen mjg59 notting pjones
19:35:52 <mjg59> We're missing kyle and nirik
19:36:01 <mjg59> #topic init process
19:36:05 <pjones> zodbot uses LC_COLLATE=en_US . Ew.
19:36:13 <mclasen> kyle said he might not make it (still in UK)
19:36:20 <mjg59> Yeah
19:36:22 <mjg59> Ok
19:36:30 <mjg59> #info kylem and nirik unable to attend today
19:36:37 <mjg59> Ok, shall we start?
19:36:47 <mjg59> #topic #351 Create a policy for updates - status report on implementation
19:36:48 <pjones> oh, wait, I'm backwards.
19:36:50 <mjg59> .fesco 351
19:36:51 <zodbot> mjg59: #351 (Create a policy for updates) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/351
19:37:02 <mjg59> Anyone got anything to say on this this week?
19:37:15 <notting> no change on my end from this week. jlaska was doing some critpath documentation
19:37:51 <mjg59> Anyone else?
19:37:57 <mclasen> did lukes bodhi update happen ?
19:38:06 <mjg59> lmacken: Around?
19:38:10 * mclasen should know this, but doesn't...
19:38:17 <jlaska> notting: feel free to make suggestions, I'm missing important critpath information for maintainers -- http://fedoraproject.org/wiki/Critical_Path_Packages
19:39:06 * mclasen will look it over after the meeting too
19:39:32 * cwickert is here...
19:40:02 <jlaska> mclasen: thx
19:40:18 <mjg59> #action mclasen to look over critpath documentation
19:40:33 <mjg59> Anyone else? Or shall we move on?
19:40:57 <mjg59> (going once.. twice...)
19:41:12 <mjg59> #topic #382 Implementing Stable Release Vision
19:41:13 * cwickert thinks that the KDE list of critical path is way to short
19:41:34 <mjg59> cwickert: We've had that conversation - unfortunately it seems difficult to come up with a better one that doesn't cover pretty much the entirity of KDE
19:42:18 <mjg59> And an overly broad critpath would likely serve as a deterrent to maintainers at the moment. If KDE ends up looking bad compared to the rest of the distribution, then with luck that's something they'll pay attention to
19:42:32 <cwickert> mjg59: well, the Xfce critpath covers half of Xfce, the LXDE one too
19:42:44 <pjones> cwickert: sure, but those are both smaller than kde, right?
19:43:07 <cwickert> yes, but the percentage is way higher
19:43:12 <cwickert> anyway, lets move on
19:43:45 <mjg59> Ok, so we have the wiki page at https://fedoraproject.org/wiki/Stable_release_updates_vision_implementation_ideas
19:43:56 <mjg59> We haven't really added much to that
19:44:29 <mjg59> I'm not convinced that IRC is the best way to discuss adding ideas, but a private mailing list also doesn't seem reasonable
19:44:43 <SMParrish_mobile> Think I added something to it
19:44:49 <mjg59> Maybe people could just scribble some more stuff in there and annotate it on the discussion page if necessary?
19:44:57 <mjg59> It'd be nice to have something to bring to the board at some point
19:45:05 <mjg59> (Yeah, I know that I'm guilty here)
19:45:26 <cwickert> sorry for the stupid question, but I was away for the last weeks... what about the "only security updates and bugfixes" rule? are we trying to drive our users away from Fedora?
19:45:56 <ajax> that's a bit of a bait-y question, but.
19:45:58 <cjb> cwickert: you get the minus points for assuming bad faith.
19:45:58 <ajax> here's the thing
19:46:32 <ajax> if releases are continuously updated with new features, then what is to distinguish them from rawhide
19:46:37 <cwickert> I mean, new versions in a release are one thing that makes us different from the rest of the distributions and according to several polls, users want this. IMO this is a completely different issues from bad/no testing and low quality
19:46:53 <pjones> cwickert: you skipped "far too many updated packages"
19:47:15 <cwickert> pjones: "too many" for whom?
19:47:21 <pjones> for users?
19:47:39 <pjones> If I obeyed packagekit I'd be rebooting my desktop machine 3 times per day for all of F13.
19:47:43 <cwickert> if the users want the updates, then it cannot be "too many"
19:48:01 <cwickert> strange, I only once a day if at all.
19:48:05 <mclasen> people for whom software updates are not the central piece of their fedora experience and interest
19:48:15 <mjg59> Wait. Hang on.
19:48:25 <mjg59> "Stable releases should not be used for tracking upstream version closely when this is likely to change the user experience beyond fixing bugs and security issues."
19:48:26 <cwickert> mclasen: updates are an offer, not a duty.
19:48:30 <pjones> (also, this is far off topic at the moment)
19:48:31 <mjg59> This is from the Board's statement
19:48:46 <pjones> mjg59: right. it's not up for discussion.
19:48:48 <mjg59> It's a position we've been asked to implement
19:49:04 <cwickert> mjg59: you have the link to the boards statement?
19:49:08 <mclasen> cwickert: until the next security update pulls them all in...
19:49:09 <notting> i last updated on 06-05. i have 141 updates waiting.
19:49:10 <mjg59> https://fedoraproject.org/wiki/Stable_release_updates_vision
19:49:19 <LinuxCode> cwickert, too many updates that add new features are not a good user experience
19:49:30 <mjg59> We've had this discussion many times now
19:49:36 <mjg59> I don't think there's a great deal we can add to it here
19:49:43 <LinuxCode> specially as it is, imho, more important to make sure our current stuff works
19:49:44 <cwickert> LinuxCode: again, this is your definition of "too many" updates
19:49:51 <mjg59> The Board have taken a position, and we get to implement it
19:50:02 <LinuxCode> cwickert, updates are fine, as long as they dont add too many new features
19:50:04 <cwickert> LinuxCode: making sure stuff works is a completely different issue
19:50:09 <mjg59> If we disagree with that position then we're free to tell them that we disagree, but unless they change their mind then we still get to implement it
19:50:22 <LinuxCode> cwickert, agreed
19:50:25 <LinuxCode> mjg59, yeh
19:50:29 <mjg59> But I don't think that this is the right place to have that conversation
19:50:34 <cwickert> LinuxCode: but the users WANT the new features, that's why they use Fedora and not Ubuntu, SUSE and whatnot
19:50:53 <mclasen> lets not repeat that discussion here and now
19:50:57 <pjones> derail: success!
19:51:02 <notting> mjg59: unless we want to take the position that 'fesco refuses to work on it'. which, as of the last couple of meetings, we are *not* taking that position.
19:51:13 <mjg59> notting: Right
19:51:21 <ajax> cwickert: that's a reasonable position, but i think what we're aiming for there is "if you want that, use rawhide"
19:51:41 <gholms> ...or the branched release, if any
19:52:09 <mjg59> But in terms of the implementation, does anyone want to add anything at this point?
19:52:26 <mjg59> Or shall we just try to be better at sticking ideas in this week, and try to come up with a cohesive plan?
19:53:38 <mjg59> Ok, moving on?
19:54:02 <cwickert> one question and I will stop arguing. what abput parrot and rakudo for example. people want the monthly updates and they want it on a stable release and not in rawhide. what about these kind of updates?
19:54:21 <mjg59> cwickert: One option is to provide separate repositories for well-bounded package subsets
19:54:32 <mjg59> Let people opt-in to more active updates
19:54:45 <cwickert> mjg59: we already have a yum plugin for that
19:54:46 <pjones> cwickert: if there's an overwhelming reason for some specific case, put it on the agenda and we can discuss it as an exception to the normal process?
19:54:49 <mjg59> As long as there's little interaction between them and the rest of the distribution, that should be pretty safe
19:55:02 <pjones> cwickert: that is, write a specific proposal for an exception to the process and put it on the agenda?
19:55:21 <mjg59> And, yeah, if it's the case that basically all users of a package are going to want updates, then maybe it'd be outside the bounds of the proposal
19:55:23 <cwickert> pjones: I have asked this several times, it was discussed and there was no solution.
19:55:40 <mjg59> cwickert: We're still very much at the stage of trying to brainstorm out an implementation
19:55:42 <pjones> but you're asking vague questions, not coming up with a specific proposal
19:55:46 <notting> cwickert: there's an option for updates-features on the wiki page referenced above
19:55:57 <cwickert> notting: sounds fair
19:55:58 <mjg59> We're not trying to make anything impossible at this point
19:56:41 <mjg59> #action fesco to continue to update https://fedoraproject.org/wiki/Stable_release_updates_vision_implementation_ideas and aim to produce a comprehensive policy for the board
19:56:54 <mjg59> Sound ok?
19:56:56 <cjb> n
19:56:58 <cjb> oops
19:57:04 <pjones> mjg59: yeah
19:57:18 <mjg59> #topic #387
19:57:24 <mjg59> #topic #387 compile list of supported CPUs and react to recent loss of support for Geode LX on F13
19:57:27 <mjg59> .fesco 387
19:57:28 <zodbot> mjg59: #387 (compile list of supported CPUs and react to recent loss of support for Geode LX on F13) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/387
19:57:52 <mjg59> So by the sounds of it, this problem is due to the glibc spec file passing more aggressive optimisation arguments to the linker?
19:58:10 <dsd_> not the spec file
19:58:15 <mclasen> but jakub seems to be unwilling to change that
19:58:17 <dsd_> the hand-crafted Makefile system inside of glibc upstream
19:58:19 <pjones> the configure script or somesuch, but yeah
19:58:28 <pjones> dsd_: right
19:58:29 <mjg59> Sorry, right
19:58:35 <mgoetze> assembler, not linker
19:58:40 <dsd_> and its assembler optimizations
19:58:45 <Guest13239> well, the problem is more general, but the symptom is caused by the glibc scripting.
19:59:14 <mjg59> Right, so other than the facts I was entirely correct
19:59:31 <mjg59> Jakub makes a reasonable point that it's entirely plausible that gcc will start doing the same at some point
19:59:45 <pjones> well, not during f13 one would hope
19:59:49 <mjg59> One would hope
19:59:57 <pjones> and if it does, ...
20:00:01 <mjg59> So we can either "fix" the glibc build system
20:00:10 <mjg59> (Which the maintainer is unenthusiastic about, but still)
20:00:18 <mjg59> Or we can change to start building for 586 again
20:00:21 <ajax> notting: what are we winning by chopping off 586?
20:00:35 <notting> actual performance gains, as i recall
20:00:39 <mjg59> We made a conscious decision to go from 586 to 686, so that doesn't seem immediately appealing
20:00:48 <dsd_> i presume that if we fix both F13 and F14+, we are talking about 2 separate fixes. which one are we discussing?
20:00:49 <mclasen> as a f13 hotfix, patching glibc seems reasonable to me; long-term, jakub's argument is right
20:00:57 <mjg59> And I don't think it's worth dropping back to 586 just to support a cpu that's genuinely a 686
20:01:07 <pjones> mclasen: I'm leaning that same way
20:01:36 <mjg59> Yeah, I think we should just make the change in F13
20:01:53 <dsd_> which change?
20:02:02 <cjb> dsd_: are you interested in coming up with a glibc patch we could review? (the glibc makefile change to avoid asm optimizations.)
20:02:05 <mjg59> And I think we should encourage the binutils and gcc maintainers to let us know if optimisation parameters will change the set of supported CPUs
20:02:05 <ajax> so the story for f14 is... secondary arch? i586?
20:02:23 <Guest13239> a decision to not fix this in the long term = a decision to drop support for the Geode LX in Fedora.
20:02:25 <mjg59> ajax: Well, it's also entirely possible that f14 won't change things
20:02:27 <mclasen> dsd_: make glibc not pass -mtune=i686 to as
20:02:32 <dsd_> cjb: yes, its a trivial patch that i can make this afternoon
20:02:41 <ajax> Guest13239: excluded middle, boo.
20:03:33 <Guest13239> it's ok to decide that, just be aware of what you are deciding.
20:03:40 <mjg59> Guest13239: Correct, a decision not to support machines that don't implement 686 nopl is a ecision to drop Geode LX support
20:03:46 <cwickert> I don't think that it is right to do something because upstream will do it "at some point". we have a policy to follow upstream, but this means to honor upstreams decision and not to go ahead. dropping the support for the geode would be a shame as long as the OLPC is our largest deployment
20:04:11 <ajax> mjg59: which is not the same as "build two, one for 586 and one for 686"
20:04:13 <ajax> which is my point
20:04:19 <pjones> cwickert: I'm not sure those numbers will stand up to scrutiny, but I agree with the sentiment.
20:04:22 <Guest13239> In F12 the decision was explicitly made to drop the i586 but to retain the Geode LX. That is the decision that's being considered for revision if no fix is put in place for this bug.
20:04:33 <pjones> Guest13239: yes, we know.
20:04:44 <notting> have we discussed with kylem and davej the possibility of carrying the emulation patch?
20:05:00 <pjones> not really; last week we all pretty much all thought that was a non-started
20:05:00 <mjg59> Guest13239: ...we're explicitly talking about fixing this bug.
20:05:03 <pjones> especially upstream
20:05:44 <mclasen> notting: upstream first, yadda yadda
20:05:56 <notting> we *do* carry the occasional patch, though
20:05:59 <pjones> yeah
20:06:04 <pjones> davej: any thoughts?
20:06:14 <cjb> notting: according to https://bugzilla.redhat.com/show_bug.cgi?id=579838#c15 , this particular patch is very not a good idea.
20:06:16 <notting> (sorry, i have to drop off. will read scrollback)
20:06:21 <mjg59> notting: Thanks!
20:06:45 <davej> I tried to get it upstream a while back, but it turned into a wankfest of "we should generically support missing instructions"
20:07:24 <mjg59> Yeah, upstream has ideas about how they want this done
20:07:33 <davej> and it's a non-trivial amount of work
20:07:34 <mjg59> In theory it could be integrated with the kvm instruction emulation stuff
20:07:48 <ajax> in practice maybe it'd be nice to have something that works instead.
20:07:59 <davej> you'd think that, but people suck
20:08:19 <cjb> note that "works" doesn't really describe the emulation patch, since it's turning a NOP into a gigantic kernel trap.
20:08:39 <mclasen> and just like the kernel people don't like emulation, the toolchain people probably don't like a new IS for geode ?
20:08:52 <cjb> mclasen: it's not geode only.
20:08:52 <ajax> cjb: well you could always fix mmap(PROT_EXEC) to hunt for NOPL and patch...
20:08:59 <ajax> (note: do not do this)
20:09:40 <pjones> ajax: I think I just threw up a little.
20:09:56 <ajax> pjones: there's pepto in the break room
20:10:22 * LinuxCode gets a glass of water for pjones
20:10:38 <ajax> so: do we want to carry this patch (or a similar one) for f14, or do we want to drop geode from f14, or do we want to do something else to support geode in f14.
20:11:00 <pjones> it seems like there's a better glibc fix we're missing in this discussion
20:11:51 <pjones> but maybe there's not.
20:11:58 <ajax> i can't really think of one.
20:12:07 <pjones> I dunno, something that doesn't involve switching the definition of i686 every 10 minutes.
20:12:14 <mjg59> Is it worth lobbying for a "586+cmov" gcc and binutils architecture?
20:12:27 <pjones> mjg59: that's never worked before...
20:12:34 <cjb> pjones: to include an undocumented instruction that isn't in the spec. :/
20:12:40 <dsd_> last week it was pointed out that geode LX does not support all types of cmov
20:12:41 <pjones> cjb: exactly.
20:12:49 <ajax> pjones: i think they've actually been pretty consistent, as long as you consider 686 to mean Pentium Pro
20:12:54 <cjb> dsd_: is that actually an issue?
20:12:58 <dsd_> (another reason to stop pretending that geode LX is i686, in my opinion)
20:13:02 <cjb> (do we emit any of the unsupported types?)
20:13:05 <dsd_> cjb: not sure.. and if it isnt, it might be in 6 months time
20:13:27 <cjb> dsd_: seems separate to me.. I've never seen any problem with that.
20:13:30 <mjg59> Well, in any case. Are we decided on what to do for f13?
20:13:55 <dsd_> cjb: similarly Fedora worked up til F13, and now we have a problem. same could happen again
20:14:31 <LinuxCode> would it not be possible to just freeze glibc in terms of version ?
20:14:31 <cjb> dsd_: (it could, but we don't have any evidence that it will from the last couple of years, so it's hardly likely.)
20:14:39 <ajax> i think we were pretty clearly unanimous for f13: patch glibc.
20:14:42 <LinuxCode> just throwing this in
20:14:44 <cjb> mjg59: sounds like dsd_ comes up with a patch and attaches to the bug
20:14:55 <mclasen> dsd_: that can always happen; best prevention against that is continuous integration
20:14:56 <mjg59> Yeah
20:15:01 <dsd_> i'll make that patch this afternoon
20:15:07 <mjg59> So then, there's a question for the long term
20:15:21 <pjones> LinuxCode: not terribly effective.
20:15:21 <mjg59> We can't guarantee that this won't happen again unless we revert to i586
20:15:25 <LinuxCode> pjones, i see
20:15:32 <LinuxCode> just a wild idea
20:15:55 <ajax> and it sounds like we do not want to revert to 586 for the distro as a whole.
20:15:58 <cjb> mjg59: it sounds like none of us actually want that to happen :)
20:15:58 <mjg59> Right
20:16:18 <pjones> yeah, I think we all agree that's generally undesirable.
20:16:22 <ajax> so i'm just going to throw that whole "secondary arch" thing out there again
20:16:47 <ajax> we haven't had one since ppc fell off the map, really.
20:16:47 <Guest13239> I haven't figured out what's undesirable about 586 support. Tossing whole architectures over the wall to get a <1% speedup never struck me as wise.
20:16:50 <mjg59> So from my point of view, Geode developers get an emulation patch upstream, Geode developers get a geode architecture into gcc and binutils or we do secondary arch for i586
20:17:09 <pjones> ajax: all of 32-bit x86 ? We've got no precedent for subarches as secondaries, and that might be... complicated.
20:17:14 <mclasen> don't we have some way to install IS-variants of libraries, anyway ? like no-mmx variants, etc
20:17:22 * mclasen fuzzy on how that works
20:17:39 <ajax> mclasen: it's not especially clear, period.
20:17:45 <pjones> mclasen: that's a good question - could we not do an i586 glibc?
20:18:05 <mjg59> pjones: Sure, but any other package that passes the same build options may hit the same issue
20:18:09 <pjones> which would be fine unless kernel (or other things) start using the flags that get them nopl
20:18:09 * gholms rings the 20 minute bell
20:18:09 <ajax> pjones: if gcc starts invoking gas with -march...
20:18:27 <pjones> ajax: yeah
20:18:30 <ajax> vote to continue (+1 from me)
20:18:34 <pjones> +1 to continue
20:18:36 <dsd_> continue
20:18:51 <mclasen> continue
20:18:56 <pjones> ajax: I think the gcc issue there might be a "ask them to not do that" and/or "kill that horse when we come to it" sortof thing.
20:18:58 <SMParrish_mobile> Continue
20:19:43 <ajax> that's four (dsd doesn't count, not fesco, sorry dude). but i'll take it as majority since we're short.
20:19:57 <ajax> pjones: they're going to balk at that. every 0.1% on specint...
20:20:08 <pjones> ajax: and anything else in userland gets to make a choice of "don't do that manually" or "also ship a i586..."
20:20:15 <pjones> ajax: I'm still talking about F13 here
20:20:26 <pjones> F14 may need a differently nuanced answer.
20:21:09 <ajax> pjones: fair. we _could_ potentially do an i586 build of glibc for f13; we did it before, after all.
20:21:18 <pjones> I think it's reasonable to ask gcc not to randomly switch stuff like that intra-release.
20:21:40 <ajax> and i think that's actually a point in favor of subarch as secondary arch; yum knows that if you installed i586 it's probably because you meant it and can't do i686
20:21:40 <mclasen> yeah, stable updates, and all that...
20:22:03 <cjb> ok. so would we apply dsd's patch to F13 and F14, pending a better outcome?
20:22:21 <cjb> (I mean, F13 and devel)
20:22:50 <ajax> f13 certainly. not sure devel is necessary yet.
20:22:55 <pjones> ditto ajax
20:23:30 <pjones> so maybe we should be discussing subarch secondary for F14
20:23:33 <LinuxCode> ajax, but would you push the packages into the same repos for i686 ?
20:23:39 <pjones> but let's discuss F13 to completion _before_ we get on that.
20:23:55 <pjones> LinuxCode: we certainly could.
20:24:05 <cwickert> IMHO F13 is most important, but if we want to encourage the OLPC folks to continue developing on Fedora, we need F14 as well
20:24:05 <LinuxCode> might make it more transparent
20:24:07 <ajax> pjones: i think we _ought_ not, but we could.
20:24:32 <LinuxCode> ajax, question is if that technically works
20:24:47 <ajax> alright then, f13 proposal: patch glibc to not emit nopl because geode lx is supported in f13.
20:24:52 <ajax> (+1 from me)
20:24:57 <mjg59> +1
20:25:02 <pjones> +1
20:25:02 <mclasen> +1
20:25:16 <davej> s/glibc/gcc/ no ?
20:25:26 <ajax> davej: no because right now only glibc does it.
20:25:31 <davej> ugh
20:26:01 <SMParrish_mobile> +1
20:26:15 <ajax> +5, majority, motion carries.
20:26:17 <ajax> so! f14.
20:27:06 <pjones> F14 ... um... shit.
20:27:12 <ajax> i don't think 586 belongs in the same repos as 686 anymore. the isos and boot images would conflict.
20:27:36 <ajax> or you'd build them out of 586 and then something would have to figure out if you could do real i686
20:27:40 <ajax> and that's dumb too
20:27:51 <pjones> Is asking olpc to (help) maintain a secondary arch that's essentially "i686 + i586 rebuilds of anything that doesn't work as i686" going to cause them to run screaming to debian?
20:28:00 <pjones> it's probably not all that much work...
20:28:05 <LinuxCode> pjones, hehe
20:28:14 <mjg59> Well, we should probably do a better job of documenting what building a secondary arch involves
20:28:19 <mclasen> they didn't seem enthusiastic last week...
20:28:53 <ajax> pjones: if gcc changes to invoke gas with optimizations, "anything that doesn't work" is "everything"
20:28:53 <mjg59> Bootstrapping from 686 should be trivial - the only real problematic case I can think of is static libraries
20:28:53 <dsd_> pjones: (speaking as OLPC community member) i dont think anyone at OLPC has that experience. also OLPC resources are veeeery tight.
20:28:58 <dsd_> what kind of work is involved?
20:29:05 <pjones> ajax: yes, but it's really simple rebuilds
20:29:09 <mjg59> jwb: You around?
20:29:27 <jwb> mjg59, will be in a bit
20:29:41 <pjones> dsd_: basically rebuilding every package inside an i586 mock builder and then doing a compose.
20:29:44 <mgoetze> if i may make a comment as a debian user :) while i would welcome olpc users i can also share our solution: debian is 486+ but we allow optimized libs in e.g. /lib/i686/cmov/libc.so.6
20:29:48 <mjg59> jwb: Ok - we were just curious as to the amount of effort you'd guess would be involved in running 586 as a secondary
20:29:59 <gholms> Would that proposal be similar to ppc64's being "ppc, with 64-bit exceptions?"
20:30:00 <pjones> dsd_: I think spot + dgilmore have more details that make that a lot easier than I just made it sound.
20:30:13 <mclasen> mgoetze: thats the is-variant-libraries thing I alluded to earlier
20:30:15 <pjones> gholms: unless gcc changes the options it passes to gas, yes
20:30:25 <cjb> (koji-shadow can make it easier, aiui)
20:30:27 <dsd_> i'm quite ignorant about the Fedora infrastructure, but wouldnt this be trivial to do on the official koji side?
20:30:30 <mclasen> mgoetze: doesn't help for forbidden instructions in apps or kernels, though...
20:30:38 <dsd_> just adding 1 more build profile with 1 little change of -march flag
20:30:41 <pjones> mgoetze: the "sunos 2.5" solution, eh?
20:30:56 <mgoetze> mclasen: the question is, how badly do you need these optimizations...
20:31:03 <pjones> mgoetze: think about openoffice
20:31:09 <LinuxCode> dsd_, but somebody still has to glance over things to make sure packages have built
20:31:20 <mjg59> mgoetze: We did some work on this in a previous release and decided it was worth it
20:31:29 <pjones> dsd_: yeah, for i586 you wouldn't need to set up your own builders
20:31:32 <dsd_> LinuxCode: ok, the "glacing over and testing" can certainly be left to OLPC community
20:31:41 <ajax> dsd_: the only issue with doing that (besides storage, which we may not have to burn) is that primary arches are mandatory for build success.
20:31:58 <ajax> so a build failure on i586 would mean no update for any arch
20:32:09 <pjones> ajax: sure, but the number of build failures that are i586 vs i686 only is going to be /fairly/ low.
20:32:16 <dsd_> i would suspect it would be tiny
20:32:16 <LinuxCode> puts a bigger burden on maintainers
20:32:22 <LinuxCode> pjones, yeh
20:32:28 <LinuxCode> minimal
20:32:29 <mjg59> So in the worst case scenario that i686 nopls become common, we're faced with two options - kernel emulation and a secondary arch, right?
20:32:31 <dsd_> we're talking about a pretty safe change, in my opinion
20:32:32 <mclasen> ajax: a x86 variant is not going to be a big issue for build failures, compared to ppc, no ?
20:32:38 <mgoetze> mjg59: i mean would it still be good enough if you only have it in certain important/performance-critical libs
20:32:38 <pjones> it's not going to be like sparc or ppc where helper binaries take a shit with a bus error
20:32:42 <dsd_> and fedora already shipped for a while with -march=i586
20:32:46 <pjones> mclasen: exactly
20:32:48 <ajax> pjones/mclasen: agreed, not likely to cause new failures.
20:32:48 * mgoetze goes back into his debian corner now :)
20:32:49 <mjg59> mgoetze: There's several binaries that matter
20:32:49 <LinuxCode> dsd_, well, if infra has capacity
20:32:54 <cjb> mjg59: I think you'd take a performance nose-dive if you were emulating them *and* they were everywhere.
20:33:04 <jwb> so
20:33:05 <ajax> the storage thing is pretty real though.
20:33:14 <pjones> ajax: yeah.
20:33:16 <jwb> if you mean a full secondary koji instance
20:33:16 <mjg59> cjb: Yeah, so if that's a problem then an arch rebuild would seem to be it
20:33:21 <pjones> Oxf13: hey, do you know anything relevant here?
20:33:34 <cjb> jwb: then you could koji-shadow, right?
20:33:39 <jwb> it's not trivial. there is setup, hardware, shadowing, and then doing the releng aspects all on your own
20:33:44 <pjones> jwb: I think we just need to auto-rebuild them in the normal koji instance.
20:33:47 <LinuxCode> + mirrors
20:33:59 <ajax> cjb: at best, you could fix up every page where you find nopls, as you hit them. so you'd only take the trap the first time. but that's a full disassembler in the kernel...
20:34:01 <jwb> cjb, koji-shadow helps but it is not a perfect solution
20:34:02 <pjones> jwb: they should build fine on x86 builders, after all
20:34:25 <cjb> ajax: nod.
20:34:29 <jwb> pjones, you mean reuse existing koji as a secondary instance?
20:34:36 <pjones> jwb: yeah, basically
20:34:40 <jwb> koji can't do that
20:34:40 <Oxf13> *twitch*
20:34:43 <jwb> builders talk to one hub
20:34:56 <dsd_> are we decided against adding i586 as a primary arch?
20:34:58 <Oxf13> right, we don't have ways of having builders talk to multiple hubs, or having multiple builders on a single machine
20:35:00 <pjones> I don't mean "set up another hub" either.
20:35:08 <jwb> pjones, then you mean primary
20:35:13 <pjones> dsd_: I think that's gone as an assumption so far.
20:35:19 <dsd_> i understand the concerns about storage on the current infra. apart from that, it would be a really good option i think
20:35:28 <kalev> I think a whole lot more easier for all involved parties would be to revert back to i586 as main architecture
20:35:28 <jwb> koji doesn't have the concept of 'primary/secondary' at the hub level
20:35:30 <dsd_> i am admittedly talking from the sidelines though, you guys are the expects
20:35:33 <Oxf13> We could add i586 as another build arch, which would mean another compose arch and....
20:35:44 <mclasen> what was this talk about 'subarch' earlier ?
20:35:59 <mclasen> is that a variant where we don't build everything, but only stuff that needs it ?
20:36:00 <pjones> Oxf13: could we do it as another build arch without doing it as another compose arch? and then let them handle compose as a secondary arch?
20:36:02 <mjg59> mclasen: If we could just provide a subset of packages that were built with 586
20:36:04 <mclasen> do we have that capability ?
20:36:08 <mjg59> mclasen: Except that's potentially "everything"
20:36:18 <mclasen> you can't predict the future...
20:36:30 <mclasen> if it turns into everything, subarch turns into secondary arch...
20:36:31 <Oxf13> pjones: maybe. I'm a bit fuzzy if we can have i586 and i686 show up as two completely separate arches
20:36:43 <pjones> mclasen: no, but it stands to reason that there's a fairly high chance it'll turn to "everything"
20:37:07 <ajax> mclasen: i think "subarch" before was being used in the sense of "rpm's notion of i686 being a superset of i586"
20:37:08 * gholms rings the 15-minute bell
20:37:27 <pjones> ajax: mclasen: yeah, ajax is right about what I meant when I said "subarch as a secondary"
20:37:30 <dgilmore> problem with doing i586 and i686 is that everything will end up getting the i686 rpms in teh buildroot and thinsg taht get statically linked will link against i686 and not i586 objects
20:37:34 <ajax> i think we have things we need to know before we can move on:
20:37:40 <ajax> (ie, decide on f14)
20:37:52 <ajax> - whether we can do 586 and 686 separately within koji as primaries
20:38:02 <ajax> - what running a secondary would really entail
20:38:09 <ajax> - what the storage requirements would be
20:38:23 <mclasen> I concur. we need to do some fact-finding and revisit f14 next week
20:38:26 <ajax> so i move to _not_ continue, and come back in a week with more info
20:38:32 <mjg59> Yeah, that sounds fair
20:38:41 <cjb> sounds right to me too, thanks.
20:38:43 <pjones> ajax: reasonable.
20:38:43 <mjg59> Anyone disagree?
20:38:44 <dsd_> who are we expecting to do this work?
20:38:45 <ajax> (i won't be here next week, but that just makes it not my duty)
20:38:52 <pjones> I won't either.
20:39:04 * gholms reminds mjg59 that the minutes still need an #agreed for f13
20:39:16 <dgilmore> ajax: we cant do them seperatly in a single koji
20:39:26 * mclasen is willing to do some digging
20:39:30 <mjg59> #agreed dsd_ to post a patch to disable 686 assembler optimisations for glibc for f13
20:39:49 <gholms> Thanks
20:39:52 <cjb> mclasen: thanks!
20:40:00 <dsd_> thanks mclasen
20:40:17 <dsd_> (i'm testing the patch now)
20:40:18 <mjg59> #agreed more research on the details of building i586 separately to be carried out before deciding on f14
20:40:26 <mjg59> Everyone ok?
20:40:40 <cjb> Yep, thanks for your time, all.
20:41:01 <mjg59> Ok
20:41:05 <pjones> maybe we should add to ajax's list of things we need to know: - do gcc plan to change that switch by default?
20:41:18 <mjg59> #topic #394 F14Feature: MeeGo 1.0 https://fedoraproject.org/wiki/Features/MeeGo_1.0
20:41:35 * LinuxCode gets big eyed
20:41:49 <cjb> pjones: (Jakub kinda threatened it in the bug -- "gcc should switch to telling as to use i686 optimized code for -march=i686, so all packages will eventually have i686 nops.")
20:41:51 <mjg59> Looks fine to me. Anyone have any concerns?
20:42:10 <pjones> cjb: oh, pooh.
20:42:27 * mclasen is a bit dubious on this kind of feature in general
20:42:38 <mclasen> fedora is an OS, meego is an OS
20:42:47 <mclasen> meego-in-fedora is what ?
20:42:53 <LinuxCode> is this referring to Intels new chips or something ?
20:42:54 <pjones> mclasen: likewise, but at the same time: so what? why stop them?
20:42:55 <mjg59> A trademark violation?
20:43:13 <drago01> a sub OS ;)
20:43:16 * LinuxCode is confused
20:43:18 <mclasen> yeah, don't mean that as stop-energy
20:43:39 <mjg59> It may have to be something like "Fedora 14 implements the Meego software platform" or some such
20:43:42 <mclasen> just some lingering doubt about features that don't necessarily enhance fedora, but replace it
20:43:56 <drago01> LinuxCode: this is a merger of moblin and maemo has nothing to do with "new chips"
20:44:02 <LinuxCode> drago01, I know
20:44:07 <cjb> mclasen: they're talking about the "MeeGo Netbook UX", which is just a UI, I think.
20:44:20 <LinuxCode> I was just confused, why thats there in the first place
20:44:43 <mjg59> Shall we vote?
20:44:45 <ajax> the documentation is broken, at a minimum. one place they say yum install, another they say yum groupinstall.
20:44:53 <pjones> I'm +1 on letting them do their thing in their packages.
20:44:57 <ajax> but the intent is clear
20:44:59 <mclasen> cjb: I'd say thats about as meaningful as stating 'vista is just a UX'
20:45:12 <cwickert> mclasen: if this is not a feature, I wonder how your polkit authentication agent reorganization can be a feature
20:45:20 <LinuxCode> be nice if we would get Hildon into Fedora itself
20:45:22 <ajax> and yeah, have fun i guess. +1.
20:45:44 <mjg59> Yeah, +1
20:45:47 <cwickert> +1
20:45:48 <mclasen> I'm +/- 0 here
20:45:59 <SMParrish_mobile> +1
20:46:05 <pjones> that's 5
20:46:12 <mjg59> #agreed Meego 1.0 is an F14 feature
20:46:27 <mjg59> #topic #395 F14Feature: Sugar 0.90 https://fedoraproject.org/wiki/Features/Sugar_0.90
20:46:31 <mjg59> .fesco 395
20:46:31 <zodbot> mjg59: #395 (F14Feature: Sugar 0.90 https://fedoraproject.org/wiki/Features/Sugar_0.90) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/395
20:46:48 <mjg59> Entertainingly similar
20:46:53 <pjones> indeed.
20:46:59 <ajax> up, enter
20:47:03 <mjg59> +1
20:47:05 <LinuxCode> hehe
20:47:10 <cwickert> ...and one more reason to have Fedora work on the Geode
20:47:13 <pjones> I'm +1 on letting them do their thing in their packages.
20:47:16 <cwickert> +1
20:47:29 <SMParrish_mobile> +1
20:47:32 <mclasen> +1
20:47:41 <ajax> 5, done.
20:47:45 * mclasen votes refreshingly self-inconsistent
20:47:47 <mjg59> #agreed Sugar 0.90 is an F14 feature
20:47:52 <LinuxCode> mclasen, haha
20:48:02 <mjg59> #topic #396 F14Feature: systemd - https://fedoraproject.org/wiki/Features/systemd
20:48:05 <mjg59> .fesco 396
20:48:06 <zodbot> mjg59: #396 (F14Feature: systemd - https://fedoraproject.org/wiki/Features/systemd) - FESCo - Trac - https://fedorahosted.org/fesco/ticket/396
20:48:10 <mclasen> LinuxCode: think of the kids, instead of the telcos
20:48:13 <mclasen> or something
20:48:21 <LinuxCode> mclasen, biased towards kids ?
20:48:23 <LinuxCode> lol
20:48:30 <ajax> boy do i ever not care about init.
20:48:34 <pjones> yeah. always try to aim right at them.
20:48:44 <cwickert> does Lennart really want this systemd to be a feature?? AFAIKS it was Rahul who made the wiki page
20:48:52 * mclasen thinks the feature is a bit premature
20:48:57 <mjg59> Is the aim really to replace upstart in the F14 cycle?
20:49:11 <mclasen> rahul was very enthusiastic and wrote it, lennart hasn't really declared his intentions about f14 yet
20:49:11 <mjg59> (And is anyone here to speak on behalf of this feature today?)
20:49:15 <mezcalero> cwickert: i have made quite a few changes to the feature page yesterday
20:49:23 * mclasen sees mezcalero and kay in the room
20:49:25 <mjg59> mezcalero: Oh, good, you are here
20:49:32 <pjones> if lennart and rahul really want to make systemd installable in parallel with upstart, I think that's okay. If the goal is to replace upstart, I think this is silly. If they don't know the goal yet, ...
20:49:42 <mjg59> mezcalero: Is the intent to replace upstart for F14?
20:49:53 <mezcalero> so, basically my own opinion on this i have explained on the wiki page and basically goes like this:
20:49:54 <cwickert> mezcalero: when we talked on Linuxtag this sounded a bit different, you were not that enthusiastic
20:50:20 <cwickert> (only 5 days before today)
20:50:24 <mezcalero> yes, it is ambitious, but so far things look rosy, and we can at least try
20:50:33 <mclasen> lets hear lennarts opinion, shall we ?
20:50:39 <cwickert> mezcalero: if you want to do it, go for it!
20:50:40 <pjones> mezcalero: try _what_, exactly?
20:50:57 <mezcalero> if fesco agrees on this, then we should set a date where we decide whether we should adopt it by default
20:51:06 <mezcalero> pjones: the feature page says "default"
20:51:09 <mjg59> mezcalero: Well, strictly speaking that's feature freeze
20:51:20 <mezcalero> mjg59: well, yes, but i'd like to see an earlier date for that
20:51:23 <mjg59> Sure
20:51:27 <LinuxCode> quote "we do not recommend users to convert to using systemd services yet for this release" ???
20:51:41 <drago01> LinuxCode: has not been any different for upstart
20:51:41 <mezcalero> so, i would like to keep the options open, but i'd like to see the testing for the case "systemd as default"
20:51:42 <mjg59> LinuxCode: We haven't recommended that people use upstart native scripts yet either
20:51:43 <ajax> LinuxCode: which is what we do with upstart.
20:51:52 <LinuxCode> k
20:51:56 <LinuxCode> kk
20:51:58 <LinuxCode> sorry for noise
20:51:59 <mjg59> mezcalero: The feature owner is always free to decide to stop pushing it part way through the release
20:52:13 <ajax> and i'm happy to say no to anything you like ;)
20:52:33 <pjones> I don't have a problem with you trying to make sure it's ready to _try_ to use as default fairly early
20:52:41 <mezcalero> mjg59: well, yes, but i want to make clear that with this proposal this case is not unlikely
20:52:47 <mjg59> mezcalero: If you're confident that it'll behave itself, I don't have any problem with aiming for it to replace upstart - but I'd like to see the initial transition happen early, so we have good confidence that it works for everyone
20:52:47 <pjones> I think we need to have a longer discussion as to if that's what we're going to do though, even if you do have it ready.
20:53:02 * cwickert just wanted to make sure this feature is supported by the developer. if so, it would really be a great feature and we should support mezcalero
20:53:34 <mjg59> As long as features *work*, I think Fedora is a good place to push them
20:53:37 <LinuxCode> the feature sounds good to me
20:53:38 <mezcalero> i think what be very valuable is to get the testing
20:53:49 <mezcalero> and if then it turns out not to fly for f14
20:53:53 <pjones> mjg59: that being said, that's a compelling argument for making it optional
20:53:54 <mezcalero> then we take a step back
20:53:55 <mclasen> mezcalero: can I boot rawhide with the current packages today ?
20:53:59 <mezcalero> and we are prepared to step back
20:54:07 <pjones> default is a much more in depth discussion that hasn't really been had here.
20:54:11 <mezcalero> mclasen: yes, you should
20:54:12 * drago01 notes -ENOSELINUX (yet)
20:54:15 <mjg59> So I'd lean +1 on the assumption that we'll look very carefully at it part way through the cycle to determine whether we think it's plausible to ship it
20:54:28 <mezcalero> mclasen: i run f13 with minor updates though, not rawhide itself
20:54:34 * mclasen notes a nose in ENOSELINUX
20:54:53 <LinuxCode> I dont think it should be default though
20:54:54 <mezcalero> well, the NOSELINUX this is a prob, but i think not too hard to fix
20:55:06 <mezcalero> i.e. mostly we need dwalsh to get the policy right
20:55:13 <mezcalero> upstart has no selinux support atm at all
20:55:17 <mezcalero> except for a policy
20:55:20 <mjg59> mezcalero: Are you aware of any other cases where it's not at reasonable feature parity?
20:55:27 <mjg59> s/feature/integration/ as appropriate
20:55:48 <mezcalero> i want closer selinux integration in the long run (i.e. that systemd loads the policy, instead of the initrd, but that's not really important now)
20:55:49 <cwickert> dwalsh is very responsive for this kind of changes, this shouldn't prevent us from changing
20:55:52 <mgoetze> i believe russell coker has previously offered to patch "any init system" to support selinux (though perhaps that was in a debian context)
20:56:02 <mezcalero> mjg59: i listed the missing parts on the page
20:56:17 <mjg59> mezcalero: Ok, that's what you'd consider to be the full list?
20:56:21 <mjg59> mezcalero: Just wanted to make sure
20:56:31 <mezcalero> mjg59: yes, i am pretty sure it is complete
20:56:35 <mjg59> Ok
20:56:39 <mjg59> Anyone else have any questions?
20:56:53 <mezcalero> mjg59: oh, and regarding the item "We need sysv compatible implementations for /bin/reboot, /bin/shutdown an" part: we can actually split that off upstart
20:56:54 <ajax> do i get to see before/after numbers for boot speed?
20:56:56 <mezcalero> and use that
20:57:05 <pjones> bootcharts would be good
20:57:08 * mclasen is going to work with mezcalero on getting the feature page fully staffed in terms of roadmap, testing, and fallback
20:57:09 <mezcalero> so the absolute minimal adoption would require only the man pages
20:57:24 <mezcalero> pjones: i actually have plenty of those
20:57:33 <pjones> I'm +1 to letting you go ahead as a feature with the caveat that I'm not necessarily +1 to switching the default.
20:57:41 <LinuxCode> mezcalero, be nice to see those on the page
20:57:46 <pjones> (and I actually lean against it so far)
20:57:55 <mjg59> pjones: I think the default is something that we can revisit partway through the cycle
20:58:05 <mjg59> It's obviously hard for us to make that decision now
20:58:09 <pjones> mjg59: yeah
20:58:16 <mezcalero> pjones: however, to be frank they are not particularly impressive on rotating media without readahead, which is why i haven't published them; and readahead is currently upstartified and i haven't deupstartidized it yet
20:58:54 <mclasen> obviously, we can only make the default-or-not decision at feature freeze time
20:58:56 <mezcalero> on ssd systemd is much nicer though, regardless of readahead
20:59:00 <mjg59> Ok. So for now, should we just vote on it as a feature rather than inherently as a default?
20:59:01 <drago01> well doing everything in parallel is pretty much expected to end in "disk seek madness"
20:59:01 <ajax> +1 to go ahead and see how far this gets.
20:59:05 <cwickert> +1 for systemd and trusting mezcalero
20:59:05 <drago01> on rotating media
20:59:11 <mezcalero> mclasen: well, but it would be nice if rawhide would ship it enabled by default
20:59:12 <drago01> but readahead should help
20:59:17 <mezcalero> mclasen: just to get it tested
20:59:23 <mclasen> mezcalero: sure, if it boots :-)
21:00:02 <wacka> mezcalero: how would you enable it by default, create a /sbin/init symlink or mangle the boot loader config?
21:00:07 <mezcalero> so yepp, again i am aware this is ambitious, but i'd really like to see the testing and i am happy to revert it later if this really doesn't work out
21:00:14 <mclasen> shall we vote on switching rawhide to systemd for some time during f14, separately ?
21:00:16 <mjg59> Ok, I'm +1
21:00:36 <mjg59> Any other votes? We're at +3 so far.
21:00:43 <mclasen> +1 also
21:00:45 <mezcalero> and even if we don't pull it off for f14, the work is not lost after all, and will be good for f15
21:00:45 <SMParrish_mobile> +1
21:00:48 <mjg59> Ok
21:00:57 <mezcalero> wacka: symlink
21:00:57 <mjg59> #agreed Systemd is a feature for F14
21:01:01 <mezcalero> ah, yay!
21:01:05 <mezcalero> yippieh
21:01:07 <pjones> mclasen: I think we need to discuss if switching for the release is a goal before we talk about switching for some time in the middle.
21:01:13 <mjg59> mezcalero: And if you break everything, we break your legs
21:01:34 <mclasen> pjones: the rawhide testing would be valuable, even if it remains optional in f14
21:01:40 <mjg59> Ok, that's the end of the tickets
21:01:44 <mezcalero> mjg59: as long as you don't break my typing fingers i am fine
21:01:45 <mclasen> at least thats how I understand mezcaleros request
21:02:00 <mjg59> I, uh, can't remember how to do the engineering services report. Does anyone have anything on that?
21:02:02 <mezcalero> mclasen: yes, you understood me ;-)
21:02:11 <pjones> mclasen: true, but that same rawhide time is a scarcity I'm not sure is wise to spend on our non-default init.
21:02:38 <mezcalero> one of the main reasons i want to see this as default is btw that i can ask people to ship native service files
21:02:51 <mezcalero> if it is not default convincing people to do that will be much harder
21:03:09 <pjones> I think if that's the _best_ argument for switching init systems, we'd have to be crazy.
21:03:57 <wacka> mezcalero: is the config file format stable enough already or do you expect incompatible changes?
21:04:26 <mjg59> #topic Open Floor
21:04:34 <gholms> FES?
21:04:45 <mezcalero> wacka: at this point i see nothing i'd want to change
21:04:47 <mjg59> Sorry, yes
21:05:07 <mjg59> #topic FES ticket updates - https://fedorahosted.org/fedora-engineering-services/report/6
21:05:17 <mezcalero> wacka: and the format is well-known, and understood as it is just .desktop
21:05:31 <mezcalero> wacka: so it's not exactly something completely new which needs to settle
21:05:44 <wacka> sure, but the keywords you use etc are systemd specific
21:05:52 <mjg59> Anyone want to talk about FES?
21:06:32 <mjg59> Man. And I went to all the trouble of finding that URL and everything
21:06:51 <gholms> mmcgrath isn't here...
21:06:54 <mjg59> Yeah
21:06:57 <mjg59> Oh well
21:06:57 <LinuxCode> indeed
21:07:00 <mjg59> #topic Open Floor
21:07:01 <LinuxCode> hes off
21:07:08 <mjg59> We'll come back to that next week
21:08:36 <mjg59> Well, isn't everyone dull
21:08:37 <mjg59> Ok
21:08:43 <LinuxCode> lol
21:08:46 <gholms> [a cricket chirps in the distance]
21:08:46 <mjg59> Thanks, everyone!
21:08:52 <mjg59> #endmeeting
devel mailing list
|All times are GMT. The time now is 02:33 AM.|
VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.