FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Kernel Team

 
 
LinkBack Thread Tools
 
Old 10-10-2012, 08:43 PM
David Cantrell
 
Default Review of Fedora 18 Release Criteria

On Wed, Oct 10, 2012 at 01:18:37PM -0700, Adam Williamson wrote:
> On Wed, 2012-10-10 at 12:17 -0400, David Cantrell wrote:
>
> > You're not understanding what I was pointing out. The blocker criteria
> > between alpha and beta should be more open than the blocker criteria between
> > beta and rc. The idea is that we start accepting fewer bugs as blockers as
> > we get closer to RC. Every problem encountered can be evaluated along the
> > lines of:
> >
> > 1) Who is impacted?
> > 2) Is there a workaround?
> > 3) Is the workaround documented?
> > 4) Is the problem in the standard install path?
> >
> > And so on. I'm not saying we should compromise on release quality or
> > anything like that, but just start to ask more and more questions when
> > proposed blockers show up late. Is it really a blocker or not.
>
> > Again, that's not what I'm saying we should do. But if there's a traceback
> > that appears when someone is doing a LUKS installation to USB attached
> > storage that also has a pre-existing NTFS volume for "music"....that doesn't
> > sound like a really suport important use case late in the cycle. Maybe we
> > should consider documenting that as a limitation with or without a
> > workaround for that release and slating it for the next release.
>
> I have a better understanding of where you're coming from now, and
> particularly that you're basing this on a RHEL model (I didn't know this
> was how RHEL did blocker tracking).

That really shouldn't come as a surprise. RHEL drives our work.

> But I still disagree.
>
> I don't think my disagreement is necessarily a problem for you,
> though...let me see if I can put it a different way:
>
> Take the example you just gave. My position isn't 'we should take that
> bug as a blocker both four weeks out and one week out'. My position is
> 'we shouldn't take it as a blocker four weeks out _or_ one week out'.
>
> I'd phrase the problem more as 'there is a tendency in blocker review to
> accept things too casually early on'. We should only accept things as
> blockers that we really believe we should hold the release up for - even
> if it's six weeks until release and we know the bug is going to be fixed
> in two days, we should not accept it as a blocker unless it passes the
> 'if today was go/no-go and this was the last bug, would we block the
> release' smell test.
>
> Rather than throw almost everything on the blocker list when it's early
> and get more rigorous later on, I'd rather we just be rigorous the whole
> time. The blocker list isn't a work list (using the blocker list as a
> 'todo list' / reminder list is a tendency I've noticed in both RHEL and
> Fedora, and something I think we should fight), it is a list of bugs
> that must be fixed for the release to go out. No more, no less. If we
> wouldn't really block the release for a bug, it doesn't belong on the
> list at any point in time. I think we've been making a good effort to
> improve this in recent releases; it's not just an aspiration.

That's fine, but how is that not a to do list?

> So if there are concrete cases where you think bugs have been accepted
> as blockers that you wouldn't be comfortable taking as blockers the day
> before release, that's _always a problem_, and please highlight those
> bugs to us. I don't think it's 'not a problem' if we accepted them as
> blockers a long time before release and they got fixed before any chance
> of causing a slip, it's still a problem. We should stick firmly to our
> tight, minimal definition of what constitutes a blocker, regardless of
> the point in the release schedule.

The big difference here and what we see on the RHEL side is that RHEL allows
for blocker bugs to be re-evaluated and booted if a later workaround or some
other acceptable solution besides a bug fix is found. It becomes a schedule
thing, which I am routinely told is very important for Fedora.

If something marked as a blocker early on is still not fixed through several
test releases, does it really matter anymore for the release? If not, why
not boot it off the blocker list?

> > We do want to make a good release and keep the users happy, but we need to
> > also remember that developers are people too and not just meat grinders for
> > bug reports.
>
> Sure, hence the need to keep the criteria realistic.
>
> > 1) Adjustments to the blocker acceptance criteria that I explained.
> > 2) Acknowledgement that concurrent releases are always in play for the
> > installer team. Generally two RHEL releases and one Fedora release.
> > RHEL gets priority.
>
> I would prefer to phrase 2) the way I already did: Fedora should ensure
> its release requirements are realistic in the context of the resources
> available. If the 'resources available' are to an extent defined by
> other commitments on the part of the anaconda team, then so be it, but
> I'd rather consider it in that context, rather than haul RHEL
> specifically into a Fedora context.

Call it want you want, but we all know it's RHEL.

> > What I'm asking for is blocker acceptance criteria changes that are more
> > realistic given remaining time and other commitments. Or dump the
> > time/schedule requirement and just say we'll call it done when it's done.
>
> The schedule / blocker process balance is a bit of a delicate one, yeah.
> In practice we start getting a lot of pressure if the blocker process
> results in delays of several weeks. I think I'd like to look at this
> way, though: any time that following the release validation process
> properly results in a long slip, _that means something is wrong_. It
> usually means either that the release criteria are excessive, or it
> means that something went wrong in the feature process. I believe the
> feature process as it stands is not adequate in ensuring proposed
> features are really viable on the Fedora release schedule, and FESCo has
> also not always been assertive enough in dropping or delaying features
> that are clearly not meeting the schedule; I think sometimes problems
> are due to that, rather than to over-ambitious release requirements.

I'll agree with that. I've had that complaint for a long time, ever since
we merged Core and Extras.

As we have both already stated and agreed on, the installer is unique. Our
development process is so very much tied to the Fedora release schedule.
There is no easy way for us to develop something large like a new installer
and not have it tied to a release. This became harder for us when we
stopped doing rawhide nightly composes many years ago (and I have no idea if
those are happening again these days). People don't go and download a new
installer to try it out. You download a release of something and try
installing it. Our only way to get that to users is to follow the Fedora
release schedule, unless we want to start making our own distribution.

I know what you're getting at and it's that newui was a huge feature and
that's made our lives hard. Unfortunately, the Fedora release schedule is
such that work like this is always painful for us.

> > > To give a competing example, in anaconda 18.13 there is a bug in the new
> > > partitioning process - I call it 'guided partitioning', the dialog which
> > > attempts to help you free up space on a full disk, by deleting or
> > > shrinking partitions - which causes it to crash when trying to delete
> > > partitions. But if you go into the 'custom partitioning' interface you
> > > can successfully delete partitions. So by your principle, we would not
> > > take that bug as a release blocker. I don't think that would be a good
> > > decision: we should not release an installer which crashes when you try
> > > to follow the path you're guided to, for freeing up space to install the
> > > operating system. 'Don't do what the installer recommends you to do,
> > > instead go into this advanced process that's supposed to be for experts'
> > > is a workaround, and hence satisfies your requirement for not-a-blocker,
> > > but I really don't think it's a good story to tell people in the case of
> > > such a critical bit of functionality.
> >
> > Crashes through the standard install path would of course be a blocker. Why
> > would you think I would want otherwise? You're reading my proposal way too
> > literally.
>
> I'm afraid I have a tendency to read things literally, because processes
> and policies are written down, and if the person who wrote them is eaten
> by a raptor the only way you can reliably read them is 'literally' It
> makes drafting things a pain in the ass, but I think it makes what you
> wind up with a stronger process/policy. Your proposal didn't have escape
> clauses or exceptions, it simply read "Installer blockers should only be
> granted when there is no other way to accomplish the same task during
> installation." That's a very clear and unequivocal statement. We could
> not adopt it into the blocker evaluation process as it's written.
>
> As I said in my initial mail, I think in practice we mostly already do
> what you really want here, but we do need to document it more clearly.
> As the documentation stands, it really doesn't cover the question of
> workarounds much at all.
>
> (in reference to my points about development planning)
>
> > This is nothing new to us, we've been through it many times before.
> > But it doesn't mean we enjoy it. Unless we want to put anaconda in a
> > permanent maintenance mode, it means we will always be playing this
> > game.
>
> I don't think this is entirely correct. I don't want to get into
> specifics because I'm not plugged into the development planning process
> enough to make accurate criticisms, and I don't want to teach anyone's
> grandma to suck eggs, but to talk generally:
>
> Of course any project which involves major development will be more
> susceptible to bugs than any project which is solely in maintenance
> mode. But the _magnitude_ of the effect is certainly something that can
> be addressed by good planning. I don't want to harp on this too much as
> I'm sure it's something you're aware of, but we can certainly aim to be
> realistic and rigorous in our planning and timing of major new features.
> For instance, trying to ensure major developments are feature complete
> at Alpha time, so we're not still writing stuff during the Beta cycle.
> FESCo requiring much more detailed planning for feature submissions and
> being more decisive about rejecting/delaying features that are clearly
> lagging at Alpha stage. Stuff like that.

See above.

> > I didn't read this paragraph. This is a lot of text, but I've skimmed it.
> > What I think we should consider is whether Fedora should have a separate
> > entity or group that accepts blockers. Right now, that responsibility is
> > largely falling on your team.
>
> This isn't really the intent. The blocker meeting SOP -
> https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting - reads:
>
> "Blocker Bug Meetings are not owned by any one team in Fedora. They are
> a collaborative effort between Release Engineering, Quality Assurance,
> Development, and Project Management."
>
> and the intent is that, ideally, some or all of those groups should be
> represented at blocker review meetings. Lately, it seems that this
> hasn't been happening as much as it used to. We used to have fairly
> solid attendance from releng, FPL / FPM, and interested development
> parties. Recently it seems like these groups just aren't terribly
> interested in showing up and the meetings tend to be all or mostly QA
> people. And we go ahead and do the evaluations, because hell, someone
> has to. But as the procedure clearly states, it's not a QA-owned
> process, it's meant to be collaborative, and the onus is really on other
> interested parties to _show up and take part_. We do usually ping
> #anaconda when blocker review starts up and ask if anyone wants to take
> part; often no-one does. I think it's seen as taking away precious
> development time. It would be really nice if we starting getting more
> developers and Robyn and Jaroslav showing up for more blocker reviews.
> Admittedly, the meetings do get very long sometimes, but we've found it
> pretty hard to resolve that (reviewing 30 blockers is going to take a
> while, however you slice it). I do think it helps to have multiple
> perspectives, and that QA folks often tend to be more 'liberal' about
> accepting blockers than other groups; it would definitely be useful to
> have other groups present to act as a check on this.
>
> It might be an idea to start CCing the blocker meeting announcements out
> a bit more widely, to help with this. Or heck, we could also have some
> non-QA person run the meetings for a while, give it to FPL or FPM or
> someone. That would liven things up.

The SOP can say whatever it wants, but what actually happens in practice
matters. People don't go to the meetings because they take too long. I'm
going to make a comparison to RHEL process again, but I think it's useful
here. Meetings have a hard time limit. There is only so much work a person
can do in a day and it's ridiculous to keep going through bug status over
and over and over just because it's there. When RHEL meetings get in to
"bugzilla bingo", they cover what can be covered in the allotted time. What
isn't covered isn't forgotten, just brought up at the next meeting.

And from my own point of view, I've never seen your invites in #anaconda
simply because I do not look at IRC unless addressed by my nick. I can't.
I don't have time to sit and watch the text scroll. The length of this
reply that you sent....I get hundreds of those per day that I'm expected to
read and respond to. Plus all the meetings I'm scheduled for on a daily
basis. This is not me whining, just trying to explain how I make the most
effective use of the resources I have. Direct invites to these meetings
would help me work them in to my schedule and perhaps even attend.

--
David Cantrell <dcantrell@redhat.com>
Manager, Installer Engineering Team
Red Hat, Inc. | Westford, MA | EST5EDT

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 09:09 PM
Adam Williamson
 
Default Review of Fedora 18 Release Criteria

On Wed, 2012-10-10 at 16:43 -0400, David Cantrell wrote:

> > I don't think my disagreement is necessarily a problem for you,
> > though...let me see if I can put it a different way:
> >
> > Take the example you just gave. My position isn't 'we should take that
> > bug as a blocker both four weeks out and one week out'. My position is
> > 'we shouldn't take it as a blocker four weeks out _or_ one week out'.
> >
> > I'd phrase the problem more as 'there is a tendency in blocker review to
> > accept things too casually early on'. We should only accept things as
> > blockers that we really believe we should hold the release up for - even
> > if it's six weeks until release and we know the bug is going to be fixed
> > in two days, we should not accept it as a blocker unless it passes the
> > 'if today was go/no-go and this was the last bug, would we block the
> > release' smell test.
> >
> > Rather than throw almost everything on the blocker list when it's early
> > and get more rigorous later on, I'd rather we just be rigorous the whole
> > time. The blocker list isn't a work list (using the blocker list as a
> > 'todo list' / reminder list is a tendency I've noticed in both RHEL and
> > Fedora, and something I think we should fight), it is a list of bugs
> > that must be fixed for the release to go out. No more, no less. If we
> > wouldn't really block the release for a bug, it doesn't belong on the
> > list at any point in time. I think we've been making a good effort to
> > improve this in recent releases; it's not just an aspiration.
>
> That's fine, but how is that not a to do list?

I should have said not _solely_ a to-do list. The phenomenon I had in
mind was when people throw stuff on the blocker list not because they
really think it should block release, but just because they know if it's
on the blocker list it'll get treated with high priority (either a user
trying to 'game the system' into getting their bug fixed quickly, or a
developer abusing the blocker list as their todo list). That happens,
and it shouldn't (we've been pretty solid about rejecting such proposals
recently).

So yeah, the blocker list ultimately winds up functioning as a to-do
list, but that's a consequence, not what it's actually _for_. 'I want to
make sure we don't forget about this' isn't a valid reason for a bug to
be on the blocker list. People should be using another tracker bug for
that, or the 'Priority' field, or anything at all else that they like,
but not the blocker list

> > So if there are concrete cases where you think bugs have been accepted
> > as blockers that you wouldn't be comfortable taking as blockers the day
> > before release, that's _always a problem_, and please highlight those
> > bugs to us. I don't think it's 'not a problem' if we accepted them as
> > blockers a long time before release and they got fixed before any chance
> > of causing a slip, it's still a problem. We should stick firmly to our
> > tight, minimal definition of what constitutes a blocker, regardless of
> > the point in the release schedule.
>
> The big difference here and what we see on the RHEL side is that RHEL allows
> for blocker bugs to be re-evaluated and booted if a later workaround or some
> other acceptable solution besides a bug fix is found. It becomes a schedule
> thing, which I am routinely told is very important for Fedora.

This is certainly the case for Fedora too. Accepted blockers are
reviewed at each meeting - usually just to check progress, but the
blocker status of the bug can also be reassessed if new information that
changes the equation has showed up. We've certainly done this on many
occasions (too often, for some people's liking).

> If something marked as a blocker early on is still not fixed through several
> test releases, does it really matter anymore for the release? If not, why
> not boot it off the blocker list?

Again that doesn't seem like a good factor to take into account. The
fact that it hasn't been fixed doesn't tell us anything about how bad a
bug it is. If it hasn't been fixed _and no-one has complained_, then
that might be something to take into account, as it would indicate the
impact of the bug is narrow. But if it hasn't been fixed, but there are
200 people on the bug report baying for blood, taking it off the blocker
list doesn't seem indicated

> As we have both already stated and agreed on, the installer is unique. Our
> development process is so very much tied to the Fedora release schedule.
> There is no easy way for us to develop something large like a new installer
> and not have it tied to a release. This became harder for us when we
> stopped doing rawhide nightly composes many years ago (and I have no idea if
> those are happening again these days). People don't go and download a new
> installer to try it out. You download a release of something and try
> installing it. Our only way to get that to users is to follow the Fedora
> release schedule, unless we want to start making our own distribution.
>
> I know what you're getting at and it's that newui was a huge feature and
> that's made our lives hard. Unfortunately, the Fedora release schedule is
> such that work like this is always painful for us.

Well, there's a case there that we should change the release schedule,
then. It's a Perennial Topic, I know, but it could happen. We have a
three sided triangle here:

Shiny New Features


Quality Length of Release Schedule

You can pick any two, basically. You can have a short release schedule
and lots of shiny new features. You can have a short release cycle and
great quality. You can have great quality and shiny new features. But it
does seem rather difficult to achieve a short release schedule, great
quality, and lots of shiny new features. Broadly, we compromise on
quality to achieve this - our standards are a hell of a lot slacker
than, oh, say Debian's. But if we get to the point where we'd have to
compromise our quality standards too heavily to achieve the shiny new
features we want on the current release cycle, we have to bring the
release cycle point of the triangle into the equation too.

> > It might be an idea to start CCing the blocker meeting announcements out
> > a bit more widely, to help with this. Or heck, we could also have some
> > non-QA person run the meetings for a while, give it to FPL or FPM or
> > someone. That would liven things up.
>
> The SOP can say whatever it wants, but what actually happens in practice
> matters.

Sure, I agree, and as I said, I can see two or three practical things we
could do to try and address this.

> People don't go to the meetings because they take too long. I'm
> going to make a comparison to RHEL process again, but I think it's useful
> here. Meetings have a hard time limit. There is only so much work a person
> can do in a day and it's ridiculous to keep going through bug status over
> and over and over just because it's there. When RHEL meetings get in to
> "bugzilla bingo", they cover what can be covered in the allotted time. What
> isn't covered isn't forgotten, just brought up at the next meeting.

We've started doing that this week, actually, and limiting the meetings
to three hours. But we're not just delaying stuff to the next meeting
because it'd just keep piling up, we're just running another meeting the
next day if we overrun. So we're doing another review meeting tomorrow
A.M. because we didn't get through all the bugs today in 3 hours.

> And from my own point of view, I've never seen your invites in #anaconda
> simply because I do not look at IRC unless addressed by my nick. I can't.
> I don't have time to sit and watch the text scroll. The length of this
> reply that you sent....I get hundreds of those per day that I'm expected to
> read and respond to. Plus all the meetings I'm scheduled for on a daily
> basis. This is not me whining, just trying to explain how I make the most
> effective use of the resources I have. Direct invites to these meetings
> would help me work them in to my schedule and perhaps even attend.

I know I get wordy, sorry, it's a tendency of mine...I like to think
about things on a very broad / long-term / comprehensive scale and I
tend to develop those thoughts in discussions like this (which I find
really useful, btw, so thanks).

I'm not that happy with having a giant list of direct CCs on
announcement mails because it's just inelegant and bitrots quickly (you
wind up with announcements going to the person who used to be the FPL
three years ago). It's what announcement MLs are for, after all: posting
important announcements and nothing else. test-announce is very low
traffic, I don't think it's a huge burden to subscribe to. We could
certainly start CCing devel-announce and a couple other major announce
lists if we aren't already. Is there an anaconda-announce?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 09:20 PM
Adam Williamson
 
Default Review of Fedora 18 Release Criteria

On Wed, 2012-10-10 at 14:09 -0700, Adam Williamson wrote:

> I'm not that happy with having a giant list of direct CCs on
> announcement mails because it's just inelegant and bitrots quickly (you
> wind up with announcements going to the person who used to be the FPL
> three years ago). It's what announcement MLs are for, after all: posting
> important announcements and nothing else. test-announce is very low
> traffic, I don't think it's a huge burden to subscribe to. We could
> certainly start CCing devel-announce and a couple other major announce
> lists if we aren't already. Is there an anaconda-announce?

Er, and of course there's the point that the blocker review meetings are
on a regular schedule. We changed from Fridays to Wednesdays this cycle,
but we don't shift 'em about all over the place. There's one every
Wednesday morning at the same time, just stick the time slot in your
calendar. We also sometimes do a mini review in the QA meeting on
Mondays.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 09:25 PM
Jesse Keating
 
Default Review of Fedora 18 Release Criteria

On 10/10/2012 01:43 PM, David Cantrell wrote:

Rather than throw almost everything on the blocker list when it's early
>and get more rigorous later on, I'd rather we just be rigorous the whole
>time. The blocker list isn't a work list (using the blocker list as a
>'todo list' / reminder list is a tendency I've noticed in both RHEL and
>Fedora, and something I think we should fight), it is a list of bugs
>that must be fixed for the release to go out. No more, no less. If we
>wouldn't really block the release for a bug, it doesn't belong on the
>list at any point in time. I think we've been making a good effort to
>improve this in recent releases; it's not just an aspiration.

That's fine, but how is that not a to do list?


I think it's a "must-do" list, which isn't the entirety of a "to-do"
list. If we look at both the Blocker list /and/ the Nice to Have list
together, then it starts to look more like a "to do" list.


I think one of the problems here is the difference between RHEL process
and Fedora process. RHEL doesn't seem to have a "Nice to have" concept.
RHEL seems to be "throw everything on the blocker list, because we
would take a fix for this after a freeze, and we can remove it from the
blocker list later if we don't get it done". Fedora isn't like that.
We have two lists, a list of things we would truly delay the release
for, and a list of things we'd take after a freeze if they get fixed,
but won't otherwise delay the release.





>So if there are concrete cases where you think bugs have been accepted
>as blockers that you wouldn't be comfortable taking as blockers the day
>before release, that's_always a problem_, and please highlight those
>bugs to us. I don't think it's 'not a problem' if we accepted them as
>blockers a long time before release and they got fixed before any chance
>of causing a slip, it's still a problem. We should stick firmly to our
>tight, minimal definition of what constitutes a blocker, regardless of
>the point in the release schedule.

The big difference here and what we see on the RHEL side is that RHEL allows
for blocker bugs to be re-evaluated and booted if a later workaround or some
other acceptable solution besides a bug fix is found. It becomes a schedule
thing, which I am routinely told is very important for Fedora.

If something marked as a blocker early on is still not fixed through several
test releases, does it really matter anymore for the release? If not, why
not boot it off the blocker list?


What do you mean by test releases here? We've done multiple "test
composes" of an alpha or a beta leading up to a "release candidate". We
do these test composes, even with known blockers, because /some/ of the
things have been fixed and it's good to get testing on that. But
"release candidate" doesn't happen until all the blockers are met or
removed. Blockers are frequently removed from the list when re-evaluated.





> >We do want to make a good release and keep the users happy, but we need to
> >also remember that developers are people too and not just meat grinders for
> >bug reports.

>
>Sure, hence the need to keep the criteria realistic.
>

> >1) Adjustments to the blocker acceptance criteria that I explained.
> >2) Acknowledgement that concurrent releases are always in play for the
> > installer team. Generally two RHEL releases and one Fedora release.
> > RHEL gets priority.

>
>I would prefer to phrase 2) the way I already did: Fedora should ensure
>its release requirements are realistic in the context of the resources
>available. If the 'resources available' are to an extent defined by
>other commitments on the part of the anaconda team, then so be it, but
>I'd rather consider it in that context, rather than haul RHEL
>specifically into a Fedora context.

Call it want you want, but we all know it's RHEL.


> >What I'm asking for is blocker acceptance criteria changes that are more
> >realistic given remaining time and other commitments. Or dump the
> >time/schedule requirement and just say we'll call it done when it's done.

>
>The schedule / blocker process balance is a bit of a delicate one, yeah.
>In practice we start getting a lot of pressure if the blocker process
>results in delays of several weeks. I think I'd like to look at this
>way, though: any time that following the release validation process
>properly results in a long slip,_that means something is wrong_. It
>usually means either that the release criteria are excessive, or it
>means that something went wrong in the feature process. I believe the
>feature process as it stands is not adequate in ensuring proposed
>features are really viable on the Fedora release schedule, and FESCo has
>also not always been assertive enough in dropping or delaying features
>that are clearly not meeting the schedule; I think sometimes problems
>are due to that, rather than to over-ambitious release requirements.

I'll agree with that. I've had that complaint for a long time, ever since
we merged Core and Extras.

As we have both already stated and agreed on, the installer is unique. Our
development process is so very much tied to the Fedora release schedule.
There is no easy way for us to develop something large like a new installer
and not have it tied to a release. This became harder for us when we
stopped doing rawhide nightly composes many years ago (and I have no idea if
those are happening again these days). People don't go and download a new
installer to try it out. You download a release of something and try
installing it. Our only way to get that to users is to follow the Fedora
release schedule, unless we want to start making our own distribution.

I know what you're getting at and it's that newui was a huge feature and
that's made our lives hard. Unfortunately, the Fedora release schedule is
such that work like this is always painful for us.



So the no images in rawhide thing was kinda my doing. It came from a
meeting a few years ago with release engineering, QA, installer team,
and others, where we looked at how our development process was going and
where all the painful points where.


One of the major pain points identified was how frequently rawhide
wasn't installable, and how it would turn people away. After talking
with folks it seemed like a better idea was to /not/ present a broken
set of images each night to people when we had no idea of they would
work. Instead what we wanted to do was keep rawhide as just a
repository of packages. We would then create "known good" images that
people could use as a starting point to get on rawhide, from that point
it was just yum update every day.


Where this may have failed is in the creation of the known good images.
Of course to know they're good, they must be created first, then
tested. There was schedule points for creating composes to smoke test,
with the hopes of identifying a compose that was "good enough" to be
promoted as a last known good. I think we're probably missing some
infrastructure here, and some cooperation between installer devs and qa
and releng.


The happy world I envisioned was an installer team able to work on their
own on large changes, not bothered by people constantly telling them
that rawhide was broken, again. When the team thought they'd reached
some milestone and wanted to start getting testing, they would work with
QA and releng to get some images produced and ran through smoke tests.
If good, promoted, if not fix it and try again, iterate. If this happy
world isn't something we think can be managed, then we really should
re-evaluate the no images in rawhide thing again.


--
Jesse Keating
Fedora -- Freedom˛ is a feature!

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 10:21 PM
Tim Flink
 
Default Review of Fedora 18 Release Criteria

On Wed, 10 Oct 2012 16:43:36 -0400
David Cantrell <dcantrell@redhat.com> wrote:

snip

> > > I didn't read this paragraph. This is a lot of text, but I've
> > > skimmed it. What I think we should consider is whether Fedora
> > > should have a separate entity or group that accepts blockers.
> > > Right now, that responsibility is largely falling on your team.
> >
> > This isn't really the intent. The blocker meeting SOP -
> > https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting - reads:
> >
> > "Blocker Bug Meetings are not owned by any one team in Fedora. They
> > are a collaborative effort between Release Engineering, Quality
> > Assurance, Development, and Project Management."
> >
> > and the intent is that, ideally, some or all of those groups should
> > be represented at blocker review meetings. Lately, it seems that
> > this hasn't been happening as much as it used to. We used to have
> > fairly solid attendance from releng, FPL / FPM, and interested
> > development parties. Recently it seems like these groups just
> > aren't terribly interested in showing up and the meetings tend to
> > be all or mostly QA people. And we go ahead and do the evaluations,
> > because hell, someone has to. But as the procedure clearly states,
> > it's not a QA-owned process, it's meant to be collaborative, and
> > the onus is really on other interested parties to _show up and take
> > part_. We do usually ping #anaconda when blocker review starts up
> > and ask if anyone wants to take part; often no-one does. I think
> > it's seen as taking away precious development time. It would be
> > really nice if we starting getting more developers and Robyn and
> > Jaroslav showing up for more blocker reviews. Admittedly, the
> > meetings do get very long sometimes, but we've found it pretty hard
> > to resolve that (reviewing 30 blockers is going to take a while,
> > however you slice it). I do think it helps to have multiple
> > perspectives, and that QA folks often tend to be more 'liberal'
> > about accepting blockers than other groups; it would definitely be
> > useful to have other groups present to act as a check on this.
> >
> > It might be an idea to start CCing the blocker meeting
> > announcements out a bit more widely, to help with this. Or heck, we
> > could also have some non-QA person run the meetings for a while,
> > give it to FPL or FPM or someone. That would liven things up.
>
> The SOP can say whatever it wants, but what actually happens in
> practice matters. People don't go to the meetings because they take
> too long. I'm going to make a comparison to RHEL process again, but
> I think it's useful here. Meetings have a hard time limit. There is
> only so much work a person can do in a day and it's ridiculous to
> keep going through bug status over and over and over just because
> it's there. When RHEL meetings get in to "bugzilla bingo", they
> cover what can be covered in the allotted time. What isn't covered
> isn't forgotten, just brought up at the next meeting.

Then we should change what's going on in practice. I'm generally
all for changes in the blocker review process if they make sense. If
there are reasonable things we can do to increase developer
participation, let's make those changes.

As Adam mentioned, we started imposing a time limit on the blocker
review meetings last week. I think that everyone's eyes start to glaze
over when the blocker meetings go on too long - I know mine do.

It seems that the 3 hour time limit has been going over well so far and
I think that we're going to continue doing that for now. I'm not sure
that it'll work so well if we have go/no-go the next day but I'm also
hoping that we won't have lists of 30+ proposed blockers to review that
late in the release cycle.

> And from my own point of view, I've never seen your invites in
> #anaconda simply because I do not look at IRC unless addressed by my
> nick. I can't. I don't have time to sit and watch the text scroll.
> The length of this reply that you sent....I get hundreds of those per
> day that I'm expected to read and respond to. Plus all the meetings
> I'm scheduled for on a daily basis. This is not me whining, just
> trying to explain how I make the most effective use of the resources
> I have. Direct invites to these meetings would help me work them in
> to my schedule and perhaps even attend.

When you say direct invites, are you talking about emails a day or so
ahead of time or IRC pings when the bugs come up?

Since I've been running the blocker review meetings as of late, I'm
fine with generating emails the day before and/or pinging people when
their bugs come up. The only problem is who to contact and when - using
anaconda as an example, do I send invites to anaconda-maint-list@ the
day before? How do I know who to ping over IRC when there isn't a
single assigned person to the bug? Are email invites more useful the day
before or 2 hours before or both?

I'm also going to start sorting the bug list before the meeting starts
so that we cover all of a component's bugs at the same time instead of
having them all spread out. Admittedly, that's something I should have
started doing a while ago but I can't change that - I can change how
the list is generated going forward, though.

I have some other ideas on how to improve the review process in the
long term but those ideas would take time to implement and probably
belong in another thread.

Tim
_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 10:50 PM
Tim Flink
 
Default Review of Fedora 18 Release Criteria

On Wed, 10 Oct 2012 14:25:37 -0700
Jesse Keating <jkeating@redhat.com> wrote:

snip

> > As we have both already stated and agreed on, the installer is
> > unique. Our development process is so very much tied to the Fedora
> > release schedule. There is no easy way for us to develop something
> > large like a new installer and not have it tied to a release. This
> > became harder for us when we stopped doing rawhide nightly composes
> > many years ago (and I have no idea if those are happening again
> > these days). People don't go and download a new installer to try
> > it out. You download a release of something and try installing
> > it. Our only way to get that to users is to follow the Fedora
> > release schedule, unless we want to start making our own
> > distribution.
> >
> > I know what you're getting at and it's that newui was a huge
> > feature and that's made our lives hard. Unfortunately, the Fedora
> > release schedule is such that work like this is always painful for
> > us.
>
>
> So the no images in rawhide thing was kinda my doing. It came from a
> meeting a few years ago with release engineering, QA, installer team,
> and others, where we looked at how our development process was going
> and where all the painful points where.
>
> One of the major pain points identified was how frequently rawhide
> wasn't installable, and how it would turn people away. After talking
> with folks it seemed like a better idea was to /not/ present a broken
> set of images each night to people when we had no idea of they would
> work. Instead what we wanted to do was keep rawhide as just a
> repository of packages. We would then create "known good" images
> that people could use as a starting point to get on rawhide, from
> that point it was just yum update every day.
>
> Where this may have failed is in the creation of the known good
> images. Of course to know they're good, they must be created first,
> then tested. There was schedule points for creating composes to
> smoke test, with the hopes of identifying a compose that was "good
> enough" to be promoted as a last known good. I think we're probably
> missing some infrastructure here, and some cooperation between
> installer devs and qa and releng.

Actually, we tried to get this done for F18 (the fedora build service
GSoC project) and simply ran out of time. We have most of the
infrastructure needed and to the best of my knowledge, nobody was
against the idea - it just wasn't done in time for F18.

I know that Amit (the student who worked on the GSoC project - now
working for Red Hat) and I are still interested in making this happen,
we just haven't had the time to do anything about it since the F18
release started up and the system is not ready for public use yet.

The idea is to eventually have a system that is capable of the
following:
- keep multiple base repos for devs to work with
- allow for selective package updates in the repos
* The only requirement would be to be able to download the new
packages from koji using the koji client.
- accept custom kickstarts
- build live, DVD and netinstall images on demand
- host the built images for a certain amount of time before reclaiming
the space

Some of those features are a little farther than others (mostly repo
management instead of basing off of rawhide/f18 etc.) but it's all
do-able.

> The happy world I envisioned was an installer team able to work on
> their own on large changes, not bothered by people constantly telling
> them that rawhide was broken, again. When the team thought they'd
> reached some milestone and wanted to start getting testing, they
> would work with QA and releng to get some images produced and ran
> through smoke tests. If good, promoted, if not fix it and try again,
> iterate. If this happy world isn't something we think can be
> managed, then we really should re-evaluate the no images in rawhide
> thing again.

Honestly, I'd love to see something like this happen. I think it would
lead to less total pain for all involved - for qa, anaconda devs and
probably other devs as well. It would require quite a bit of work to
figure out how we'd manage such a process and implement any tooling
needed to do so but it would probably be worth at least looking in to.

Tim
_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-10-2012, 10:54 PM
Adam Williamson
 
Default Review of Fedora 18 Release Criteria

On Wed, 2012-10-10 at 14:09 -0700, Adam Williamson wrote:
> We have a three sided triangle here:

Just in case any geometry buffs in the audience were wondering, yes I
did catch that when I read it back, and yes I did slap myself repeatedly
in the face.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-11-2012, 01:32 PM
David Cantrell
 
Default Review of Fedora 18 Release Criteria

On Wed, Oct 10, 2012 at 02:25:37PM -0700, Jesse Keating wrote:
> On 10/10/2012 01:43 PM, David Cantrell wrote:
> >>Rather than throw almost everything on the blocker list when it's early
> >>>and get more rigorous later on, I'd rather we just be rigorous the whole
> >>>time. The blocker list isn't a work list (using the blocker list as a
> >>>'todo list' / reminder list is a tendency I've noticed in both RHEL and
> >>>Fedora, and something I think we should fight), it is a list of bugs
> >>>that must be fixed for the release to go out. No more, no less. If we
> >>>wouldn't really block the release for a bug, it doesn't belong on the
> >>>list at any point in time. I think we've been making a good effort to
> >>>improve this in recent releases; it's not just an aspiration.
> >That's fine, but how is that not a to do list?
>
> I think it's a "must-do" list, which isn't the entirety of a "to-do"
> list. If we look at both the Blocker list /and/ the Nice to Have
> list together, then it starts to look more like a "to do" list.

Yeah, that's a fair assessment.

> I think one of the problems here is the difference between RHEL
> process and Fedora process. RHEL doesn't seem to have a "Nice to
> have" concept. RHEL seems to be "throw everything on the blocker
> list, because we would take a fix for this after a freeze, and we
> can remove it from the blocker list later if we don't get it done".
> Fedora isn't like that. We have two lists, a list of things we would
> truly delay the release for, and a list of things we'd take after a
> freeze if they get fixed, but won't otherwise delay the release.

At least from my point of view, RHEL does have a nice to have concept, but
it's at devel discretion and closes off much earlier I think than Fedora
does. But to me that's just one of the differences between slow-moving RHEL
and fast-moving Fedora.

> >>>So if there are concrete cases where you think bugs have been accepted
> >>>as blockers that you wouldn't be comfortable taking as blockers the day
> >>>before release, that's_always a problem_, and please highlight those
> >>>bugs to us. I don't think it's 'not a problem' if we accepted them as
> >>>blockers a long time before release and they got fixed before any chance
> >>>of causing a slip, it's still a problem. We should stick firmly to our
> >>>tight, minimal definition of what constitutes a blocker, regardless of
> >>>the point in the release schedule.
> >The big difference here and what we see on the RHEL side is that RHEL allows
> >for blocker bugs to be re-evaluated and booted if a later workaround or some
> >other acceptable solution besides a bug fix is found. It becomes a schedule
> >thing, which I am routinely told is very important for Fedora.
> >
> >If something marked as a blocker early on is still not fixed through several
> >test releases, does it really matter anymore for the release? If not, why
> >not boot it off the blocker list?
>
> What do you mean by test releases here? We've done multiple "test
> composes" of an alpha or a beta leading up to a "release candidate".
> We do these test composes, even with known blockers, because /some/
> of the things have been fixed and it's good to get testing on that.
> But "release candidate" doesn't happen until all the blockers are
> met or removed. Blockers are frequently removed from the list when
> re-evaluated.

I was specifically referring to the Alpha and Beta releases, not the
additional test composes we do. Apologies for the vocabulary confusion. My
example was really just, "if something filed before Alpha as a blocker still
hasn't been closed by RC and no one has complained, should we really care
about it for this release?"

> >>>> >We do want to make a good release and keep the users happy, but we need to
> >>>> >also remember that developers are people too and not just meat grinders for
> >>>> >bug reports.
> >>>
> >>>Sure, hence the need to keep the criteria realistic.
> >>>
> >>>> >1) Adjustments to the blocker acceptance criteria that I explained.
> >>>> >2) Acknowledgement that concurrent releases are always in play for the
> >>>> > installer team. Generally two RHEL releases and one Fedora release.
> >>>> > RHEL gets priority.
> >>>
> >>>I would prefer to phrase 2) the way I already did: Fedora should ensure
> >>>its release requirements are realistic in the context of the resources
> >>>available. If the 'resources available' are to an extent defined by
> >>>other commitments on the part of the anaconda team, then so be it, but
> >>>I'd rather consider it in that context, rather than haul RHEL
> >>>specifically into a Fedora context.
> >Call it want you want, but we all know it's RHEL.
> >
> >>>> >What I'm asking for is blocker acceptance criteria changes that are more
> >>>> >realistic given remaining time and other commitments. Or dump the
> >>>> >time/schedule requirement and just say we'll call it done when it's done.
> >>>
> >>>The schedule / blocker process balance is a bit of a delicate one, yeah.
> >>>In practice we start getting a lot of pressure if the blocker process
> >>>results in delays of several weeks. I think I'd like to look at this
> >>>way, though: any time that following the release validation process
> >>>properly results in a long slip,_that means something is wrong_. It
> >>>usually means either that the release criteria are excessive, or it
> >>>means that something went wrong in the feature process. I believe the
> >>>feature process as it stands is not adequate in ensuring proposed
> >>>features are really viable on the Fedora release schedule, and FESCo has
> >>>also not always been assertive enough in dropping or delaying features
> >>>that are clearly not meeting the schedule; I think sometimes problems
> >>>are due to that, rather than to over-ambitious release requirements.
> >I'll agree with that. I've had that complaint for a long time, ever since
> >we merged Core and Extras.
> >
> >As we have both already stated and agreed on, the installer is unique. Our
> >development process is so very much tied to the Fedora release schedule.
> >There is no easy way for us to develop something large like a new installer
> >and not have it tied to a release. This became harder for us when we
> >stopped doing rawhide nightly composes many years ago (and I have no idea if
> >those are happening again these days). People don't go and download a new
> >installer to try it out. You download a release of something and try
> >installing it. Our only way to get that to users is to follow the Fedora
> >release schedule, unless we want to start making our own distribution.
> >
> >I know what you're getting at and it's that newui was a huge feature and
> >that's made our lives hard. Unfortunately, the Fedora release schedule is
> >such that work like this is always painful for us.
>
>
> So the no images in rawhide thing was kinda my doing. It came from
> a meeting a few years ago with release engineering, QA, installer
> team, and others, where we looked at how our development process was
> going and where all the painful points where.
>
> One of the major pain points identified was how frequently rawhide
> wasn't installable, and how it would turn people away. After
> talking with folks it seemed like a better idea was to /not/ present
> a broken set of images each night to people when we had no idea of
> they would work. Instead what we wanted to do was keep rawhide as
> just a repository of packages. We would then create "known good"
> images that people could use as a starting point to get on rawhide,
> from that point it was just yum update every day.

I remember when all that went down and I think it made sense at the time.
The name 'rawhide' has certain expectations associated with it that we will
probably never be able to change. At the time I found it frustrating
because we relied on the nightly composes to do installer development. But
I understand that since it was publicly available, we needed to stop the
constant "nightlies are broken" reports from users.

> Where this may have failed is in the creation of the known good
> images. Of course to know they're good, they must be created first,
> then tested. There was schedule points for creating composes to
> smoke test, with the hopes of identifying a compose that was "good
> enough" to be promoted as a last known good. I think we're probably
> missing some infrastructure here, and some cooperation between
> installer devs and qa and releng.

Known good rawhide composes would be a nice thing to have.

> The happy world I envisioned was an installer team able to work on
> their own on large changes, not bothered by people constantly
> telling them that rawhide was broken, again. When the team thought
> they'd reached some milestone and wanted to start getting testing,
> they would work with QA and releng to get some images produced and
> ran through smoke tests. If good, promoted, if not fix it and try
> again, iterate. If this happy world isn't something we think can be
> managed, then we really should re-evaluate the no images in rawhide
> thing again.

I think this is possible. We need to work against rawhide, so as long as we
have a regular compose process building images from that for us, we can work
this way. But it sort of only gets us so far. The more large changes we do
outside of official repos, the more integration work we stack up for
ourselves later.

--
David Cantrell <dcantrell@redhat.com>
Manager, Installer Engineering Team
Red Hat, Inc. | Westford, MA | EST5EDT

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 10-11-2012, 04:19 PM
Adam Williamson
 
Default Review of Fedora 18 Release Criteria

On Thu, 2012-10-11 at 09:32 -0400, David Cantrell wrote:

> > I think it's a "must-do" list, which isn't the entirety of a "to-do"
> > list. If we look at both the Blocker list /and/ the Nice to Have
> > list together, then it starts to look more like a "to do" list.
>
> Yeah, that's a fair assessment.

I think we're all on the same page here, but just as a side note, it
only really looks like that if you're anaconda. For any other component,
they'll have maybe a max of 5 bugs on the blocker and NTH list for any
given release. So they'll have lots of time for other stuff too. It's
only for anaconda that the bugs on the blocker and NTH lists, together,
look like a full-time job

> > I think one of the problems here is the difference between RHEL
> > process and Fedora process. RHEL doesn't seem to have a "Nice to
> > have" concept. RHEL seems to be "throw everything on the blocker
> > list, because we would take a fix for this after a freeze, and we
> > can remove it from the blocker list later if we don't get it done".
> > Fedora isn't like that. We have two lists, a list of things we would
> > truly delay the release for, and a list of things we'd take after a
> > freeze if they get fixed, but won't otherwise delay the release.
>
> At least from my point of view, RHEL does have a nice to have concept, but
> it's at devel discretion and closes off much earlier I think than Fedora
> does. But to me that's just one of the differences between slow-moving RHEL
> and fast-moving Fedora.

Even if that's the case, it leaves the door open for the problem Jesse
noted (thanks for mentioning NTH specifically, Jesse, I somehow missed
it) - once this RHEL nth concept is 'closed off', the only way to get
something through the freeze is to put it on the blocker list, yes? So
you still end up with things going on the blocker list just to get
through freeze. Which is indeed the problem the NTH process is meant to
solve (and in practice seems to solve quite well).

Honestly, one thing I'm getting out of this conversation is that RHEL
could afford to pick up some Fedora concepts

> I was specifically referring to the Alpha and Beta releases, not the
> additional test composes we do. Apologies for the vocabulary confusion. My
> example was really just, "if something filed before Alpha as a blocker still
> hasn't been closed by RC and no one has complained, should we really care
> about it for this release?"

With the 'and no one has complained' proviso, yeah, that is something we
take into account. We took a bug off the Beta blocker list on more or
less exactly this grounds, yesterday.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 

Thread Tools




All times are GMT. The time now is 04:29 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org