Hi,

On Fri, Mar 26, 2010 at 23:49, Adam Williamson <awilliam@redhat.com> wrote:
[... snip ...]
> 1. I have tried this update in my regular day-to-day use and seen no
> regressions.
>
> 2. I have tried this update in my regular day-to-day use and seen a
> regression: bug #XXXXXX.
>
> 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does fix bug #XXXXXX.
>
> 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does not fix bug #XXXXXX.
>
> 5. I have performed the following planned testing on the update: (link
> to test case / test plan) and it passes.
>
> 6. I have performed the following planned testing on the update: (link
> to test case / test plan) and it fails: bug #XXXXXX.

This is basically what Doug had proposed, except that you added 5. and 6.

I know Luke also likes the idea, and I've been toying with
implementing Doug's idea in the Bodhi2 branch. Adding those 2 more
karma types shouldn't be too hard.

>Testers should be able to file multiple types of feedback in one
> operation - for instance, 4+1 (the update didn't fix the bug it claimed
> to, but doesn't seem to cause any regressions either). Ideally, the
> input of feedback should be 'guided' with a freeform element, so there's
> a space to enter bug numbers, for instance.

That's what I had in mind, but the Bodhi2 branch currently doesn't
have any controllers/template, so it will be some time before I have
something to show.

> I think Bill's proposed policy can be modified quite easily to fit this.
> All it would need to say is that for 'important' updates to be accepted,
> they would need to have one 'type 1' feedback from a proven tester, and
> no 'type 2' feedback from anyone (or something along those lines; this
> isn't the main thrust of my post, please don't sidetrack it too
> much :>).

That's actually one of the points I was missing : rules for
(auto-)push / (auto-)unpush. But yeah, it can be agreed on later.

> The system could do a count of how many of each type of feedback any
> given update has received, but I don't think there's any way we can
> sensibly do some kind of mathematical operation on those numbers and
> have a 'rating' for the update. Such a system would always give odd /
> undesirable results in some cases, I think (just as the current one
> does). I believe the above system would be sufficiently clear that there
> would be no need for such a number, and we would be able to evaluate
> updates properly based just on the information listed.
>
> What are everyone's thoughts on this? Thanks!

I totally agree that this would be far more desirable than the current
+1/-1 system.


----------
Mathieu Bridon
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
> Hi,
>
> On Fri, Mar 26, 2010 at 23:49, Adam Williamson <awilliam@redhat.com> wrote:
> [... snip ...]
> > 1. I have tried this update in my regular day-to-day use and seen no
> > regressions.
> >
> > 2. I have tried this update in my regular day-to-day use and seen a
> > regression: bug #XXXXXX.
> >
> > 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
> > and found that it does fix bug #XXXXXX.
> >
> > 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
> > and found that it does not fix bug #XXXXXX.
> >
> > 5. I have performed the following planned testing on the update: (link
> > to test case / test plan) and it passes.
> >
> > 6. I have performed the following planned testing on the update: (link
> > to test case / test plan) and it fails: bug #XXXXXX.
>
> This is basically what Doug had proposed, except that you added 5. and 6.

Great, glad to hear we're thinking in the same direction from different
angles Do you have a link to his proposal? I don't recall reading it.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, Mar 27, 2010 at 00:31, Adam Williamson <awilliam@redhat.com> wrote:
> On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
>> This is basically what Doug had proposed, except that you added 5. and 6.
>
> Great, glad to hear we're thinking in the same direction from different
> angles Do you have a link to his proposal? I don't recall reading it.

Here it is:
http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html


----------
Mathieu Bridon
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, 2010-03-27 at 00:35 +0100, Mathieu Bridon wrote:
> On Sat, Mar 27, 2010 at 00:31, Adam Williamson <awilliam@redhat.com> wrote:
> > On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
> >> This is basically what Doug had proposed, except that you added 5. and 6.
> >
> > Great, glad to hear we're thinking in the same direction from different
> > angles Do you have a link to his proposal? I don't recall reading it.
>
> Here it is:
> http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html

Oh, yeah, that one. I even replied to it, but it obviously fell prey to
my mind-erase of the whole mess of a thread =)

in that case, yep, my proposal is more or less just a slight elaboration
on Doug's indeed. I may even have subconsciously brought in some
elements from Doug's when writing it, so thanks Doug!

if you're working in this direction in improving Bodhi, then, from the
'update testing policy' angle, I think I'll just leave it at 'wait and
see' for the new Bodhi version. Do you have a vague ETA on when we can
expect the improved Bodhi?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, 2010-03-27 at 03:50 +0100, Kevin Kofler wrote:

> So I don't think blocking an update outright for having received type 2
> feedback is sane at all.

Sigh. That was why I said not to sidetrack the discussion because it was
the least important bit of the post. It was just an example of how
easily the policy can be adapted. I'm really not interested in thrashing
out the tiny details of *that* in *this* thread, that is not what it's
for. I had a whole paragraph about the possibility of an override
mechanism for maintainers which I left out precisely in order to avoid
this kind of discussion, but apparently that wasn't enough...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On 27/03/10 04:12, Adam Williamson wrote:
> On Sat, 2010-03-27 at 03:50 +0100, Kevin Kofler wrote:
>
>> So I don't think blocking an update outright for having received type 2
>> feedback is sane at all.
>
> Sigh. That was why I said not to sidetrack the discussion because it was
> the least important bit of the post. It was just an example of how
> easily the policy can be adapted. I'm really not interested in thrashing
> out the tiny details of *that* in *this* thread, that is not what it's
> for. I had a whole paragraph about the possibility of an override
> mechanism for maintainers which I left out precisely in order to avoid
> this kind of discussion, but apparently that wasn't enough...

I'm not sure if your usage policy covers changes to Bodhi, but how about
the system emailing the upstream developers (direct and/or email lists)
when a release is made available for testing/release and also on any
problems found ?

Terry
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, 2010-03-27 at 08:19 +0000, Terry Barnaby wrote:

> I'm not sure if your usage policy covers changes to Bodhi, but how about
> the system emailing the upstream developers (direct and/or email lists)
> when a release is made available for testing/release and also on any
> problems found ?

That's not really part of this proposal. It's a reasonable feature
request for Bodhi, though. You could file it with the Bodhi developers
(I think infrastructure trac -
https://fedorahosted.org/fedora-infrastructure/ - is the appropriate
venue).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Terry Barnaby <terry1@beam.ltd.uk> writes:
> I'm not sure if your usage policy covers changes to Bodhi, but how about
> the system emailing the upstream developers (direct and/or email lists)
> when a release is made available for testing/release and also on any
> problems found ?

As an upstream developer, I can hardly think of a quicker way to piss me
off than if every distro were to start sending me such nagmail. I would
not want such a thing turned on on *any* of my packages.

regards, tom lane
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Sat, 2010-03-27 at 08:19 +0000, Terry Barnaby wrote:

>
> I'm not sure if your usage policy covers changes to Bodhi, but how about
> the system emailing the upstream developers (direct and/or email lists)
> when a release is made available for testing/release and also on any
> problems found ?

I would be _very_ careful with sending autogenerated spam to random
people that have not signed up for it. I would expect many people to
react negatively to it.

Sending a small note along the lines of 'Hey, I'm packaging your
software for Fedora. How about you mention on the website that it is not
only available in Ubuntu but also in Fedora' is a quite different
thing, and should be done.

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Fri, Mar 26, 2010 at 03:49:28PM -0700, Adam Williamson wrote:

> 1. I have tried this update in my regular day-to-day use and seen no
> regressions.
>
> 2. I have tried this update in my regular day-to-day use and seen a
> regression: bug #XXXXXX.
>
> 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does fix bug #XXXXXX.
>
> 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does not fix bug #XXXXXX.
>
> 5. I have performed the following planned testing on the update: (link
> to test case / test plan) and it passes.
>
> 6. I have performed the following planned testing on the update: (link
> to test case / test plan) and it fails: bug #XXXXXX.

I have some additions:

7. fixes bug X, but does not claim to fix it

This can often happen with hardware related bugs, e.g. with the kernel
where something starts to work again

8. The package updated sucessfully, but was not used intentionally. No
breakage noticed.

This shows, that at least on the test machine, there are no broken deps,
conflicts or broken scriptlets.

Also it would be nice to provide hardware testing feedback, e.g. for Xorg
updates to say "Works with nouveau, Geforce XY, using VGA out and XV",
which then shows that e.g. 3D support, DVI out or multi screen support
was not tested. This is kind of related to testing with a test plan, but
having this data available in a format that can be easily parsed, would
be nice, too. Maybe this could be done with adding smolt information in
the feedback and the tested features (XV, VGA, DVI, 3D, ...) and the
update needs to have some meta data, which kind of devices are supported
(e.g. only Geforce devices for the nouveau driver package).

Regards
Till
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel