Luke Macken wrote:
> This report definitely conveys the shortcomings in our testing, however,
> it does show us improving with each release. For Fedora 13, we implemented
> the No Frozen Rawhide process with improved Critical Path policies, which
> were definitely a success. With these enhanced procedures, along with the
> upcoming implementation of AutoQA and the new Package update acceptance
> criteria
> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
> think we'll see these numbers drastically improve in the future.

Only because those numbers are taylored towards that very process (they
measure the exact same things that process is going to enforce) and do not
reflect the actual quality of the packages in any way.

You can make really anything a "success" by measuring the very symptoms of
the process and calling them a metric of "quality".

The reasons for which Bodhi karma (especially in its current incarnation) is
a completely broken indicator of quality have been pointed out in several
past threads.

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, Jun 8, 2010 at 7:46 PM, Kevin Kofler <kevin.kofler@chello.at> wrote:
> Luke Macken wrote:
>> This report definitely conveys the shortcomings in our testing, however,
>> it does show us improving with each release. For Fedora 13, we implemented
>> the No Frozen Rawhide process with improved Critical Path policies, which
>> were definitely a success. With these enhanced procedures, along with the
>> upcoming implementation of AutoQA and the new Package update acceptance
>> criteria
>> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
>> think we'll see these numbers drastically improve in the future.
>
> Only because those numbers are taylored towards that very process (they
> measure the exact same things that process is going to enforce) and do not
> reflect the actual quality of the packages in any way.
>
> You can make really anything a "success" by measuring the very symptoms of
> the process and calling them a metric of "quality".
>
> The reasons for which Bodhi karma (especially in its current incarnation) is
> a completely broken indicator of quality have been pointed out in several
> past threads.
>
> * * * *Kevin Kofler
>
> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
>

I'll have to agree with Kevin. I can't how any of those numbers
represent the quality of anything.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:

> ================================================== ==============================
> Fedora 13
> ================================================== ==============================
>
> * 3562 updates
> * 3065 stable updates
> * 427 testing updates
> * 62 pending updates
> * 8 obsolete updates

Hey Luke,

are these the numbers of F13 updates since we branched, or since the F13
release ? The numbers certainly look like the former, but I think the
latter would be much more interesting.


Matthias

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, Jun 8, 2010 at 6:41 PM, Brandon Lozza <brandon@pwnage.ca> wrote:
> On Tue, Jun 8, 2010 at 7:46 PM, Kevin Kofler <kevin.kofler@chello.at> wrote:
>> Luke Macken wrote:
>>> This report definitely conveys the shortcomings in our testing, however,
>>> it does show us improving with each release. For Fedora 13, we implemented
>>> the No Frozen Rawhide process with improved Critical Path policies, which
>>> were definitely a success. With these enhanced procedures, along with the
>>> upcoming implementation of AutoQA and the new Package update acceptance
>>> criteria
>>> (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
>>> think we'll see these numbers drastically improve in the future.
>>
>> Only because those numbers are taylored towards that very process (they
>> measure the exact same things that process is going to enforce) and do not
>> reflect the actual quality of the packages in any way.
>>
>> You can make really anything a "success" by measuring the very symptoms of
>> the process and calling them a metric of "quality".
>>
>> The reasons for which Bodhi karma (especially in its current incarnation) is
>> a completely broken indicator of quality have been pointed out in several
>> past threads.

>
> I'll have to agree with Kevin. I can't how any of those numbers
> represent the quality of anything.

Well the only person I see mentioning quality is Kevin. And for some
reason he is expecting it immediately when Luke said 3 things:

1) This was what he could initially see.
2) He thinks he sees improvement of what was measured, but he is
interested in what would be better indicators.

> --
> devel mailing list
> devel@lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/devel
>



--
Stephen J Smoogen.
“The core skill of innovators is error recovery, not failure avoidance.�
Randy Nelson, President of Pixar University.
"We have a strategic plan. It's called doing things.""
— Herb Kelleher, founder Southwest Airlines
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Stephen John Smoogen wrote:
> Well the only person I see mentioning quality is Kevin. And for some
> reason he is expecting it immediately

You can't claim that there are "shortcomings in our testing" (the exact
words Luke used!) without a metric of quality. A shortcoming in testing
means less testing was done than needed. If the packages have good quality,
that means more testing is NOT needed, no matter what the actual amount of
testing was.

In addition, Luke is speculating about those numbers "improving" with
upcoming policy changes. But the word "improve" only makes sense for a
quantity which measures some kind of quality. Pure numbers without a
judgement of value can only "increase" or "decrease", not "improve".

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On 06/09/2010 10:48 AM, Kevin Kofler wrote:
> Stephen John Smoogen wrote:
>
>> Well the only person I see mentioning quality is Kevin. And for some
>> reason he is expecting it immediately
>>
> You can't claim that there are "shortcomings in our testing" (the exact
> words Luke used!) without a metric of quality. A shortcoming in testing
> means less testing was done than needed. If the packages have good quality,
> that means more testing is NOT needed, no matter what the actual amount of
> testing was.
>

Packages without any testing is of unknown quality. If someone doesn't
test them before pushing, users are unknowingly playing that role.

Rahul
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Wed, 2010-06-09 at 01:46 +0200, Kevin Kofler wrote:
> Luke Macken wrote:
> > This report definitely conveys the shortcomings in our testing, however,
> > it does show us improving with each release.

By 'shortcomings in our testing', I mean, 'shortcomings in the process
by which we currently use bodhi for "testing" updates'.

> For Fedora 13, we implemented
> > the No Frozen Rawhide process with improved Critical Path policies, which
> > were definitely a success. With these enhanced procedures, along with the
> > upcoming implementation of AutoQA and the new Package update acceptance
> > criteria
> > (https://fedoraproject.org/wiki/Package_update_acceptance_criteria), I
> > think we'll see these numbers drastically improve in the future.
>
> Only because those numbers are taylored towards that very process (they
> measure the exact same things that process is going to enforce) and do not
> reflect the actual quality of the packages in any way.
>
> You can make really anything a "success" by measuring the very symptoms of
> the process and calling them a metric of "quality".

These metrics do not reflect the actual quality of our updates, the
policy or process behind them, or even the quality of testing that they
endured before being released. It is simply a measurement of how we
have been interacting with this specific piece of infrastructure.

By "success" I mean that I felt we were successful in drafting,
implementing, deploying, and utilizing the mentioned policies as
expected, and the results show increased community engagement.
Attempting to quantify the quality of these community interactions is
another story. As to whether or not the policies behind the processes
have helped or hindered the release quality overall, that has yet to be
determined.

> The reasons for which Bodhi karma (especially in its current incarnation) is
> a completely broken indicator of quality have been pointed out in several
> past threads.

I'm well aware, and I agree that the current bodhi karma implementation
does not convey an effective measurement of update quality. The problem
is not a lack of good ideas for improvement, but rather a lack of
developers to help improve.

luke

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
> ================================================== ==============================
> Fedora 13
> ================================================== ==============================
>
> * 231 updates automatically pushed due to karma (6.49%)
> * 2 of which were critical path updates

So I thought that this last metric would take into account critical path
updates that were approved under the new policy, but it did not. The
actual metric is for Fedora 13 is:

* 333 critical path updates approved


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

On Tue, 2010-06-08 at 21:20 -0400, Matthias Clasen wrote:
> On Tue, 2010-06-08 at 16:51 -0400, Luke Macken wrote:
>
> > ================================================== ==============================
> > Fedora 13
> > ================================================== ==============================
> >
> > * 3562 updates
> > * 3065 stable updates
> > * 427 testing updates
> > * 62 pending updates
> > * 8 obsolete updates
>
> Hey Luke,
>
> are these the numbers of F13 updates since we branched, or since the F13
> release ? The numbers certainly look like the former, but I think the
> latter would be much more interesting.

These numbers are since we branched for F13.

luke

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel

Luke Macken wrote:
> By "success" I mean that I felt we were successful in drafting,
> implementing, deploying, and utilizing the mentioned policies as
> expected, and the results show increased community engagement.

This definition of "success" does not match mine nor the one you'll find in
a dictionary. So your terminology is misleading.

Kevin Kofler

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel