FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Development

 
 
LinkBack Thread Tools
 
Old 04-27-2011, 12:32 PM
Tim Niemueller
 
Default Avoiding conditioning ignorance towards AutoQA

Hi fellow Fedorans.

Recently, AutoQA has been introduced to catch typical problems early in
the update process. In general, I appreciate that effort, but currently
I find myself in a phase of conditioning ignorance towards AutoQA,
essentially because it is drowning me in irrelevant information. The
current case why I'm writing is
https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15.

Here are two ideas to make AutoQA relevant, less time-consuming, and
more helpful. In short: good QA is always quiet, only if there is a
problem it communicates.

- Post only errors
It is common, for example, in automated build or continuous integration
systems to send out emails only on errors. Similar goes for Unix tools,
which tend to be quiet if everything is ok, and only bother you with
output if something is not. Therefore, I propose to have AutoQA messages
posted only in case that there has been an error.

- Accumulate error messages
An email is sent for every single comment to Bodhi. In the case of
AutoQA, it causes one email per platform. It increases the load of email
tremendously to deal with, which in turn makes me ignore it. Therefore,
I propose to accumulate messages for all platforms. Combined with the
earlier proposal, the states for all platforms should be collected by an
intermediate node, and if and only if a test failed on any of the
platforms, one message with all status messages is posted to the update.

On a related note: it'd be much appreciated if Bodhi would provide an
option to get a daily digest with all comments of all the packages I'm
involved with.

I hope the fine folks of the AutoQA effort take these proposals into
account when proceeding in the development of the system and help me to
stop ignorance from taking over.

Regards,
Tim

--
Tim Niemueller <tim@niemueller.de> www.niemueller.de
================================================== ===============
Imagination is more important than knowledge. (Albert Einstein)
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-27-2011, 01:38 PM
Kamil Paral
 
Default Avoiding conditioning ignorance towards AutoQA

> Hi fellow Fedorans.
>
> Recently, AutoQA has been introduced to catch typical problems early
> in
> the update process. In general, I appreciate that effort, but
> currently
> I find myself in a phase of conditioning ignorance towards AutoQA,
> essentially because it is drowning me in irrelevant information. The
> current case why I'm writing is
> https://admin.fedoraproject.org/updates/lua-wsapi-1.3.4-4.fc15.
>
> Here are two ideas to make AutoQA relevant, less time-consuming, and
> more helpful. In short: good QA is always quiet, only if there is a
> problem it communicates.
>
> - Post only errors
> It is common, for example, in automated build or continuous
> integration
> systems to send out emails only on errors. Similar goes for Unix
> tools,
> which tend to be quiet if everything is ok, and only bother you with
> output if something is not. Therefore, I propose to have AutoQA
> messages
> posted only in case that there has been an error.

How can you then distinguish an update for which the tests have passed from an update for which the tests haven't yet been executed?

Moreover, currently not all updates are tested. Sometimes our tests simply don't work properly. Not just the updates are being tested, the whole AutoQA is being tested (and developed) in this whole effort.

>
> - Accumulate error messages
> An email is sent for every single comment to Bodhi. In the case of
> AutoQA, it causes one email per platform. It increases the load of
> email
> tremendously to deal with, which in turn makes me ignore it.
> Therefore,
> I propose to accumulate messages for all platforms.

We have that in plan, believe me.

> Combined with the
> earlier proposal, the states for all platforms should be collected by
> an
> intermediate node, and if and only if a test failed on any of the
> platforms, one message with all status messages is posted to the
> update.

Sending Bodhi comments is just a quick way how to inform the maintainers. We are working on a results database with API that other Fedora services (Koji, Bodhi) could query and use the results as they seem fit. For most tests I expect it will be similar to what you describe. But that's future. Until that's implemented we can only either send comments to Bodhi or send no comments at all.

>
> On a related note: it'd be much appreciated if Bodhi would provide an
> option to get a daily digest with all comments of all the packages I'm
> involved with.

Great idea, you can ask lmacken about that (or create ticket in its Trac).
Or, you can filter your emails and check the relevant folder once a day

>
> I hope the fine folks of the AutoQA effort take these proposals into
> account when proceeding in the development of the system and help me
> to
> stop ignorance from taking over.

It will take some time, but we see the deficiencies, same as you do.
We try to improve as fast as possible.

>
> Regards,
> Tim

Thanks,
Kamil


PS: We have a special mailing list for AutoQA:
https://fedorahosted.org/mailman/listinfo/autoqa-devel
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-27-2011, 07:03 PM
Tim Niemueller
 
Default Avoiding conditioning ignorance towards AutoQA

On 27.04.2011 15:38, Kamil Paral wrote:
>>
>> - Post only errors It is common, for example, in automated build or
>> continuous integration systems to send out emails only on errors.
>> Similar goes for Unix tools, which tend to be quiet if everything
>> is ok, and only bother you with output if something is not.
>> Therefore, I propose to have AutoQA messages posted only in case
>> that there has been an error.
>
> How can you then distinguish an update for which the tests have
> passed from an update for which the tests haven't yet been executed?
>
> Moreover, currently not all updates are tested. Sometimes our tests
> simply don't work properly. Not just the updates are being tested,
> the whole AutoQA is being tested (and developed) in this whole
> effort.

Please don't force testing, fixing, and maintaining AutoQA on the rest
of us. Integrate it such that stuff is pushed to testing only after
AutoQA has been run, or have a flag display "tests ran". Or post the
"PASSED" messages, but make Bodhi not sent messages in the case the
tests passed.

Keeping the current way will just make me (and possibly others) add
filters to throw away messages from AutoQA. Please be aware of how much
contributor time you waste by making them hope through useless (because
the tests have passed and no information is gained) mails. I realize you
want to improve things, but at the current stage its consuming the most
valuable resource we have, packager/developer time.


>> - Accumulate error messages An email is sent for every single
>
> We have that in plan, believe me.
>
>> Combined with the earlier proposal, the states for all platforms
>> should be collected by an intermediate node, and if and only if a
>> test failed on any of the platforms, one message with all status
>> messages is posted to the update.
>
> Sending Bodhi comments is just a quick way how to inform the
> maintainers. We are working on a results database with API that other
> Fedora services (Koji, Bodhi) could query and use the results as they
> seem fit. For most tests I expect it will be similar to what you
> describe. But that's future. Until that's implemented we can only
> either send comments to Bodhi or send no comments at all.

I understand you like to have a quick and working solution for now, and
that great stuff is coming. But you cost a lot of time right now. Please
reconsider to make your development and testing time less intrusive for
others.


>>
>> On a related note: it'd be much appreciated if Bodhi would provide
>> an option to get a daily digest with all comments of all the
>> packages I'm involved with.
>
> Great idea, you can ask lmacken about that (or create ticket in its
> Trac). Or, you can filter your emails and check the relevant folder
> once a day

That still means striving through many messages with lots of non-info
text. One concise email would make things much better.

>>
>> I hope the fine folks of the AutoQA effort take these proposals
>> into account when proceeding in the development of the system and
>> help me to stop ignorance from taking over.
>
> It will take some time, but we see the deficiencies, same as you do.
> We try to improve as fast as possible.

Please try to find a way to do this without costing as much time as atm.
There must be ways, for example have a list of packages to use it for
that packagers can opt-in and make AutoQA developers the first to use it.

> PS: We have a special mailing list for AutoQA:
> https://fedorahosted.org/mailman/listinfo/autoqa-devel

Sorry, to keep me involved in Fedora I have to make it a reasonable
effort, joining yet another project is out of my possibilities atm.

Tim

--
Tim Niemueller <tim@niemueller.de> www.niemueller.de
================================================== ===============
Imagination is more important than knowledge. (Albert Einstein)
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-27-2011, 09:00 PM
James Laska
 
Default Avoiding conditioning ignorance towards AutoQA

On Wed, 2011-04-27 at 21:03 +0200, Tim Niemueller wrote:
> On 27.04.2011 15:38, Kamil Paral wrote:
> >>
> >> - Post only errors It is common, for example, in automated build or
> >> continuous integration systems to send out emails only on errors.
> >> Similar goes for Unix tools, which tend to be quiet if everything
> >> is ok, and only bother you with output if something is not.
> >> Therefore, I propose to have AutoQA messages posted only in case
> >> that there has been an error.
> >
> > How can you then distinguish an update for which the tests have
> > passed from an update for which the tests haven't yet been executed?
> >
> > Moreover, currently not all updates are tested. Sometimes our tests
> > simply don't work properly. Not just the updates are being tested,
> > the whole AutoQA is being tested (and developed) in this whole
> > effort.
>
> Please don't force testing, fixing, and maintaining AutoQA on the rest
> of us.

Well, we don't force it, but like bodhi, koji and most other key
infrastructure tools, it is software ... and software unfortunately has
bugs. We do our best to minimize those bugs, but we've found it very
difficult to emulate a real-world bodhi+koji setup (with interesting
data) in our test instance.

> Integrate it such that stuff is pushed to testing only after
> AutoQA has been run, or have a flag display "tests ran". Or post the
> "PASSED" messages, but make Bodhi not sent messages in the case the
> tests passed.

Re: integration and only allowing updates to proceed to
'updates-testing' ... that's the goal. Unfortunately, as with many
things, arriving at a destination involves a journey. We can't simply
turn this "feature" on until we have confidence that the tests are
correctly enforcing the package update policy [1]. Therefore, we are
operating in a permissive mode at this time.

With regards to having a flag or having bodhi not send messages in
certain scenarios. Those are definitely options. We'll need to raise
those ideas the bodhi folks [2] for review, since AutoQA and bodhi are
separate projects. But your concerns are definitely worth following up
on.

[1] https://fedoraproject.org/wiki/Updates_Policy
[2] https://fedorahosted.org/mailman/listinfo/bodhi

> Keeping the current way will just make me (and possibly others) add
> filters to throw away messages from AutoQA. Please be aware of how much
> contributor time you waste by making them hope through useless (because
> the tests have passed and no information is gained) mails. I realize you
> want to improve things, but at the current stage its consuming the most
> valuable resource we have, packager/developer time.

> >> - Accumulate error messages An email is sent for every single
> >
> > We have that in plan, believe me.
> >
> >> Combined with the earlier proposal, the states for all platforms
> >> should be collected by an intermediate node, and if and only if a
> >> test failed on any of the platforms, one message with all status
> >> messages is posted to the update.
> >
> > Sending Bodhi comments is just a quick way how to inform the
> > maintainers. We are working on a results database with API that other
> > Fedora services (Koji, Bodhi) could query and use the results as they
> > seem fit. For most tests I expect it will be similar to what you
> > describe. But that's future. Until that's implemented we can only
> > either send comments to Bodhi or send no comments at all.
>
> I understand you like to have a quick and working solution for now, and
> that great stuff is coming. But you cost a lot of time right now. Please
> reconsider to make your development and testing time less intrusive for
> others.

Thanks for your feedback. We're actively working on ways to reduce the
unnecessary emails generated by bodhi when posting feedback. Obviously,
the goal is to work *for* maintainers, not against them.

> >> On a related note: it'd be much appreciated if Bodhi would provide
> >> an option to get a daily digest with all comments of all the
> >> packages I'm involved with.
> >
> > Great idea, you can ask lmacken about that (or create ticket in its
> > Trac). Or, you can filter your emails and check the relevant folder
> > once a day
>
> That still means striving through many messages with lots of non-info
> text. One concise email would make things much better.

The point Kamil was making is that bodhi and AutoQA are different tools.
If you have a good idea for bodhi, please raise that on
https://fedorahosted.org/mailman/listinfo/bodhi.

> >> I hope the fine folks of the AutoQA effort take these proposals
> >> into account when proceeding in the development of the system and
> >> help me to stop ignorance from taking over.
> >
> > It will take some time, but we see the deficiencies, same as you do.
> > We try to improve as fast as possible.
>
> Please try to find a way to do this without costing as much time as atm.
> There must be ways, for example have a list of packages to use it for
> that packagers can opt-in and make AutoQA developers the first to use it.

Opt-in support has been available for some time now. I probably could
communicate this better, but alas ...

https://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test-results/

> > PS: We have a special mailing list for AutoQA:
> > https://fedorahosted.org/mailman/listinfo/autoqa-devel
>
> Sorry, to keep me involved in Fedora I have to make it a reasonable
> effort, joining yet another project is out of my possibilities atm.

Understood. There are only so many hours in a day.

Thanks,
James
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-28-2011, 06:34 AM
Tim Flink
 
Default Avoiding conditioning ignorance towards AutoQA

On 04/27/2011 03:00 PM, James Laska wrote:
> On Wed, 2011-04-27 at 21:03 +0200, Tim Niemueller wrote:
>> On 27.04.2011 15:38, Kamil Paral wrote:
>>>>
>>>> - Post only errors It is common, for example, in automated build or
>>>> continuous integration systems to send out emails only on errors.
>>>> Similar goes for Unix tools, which tend to be quiet if everything
>>>> is ok, and only bother you with output if something is not.
>>>> Therefore, I propose to have AutoQA messages posted only in case
>>>> that there has been an error.
>>>
>>> How can you then distinguish an update for which the tests have
>>> passed from an update for which the tests haven't yet been executed?
>>>
>>> Moreover, currently not all updates are tested. Sometimes our tests
>>> simply don't work properly. Not just the updates are being tested,
>>> the whole AutoQA is being tested (and developed) in this whole
>>> effort.
>>
>> Please don't force testing, fixing, and maintaining AutoQA on the rest
>> of us.
>
> Well, we don't force it, but like bodhi, koji and most other key
> infrastructure tools, it is software ... and software unfortunately has
> bugs. We do our best to minimize those bugs, but we've found it very
> difficult to emulate a real-world bodhi+koji setup (with interesting
> data) in our test instance.
>
>> Integrate it such that stuff is pushed to testing only after
>> AutoQA has been run, or have a flag display "tests ran". Or post the
>> "PASSED" messages, but make Bodhi not sent messages in the case the
>> tests passed.
>
> Re: integration and only allowing updates to proceed to
> 'updates-testing' ... that's the goal. Unfortunately, as with many
> things, arriving at a destination involves a journey. We can't simply
> turn this "feature" on until we have confidence that the tests are
> correctly enforcing the package update policy [1]. Therefore, we are
> operating in a permissive mode at this time.
>
> With regards to having a flag or having bodhi not send messages in
> certain scenarios. Those are definitely options. We'll need to raise
> those ideas the bodhi folks [2] for review, since AutoQA and bodhi are
> separate projects. But your concerns are definitely worth following up
> on.
>
> [1] https://fedoraproject.org/wiki/Updates_Policy
> [2] https://fedorahosted.org/mailman/listinfo/bodhi
>
>> Keeping the current way will just make me (and possibly others) add
>> filters to throw away messages from AutoQA. Please be aware of how much
>> contributor time you waste by making them hope through useless (because
>> the tests have passed and no information is gained) mails. I realize you
>> want to improve things, but at the current stage its consuming the most
>> valuable resource we have, packager/developer time.
>
>>>> - Accumulate error messages An email is sent for every single
>>>
>>> We have that in plan, believe me.
>>>
>>>> Combined with the earlier proposal, the states for all platforms
>>>> should be collected by an intermediate node, and if and only if a
>>>> test failed on any of the platforms, one message with all status
>>>> messages is posted to the update.
>>>
>>> Sending Bodhi comments is just a quick way how to inform the
>>> maintainers. We are working on a results database with API that other
>>> Fedora services (Koji, Bodhi) could query and use the results as they
>>> seem fit. For most tests I expect it will be similar to what you
>>> describe. But that's future. Until that's implemented we can only
>>> either send comments to Bodhi or send no comments at all.
>>
>> I understand you like to have a quick and working solution for now, and
>> that great stuff is coming. But you cost a lot of time right now. Please
>> reconsider to make your development and testing time less intrusive for
>> others.
>
> Thanks for your feedback. We're actively working on ways to reduce the
> unnecessary emails generated by bodhi when posting feedback. Obviously,
> the goal is to work *for* maintainers, not against them.

I second the thanks for feedback. I'm not promising that we can please
everyone all of the time, but it's harder for us to fix pain points if
we don't know where they are. Like most developers, we like constructive
feedback!

>>>> On a related note: it'd be much appreciated if Bodhi would provide
>>>> an option to get a daily digest with all comments of all the
>>>> packages I'm involved with.
>>>
>>> Great idea, you can ask lmacken about that (or create ticket in its
>>> Trac). Or, you can filter your emails and check the relevant folder
>>> once a day
>>
>> That still means striving through many messages with lots of non-info
>> text. One concise email would make things much better.
>
> The point Kamil was making is that bodhi and AutoQA are different tools.
> If you have a good idea for bodhi, please raise that on
> https://fedorahosted.org/mailman/listinfo/bodhi.
>
>>>> I hope the fine folks of the AutoQA effort take these proposals
>>>> into account when proceeding in the development of the system and
>>>> help me to stop ignorance from taking over.
>>>
>>> It will take some time, but we see the deficiencies, same as you do.
>>> We try to improve as fast as possible.
>>
>> Please try to find a way to do this without costing as much time as atm.
>> There must be ways, for example have a list of packages to use it for
>> that packagers can opt-in and make AutoQA developers the first to use it.
>
> Opt-in support has been available for some time now. I probably could
> communicate this better, but alas ...
>
> https://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test-results/

Correct me if I'm wrong, but I don't think that he was talking about the
raw results from AutoQA but the notification emails that get sent from
Bodhi as a result of the comments made by AutoQA.

>>> PS: We have a special mailing list for AutoQA:
>>> https://fedorahosted.org/mailman/listinfo/autoqa-devel
>>
>> Sorry, to keep me involved in Fedora I have to make it a reasonable
>> effort, joining yet another project is out of my possibilities atm.

Again, I could be wrong here but I do believe that Kamil's suggestion
was to cc: autoqa-devel@lists.fedorahosted.org on topics related to
AutoQA to receive a faster response. It is by no means a requirement but
I know that I only check devel@ a couple of times a day. On the other
hand, if something shows up on the autoqa-devel list I read it as soon
as I notice it. Either way works, though.

> Understood. There are only so many hours in a day.
>
> Thanks,
> James
>


--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-28-2011, 08:00 AM
Kamil Paral
 
Default Avoiding conditioning ignorance towards AutoQA

> Keeping the current way will just make me (and possibly others) add
> filters to throw away messages from AutoQA. Please be aware of how
> much
> contributor time you waste by making them hope through useless
> (because
> the tests have passed and no information is gained) mails. I realize
> you
> want to improve things, but at the current stage its consuming the
> most
> valuable resource we have, packager/developer time.

We will certainly improve that. But for the near future I don't see much ways to change the notification system (using Bodhi comments). We can either have it the way it is or disable it completely (we can't use opting in/out, we must operate on all updates or none of them).

I believe the current state is good enough to have AutoQA comments enabled, even though it's not perfect. You may disagree. If many maintainers claim they would rather have AutoQA completely disabled, we can do it. But the current state helps us fix the tests tremendously. In the mean time, we work on improving the AutoQA architecture.

As a simple workaround for you (and maybe for some others), why don't you define a simple mail filter that detects an AutoQA Bodhi comment and marks it as read if it contains PASSED comment? That will save you from wasting time reading them.

Another option is to work with Bodhi authors to implement your ideas. It will be great if you do.
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 04-28-2011, 04:59 PM
Tim Flink
 
Default Avoiding conditioning ignorance towards AutoQA

On 04/28/2011 02:00 AM, Kamil Paral wrote:
>> Keeping the current way will just make me (and possibly others)
>> add filters to throw away messages from AutoQA. Please be aware of
>> how much contributor time you waste by making them hope through
>> useless (because the tests have passed and no information is
>> gained) mails. I realize you want to improve things, but at the
>> current stage its consuming the most valuable resource we have,
>> packager/developer time.
>
> We will certainly improve that. But for the near future I don't see
> much ways to change the notification system (using Bodhi comments).
> We can either have it the way it is or disable it completely (we
> can't use opting in/out, we must operate on all updates or none of
> them).
>
> I believe the current state is good enough to have AutoQA comments
> enabled, even though it's not perfect. You may disagree. If many
> maintainers claim they would rather have AutoQA completely disabled,
> we can do it. But the current state helps us fix the tests
> tremendously. In the mean time, we work on improving the AutoQA
> architecture.
>
> As a simple workaround for you (and maybe for some others), why don't
> you define a simple mail filter that detects an AutoQA Bodhi comment
> and marks it as read if it contains PASSED comment? That will save
> you from wasting time reading them.
>
> Another option is to work with Bodhi authors to implement your ideas.
> It will be great if you do.

To add a little more detail, we are working on this [1], but I don't
think that any fix will make it into the production systems for at least
a week or two.

Until that happens, the best workaround that we can suggest is to use an
email filter like Kamil suggested.

Again, thanks for your feedback and patience.

Tim

[1] https://fedorahosted.org/autoqa/ticket/314

--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 
Old 05-03-2011, 08:20 AM
Tim Niemueller
 
Default Avoiding conditioning ignorance towards AutoQA

On 28.04.2011 18:59, Tim Flink wrote:
> On 04/28/2011 02:00 AM, Kamil Paral wrote:
>>> Keeping the current way will just make me (and possibly others)
>>> add filters to throw away messages from AutoQA. Please be aware of
>>> how much contributor time you waste by making them hope through
>>> useless (because the tests have passed and no information is
>>> gained) mails. I realize you want to improve things, but at the
>>> current stage its consuming the most valuable resource we have,
>>> packager/developer time.
>>
>> We will certainly improve that. But for the near future I don't see
>> much ways to change the notification system (using Bodhi comments).
>> We can either have it the way it is or disable it completely (we
>> can't use opting in/out, we must operate on all updates or none of
>> them).
>>
>> I believe the current state is good enough to have AutoQA comments
>> enabled, even though it's not perfect. You may disagree. If many
>> maintainers claim they would rather have AutoQA completely disabled,
>> we can do it. But the current state helps us fix the tests
>> tremendously. In the mean time, we work on improving the AutoQA
>> architecture.
>>
>> As a simple workaround for you (and maybe for some others), why don't
>> you define a simple mail filter that detects an AutoQA Bodhi comment
>> and marks it as read if it contains PASSED comment? That will save
>> you from wasting time reading them.
>>
>> Another option is to work with Bodhi authors to implement your ideas.
>> It will be great if you do.
>
> To add a little more detail, we are working on this [1], but I don't
> think that any fix will make it into the production systems for at least
> a week or two.
>
> Until that happens, the best workaround that we can suggest is to use an
> email filter like Kamil suggested.
>
> Again, thanks for your feedback and patience.

Thanks for taking this into consideration and starting work on it, it's
really appreciated and at least in my case will help to raise acceptance
and willingness to accept AutoQA.

Tim

--
Tim Niemueller <tim@niemueller.de> www.niemueller.de
================================================== ===============
Imagination is more important than knowledge. (Albert Einstein)
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
 

Thread Tools




All times are GMT. The time now is 04:43 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org