Increasing user testing and feedback (was Bad-mouthing and hostility)

On Wed 10 March 2010 7:52:11 am Tom "spot" Callaway wrote:
> On 03/10/2010 09:48 AM, Matt Domsch wrote:
> > We're complaining that we can't find 3 users of a single package to
> > provide feedback.
>
> To be fair, I would restate this as:
>
> We cannot find 3 users of a package to:
>
> * Independently know that an update needs to be tested (is sitting in
> updates-testing)
> * Feel comfortable that they are qualified to test it
> * Download and install the update that needs testing (this assumes they
> know where to get it)
> * Perform "testing" on that package, when this is loosely defined at best.
> * Find the Bodhi page for this package update.
> * Create a FAS account if they don't have one.
> * Go back to the Bodhi page for this package update and figure out that
> "Leave a Comment" means "give karma so this package can move from
> updates-testing to updates".
> * Leave a comment.
>
> Gets even uglier when the package isn't an application. How does a user
> know that a library is worth giving a karma vote on? "It installed." "My
> computer didn't catch fire." "Linux still boots."
>
> ~spot

Well said Tom!

Can I put forth a suggestion to ease this process for contributors (and
users!) to easily contribute to this testing, and make it as easily as
possible for applications and libraries to have themselves "heard."

The creation of a Fedora-updates-testing mailing list.
* Mailing list would be prominetely placed on the wiki or main fedoraproject
website "Hey, want to help? Have a few free minutes? Looking for a fix in
your favorite package? test!" or something similar.
* The "welcome" message that mailman sends for new subscribers to a mailing
list would lay out exactly what it is the mailing list does, and how they can
help. "Please spend five minutes registering a Fedora Account Systems user
name for submitting karma. If you can't do this, your testing is basically a
waste of time as you can't submit karma" (but put it much nicer!) Install
this python script so that you can more easily submit karma.
** Messages will be posted to this list by package owners requestings testing
for an update sitting in updates-testing (or koji, if more approiate).
** Package owners list _user_facing_ changes and how they can test them,
reference bugzilla URLs if necessary.
** If a user has any interest in testing this package, they can download it
either using this yum command, or at this koji url and install it with this
yum command.
** If the update works run _this_ command to update the karma or _this_
command to downgrade the karma and leave a comment as to why.

By creating such a list and differentiating it from the current QA list, we
can attract people who may not usually test software. By putting everything
right there on the mailing list we can lower the entry barrier significantly
as well.. If there is any interest, I can write a full proposal with some
help this weekend.

Just an idea,
Ryan

--
Ryan Rix
== http://hackersramblings.wordpress.com | http://rix.si/ ==
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 03/11/2010 12:29 AM, Ryan Rix wrote:
> Can I put forth a suggestion to ease this process for contributors (and
> users!) to easily contribute to this testing, and make it as easily as
> possible for applications and libraries to have themselves "heard."
>
> The creation of a Fedora-updates-testing mailing list.

First of all, thank you for proposing a solution. Now, for my
constructive criticism:

* I'm not sure yet-another-mailing-list will go far enough to solve the
problem. Even if it is placed prominently on the wiki or on the main
fedoraproject website.

A mailing list is a flood of email that a user has to really really look
at and keep up with, then proceed to a manual process. I'd argue that
the majority of community members willing to watch a mailing list for
updates that need testing are already doing so on the existing testing
mailing list.

I posit an alternative suggestion:

* At firstboot, the installing user is asked if they would be willing to
participate in user-driven updates testing. It is explained to them that
in Fedora, updates to packages need to be tested by users, and that if
they opt-in, they will be prompted from PackageKit about updates which
need user testing. They can choose an update which needs testing from a
list. Once an update is selected from the list, PackageKit will apply
the update from updates-testing, then open a new window which contains:

* General update testing advice
* Package specific update testing advice (this can live on the wiki)
* A graphical selector for giving +1 (works great!), 0 (cannot determine
state) or -1 (something didn't work)
* A text box for inputting comments

The user then submits the results, which go into Bodhi. Once results are
submitted, that update no longer appears in the PackageKit "updates
which need testing" list.

If they report a 0 or -1, they are then prompted to back out the update
by PackageKit (at their choice).

* On the backend, should a user choose to opt-in, they would be prompted
to create a FAS account (or authenticate to an existing FAS account)
(e.g. RHN handling in the past). They would _NOT_ be required to sign
the Fedora CLA in order to participate in user-driven testing, as
reported results from QA testing has already been determined to be
non-copyrightable and thus, not considered a contribution.

Each user who opts-in to perform user-driven testing will have it
flagged in their account. Each successful update testing submission will
be minimally logged (package, target, timedate stamp) and a count
incremented for unique update feedback performed.

In thanks for their testing, users will be informed (at firstboot) that
they will receive Fedora swag, both in random drawings and at certain
threshold points (give good feedback on N updates and get a Fedora
Tester T-shirt).

Users can choose to opt out at any time.

*****

Yes, it's more coding change than a mailing list, but I think it will
gather the attention of a lot more potential testers. What it will also
need is the efforts of existing QA contributors in the Fedora community
to help draft the general update testing advice and work with
maintainers (and upstreams) to create package specific update testing
advice.

Note that this goes beyond the "things we can test in an automated
fashion", I'm not looking for this as a mechanism for our users to
mindlessly run "rpmlint" for us, but instead as a mechanism to perform
human testing which cannot be easily automated. It also gives our users
a low-impact way to be involved with Fedora.

~spot


_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Thu, Mar 11, 2010 at 08:39:45AM -0500, Tom spot Callaway wrote:
> On 03/11/2010 12:29 AM, Ryan Rix wrote:
> > Can I put forth a suggestion to ease this process for contributors (and
> > users!) to easily contribute to this testing, and make it as easily as
> > possible for applications and libraries to have themselves "heard."
> >
> > The creation of a Fedora-updates-testing mailing list.
>
> First of all, thank you for proposing a solution. Now, for my
> constructive criticism:
>
> * I'm not sure yet-another-mailing-list will go far enough to solve the
> problem. Even if it is placed prominently on the wiki or on the main
> fedoraproject website.
>
> A mailing list is a flood of email that a user has to really really look
> at and keep up with, then proceed to a manual process. I'd argue that
> the majority of community members willing to watch a mailing list for
> updates that need testing are already doing so on the existing testing
> mailing list.

I agree -- mailing lists are a higher barrier tool than some sort of
GUI helper that's well integrated with the installation of Fedora.

> I posit an alternative suggestion:
>
> * At firstboot, the installing user is asked if they would be willing to
> participate in user-driven updates testing. It is explained to them that
> in Fedora, updates to packages need to be tested by users, and that if
> they opt-in, they will be prompted from PackageKit about updates which
> need user testing. They can choose an update which needs testing from a
> list. Once an update is selected from the list, PackageKit will apply
> the update from updates-testing, then open a new window which contains:
>
> * General update testing advice
> * Package specific update testing advice (this can live on the wiki)
> * A graphical selector for giving +1 (works great!), 0 (cannot determine
> state) or -1 (something didn't work)
> * A text box for inputting comments
>
> The user then submits the results, which go into Bodhi. Once results are
> submitted, that update no longer appears in the PackageKit "updates
> which need testing" list.
>
> If they report a 0 or -1, they are then prompted to back out the update
> by PackageKit (at their choice).
>
> * On the backend, should a user choose to opt-in, they would be prompted
> to create a FAS account (or authenticate to an existing FAS account)
> (e.g. RHN handling in the past). They would _NOT_ be required to sign
> the Fedora CLA in order to participate in user-driven testing, as
> reported results from QA testing has already been determined to be
> non-copyrightable and thus, not considered a contribution.
>
> Each user who opts-in to perform user-driven testing will have it
> flagged in their account. Each successful update testing submission will
> be minimally logged (package, target, timedate stamp) and a count
> incremented for unique update feedback performed.
>
> In thanks for their testing, users will be informed (at firstboot) that
> they will receive Fedora swag, both in random drawings and at certain
> threshold points (give good feedback on N updates and get a Fedora
> Tester T-shirt).
>
> Users can choose to opt out at any time.

It's a brilliant idea and also gets us toward one of the concepts
you've proposed in the past, which is a sort of "bonus point" system
for contributors where we can give additional rewards to people. Some
people don't require those types of rewards to participate, others
will find it delightful. Providing small incentives for people to
contribute is a good way to cement a growing relationship with the
Fedora Project.

> *****
>
> Yes, it's more coding change than a mailing list, but I think it will
> gather the attention of a lot more potential testers. What it will also
> need is the efforts of existing QA contributors in the Fedora community
> to help draft the general update testing advice and work with
> maintainers (and upstreams) to create package specific update testing
> advice.
>
> Note that this goes beyond the "things we can test in an automated
> fashion", I'm not looking for this as a mechanism for our users to
> mindlessly run "rpmlint" for us, but instead as a mechanism to perform
> human testing which cannot be easily automated. It also gives our users
> a low-impact way to be involved with Fedora.

Is this something that would make sense for a summer coding project?

--
Paul W. Frields http://paul.frields.org/
gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717
http://redhat.com/ - - - - http://pfrields.fedorapeople.org/
Where open source multiplies: http://opensource.com
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

Tom spot Callaway (tcallawa@redhat.com) said:
> I posit an alternative suggestion:
>
> * At firstboot, the installing user is asked if they would be willing to
> participate in user-driven updates testing. It is explained to them that
> in Fedora, updates to packages need to be tested by users, and that if
> they opt-in, they will be prompted from PackageKit about updates which
> need user testing. They can choose an update which needs testing from a
> list. Once an update is selected from the list, PackageKit will apply
> the update from updates-testing, then open a new window which contains:
>
> * General update testing advice
> * Package specific update testing advice (this can live on the wiki)
> * A graphical selector for giving +1 (works great!), 0 (cannot determine
> state) or -1 (something didn't work)
> * A text box for inputting comments
>
> The user then submits the results, which go into Bodhi. Once results are
> submitted, that update no longer appears in the PackageKit "updates
> which need testing" list.
>
> If they report a 0 or -1, they are then prompted to back out the update
> by PackageKit (at their choice).
>
> * On the backend, should a user choose to opt-in, they would be prompted
> to create a FAS account (or authenticate to an existing FAS account)
> (e.g. RHN handling in the past). They would _NOT_ be required to sign
> the Fedora CLA in order to participate in user-driven testing, as
> reported results from QA testing has already been determined to be
> non-copyrightable and thus, not considered a contribution.

I'd genericize this. On firstboot, *EVERY* user should be asked if
they want to create a fedora account. It would describe the privacy
policy (in brief, with a link), describe the things that can be done
(comment on updates, file bugs[*], respond to surveys, participate
in leadership, other stuff here.) They'd then enter their proposed
account name (which would do an online check), an e-mail address, and
a password. It's sort of a 'register your product' approximation.

Then, whatever update-testing tool you create can just use this existing
infrastructure - I wouldn't gate the account registration on just wanting
to do updates-testing. Heck, you could extend this into the initial
'create a user' dialog.

Bill
[*] The idea being that with an e-mail + password, you can *also* register
for a bugzilla account at the same time. Woo, faked single-sign on!
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 03/11/2010 10:52 AM, Colin Walters wrote:
> And actually how the heck would this even work at a technical level?
> There's no networking in firstboot because NetworkManager isn't set
> up.
>

Sure there is. How do you think we're doing smolt now?

~spot
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Thu, 11 Mar 2010, Colin Walters wrote:

> On Thu, Mar 11, 2010 at 10:31 AM, Bill Nottingham <notting@redhat.com> wrote:
> >
> > I'd genericize this. On firstboot, *EVERY* user should be asked if
> > they want to create a fedora account.
>
> This needs to go through the Fedora Design group.
>
> I'd beg to please, pretty please, don't add stuff to firstboot; we
> need to get people to a functioning desktop as the primary goal.

Random thought: We could put a pretty ajaxy signup thing on
start.fedoraproject.org that keeps them logged in via community.

I'm thinking of something similar to how google.com works now. With no
login you get the generic pageg. With a login you might get a more
feature rich "targeting contributors" page. If they've gone far enough to
do the ajax sign up, we can try to consider them a potential contributor.

-Mike
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Thu, 11 Mar 2010, Colin Walters wrote:

> On Thu, Mar 11, 2010 at 10:48 AM, Colin Walters <walters@verbum.org> wrote:
> >
> > This needs to go through the Fedora Design group.
> >
> > I'd beg to please, pretty please, don't add stuff to firstboot;
>
> And actually how the heck would this even work at a technical level?
> There's no networking in firstboot because NetworkManager isn't set
> up.
>

You'd do what smolt does and queue the request until after network is up.
But yes, firstboot isn't quite the right place for this. Even the way
smolt does it is pretty nasty.

-Mike
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 03/11/2010 10:48 AM, Colin Walters wrote:
> Design strawman: PackageKit updates have a little "advertisement"
> section at the bottom "Participate in testing updates [ Create Fedora
> Account ]".
>
> While the target audience says that we're targeting people willing to
> participate, that's a far cry from forcing them to before they can
> even boot fully into the OS and say set up Ruby On Rails or browse
> Facebook or whatever they want to do that matters to them in their
> life.

As long as we're still getting in front of them without being annoying
(e.g. "WHY WONT THAT BLINKING PACKAGEKIT ICON STOP BLINKING!!!!!"), I
don't care how we get them signed up.

~spot
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On 03/11/2010 12:51 PM, Greg DeKoenigsberg wrote:
> On Thu, 11 Mar 2010, Tom "spot" Callaway wrote:
>
>> I posit an alternative suggestion:
>
> (brilliant suggestion snipped)
>
> A big +1 -- but it's a huge feature. Is it something that could happen
> for Fedora 14?
>
> I think it's big enough that it would be worth organizing a FAD for. It's
> exactly the kind of "infrastructure of participation" solution that
> requires significant investment, but drives major change.

Agreed. I think if we had the right people in a room for a few days we
could bang out something pretty solid for F14.

~spot
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board

On Thu, 11 Mar 2010, Tom "spot" Callaway wrote:

> I posit an alternative suggestion:

(brilliant suggestion snipped)

A big +1 -- but it's a huge feature. Is it something that could happen
for Fedora 14?

I think it's big enough that it would be worth organizing a FAD for. It's
exactly the kind of "infrastructure of participation" solution that
requires significant investment, but drives major change.

--g

--
Educational materials should be high-quality, collaborative, and free.
Visit http://opensource.com/education and join the conversation.
_______________________________________________
advisory-board mailing list
advisory-board@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/advisory-board