A discussion regarding testing packages in the -proposed pocket has
recently come up. Obviously our own personal interest lies with testing
of the kernels which land in -proposed before they are promoted to
-updates. Below is an initial summary of requirements that need to be
resolved as well as some follow up questions which have been raised.
Initial list of requirements from QA Team (ie Mathieu Trudel-Lapierre):
* an infrastructure to support adding these special tasks to the
installation preseed, in other words what will add the repository and
install/upgrade packages (mostly done)
* a tracker to kick-start installations once new packages are found in
-proposed
* a way to display the fact, on the certification website, that testing
was done on a release image (or point-release image), with additional
packages from proposed, and *which* additional packages
So far, the tracker and display changes have not been started yet. This
is all development on the hardware certification side of things.
Our comments/concerns from a kernel point of view are as follows:
* Are the above requirements blocking current testing of packages in
-proposed? It seems QA could/should be testing every kernel uploaded to
-proposed already, even if it needs to be done manually until the
automation is in place.
* How long is it going to take for the infrastructure to be completed
to an extent where you/we can start using it?
* When will QA start on the tracker / display changes and what is an
estimate for having them completed?
* What constitutes the set of automated tests that are currently being
run?
* Where can we expect to view test results and what permissions will we
need to access the results?
I think this summarizes the current discussion up to now. Please feel
free to add anything I've missed or raise any additional questions.
Thanks,
Leann
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
08-02-2010, 02:27 PM
Brad Figg
Automate testing from -proposed
On 07/30/2010 09:46 AM, Leann Ogasawara wrote:
> Hi All,
>
> A discussion regarding testing packages in the -proposed pocket has
> recently come up. Obviously our own personal interest lies with testing
> of the kernels which land in -proposed before they are promoted to
> -updates. Below is an initial summary of requirements that need to be
> resolved as well as some follow up questions which have been raised.
>
> Initial list of requirements from QA Team (ie Mathieu Trudel-Lapierre):
>
> * an infrastructure to support adding these special tasks to the
> installation preseed, in other words what will add the repository and
> install/upgrade packages (mostly done)
> * a tracker to kick-start installations once new packages are found in
> -proposed
> * a way to display the fact, on the certification website, that testing
> was done on a release image (or point-release image), with additional
> packages from proposed, and *which* additional packages
>
> So far, the tracker and display changes have not been started yet. This
> is all development on the hardware certification side of things.
>
> Our comments/concerns from a kernel point of view are as follows:
> * Are the above requirements blocking current testing of packages in
> -proposed? It seems QA could/should be testing every kernel uploaded to
> -proposed already, even if it needs to be done manually until the
> automation is in place.
> * How long is it going to take for the infrastructure to be completed
> to an extent where you/we can start using it?
> * When will QA start on the tracker / display changes and what is an
> estimate for having them completed?
> * What constitutes the set of automated tests that are currently being
> run?
> * Where can we expect to view test results and what permissions will we
> need to access the results?
>
> I think this summarizes the current discussion up to now. Please feel
> free to add anything I've missed or raise any additional questions.
>
> Thanks,
> Leann
>
>
I think we (the kernel team) would like to get a specific date from QA
when this testing will start. The results could be email posted to this
list or something else. We just want this to get going as soon as
possible.
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
08-04-2010, 03:01 PM
Mathieu Trudel-Lapierre
Automate testing from -proposed
On lun, 2010-08-02 at 07:27 -0700, Brad Figg wrote:
> On 07/30/2010 09:46 AM, Leann Ogasawara wrote:
> > Hi All,
> >
> > A discussion regarding testing packages in the -proposed pocket has
> > recently come up. Obviously our own personal interest lies with
testing
> > of the kernels which land in -proposed before they are promoted to
> > -updates. Below is an initial summary of requirements that need to
be
> > resolved as well as some follow up questions which have been raised.
> >
> > Initial list of requirements from QA Team (ie Mathieu
Trudel-Lapierre):
> >
> > * an infrastructure to support adding these special tasks to the
> > installation preseed, in other words what will add the repository
and
> > install/upgrade packages (mostly done)
> > * a tracker to kick-start installations once new packages are
found in
> > -proposed
> > * a way to display the fact, on the certification website, that
testing
> > was done on a release image (or point-release image), with
additional
> > packages from proposed, and *which* additional packages
> >
> > So far, the tracker and display changes have not been started yet.
This
> > is all development on the hardware certification side of things.
> >
> > Our comments/concerns from a kernel point of view are as follows:
> > * Are the above requirements blocking current testing of packages
in
> > -proposed? It seems QA could/should be testing every kernel
uploaded to
> > -proposed already, even if it needs to be done manually until the
> > automation is in place.
It's certainly making the whole process very complex and labor
intensive. Since test results currently end up in the certification
website, we'll at least need to get those out and build a report. That's
only for the reporting, as we still need to start installs of systems
and the upgrade on them to what's in -proposed.
On that subject, is testing -proposed once a day a reasonable way of
taking care of the kernel uploads to -proposed and any other packages?
That means there is much less development to be done for automation,
especially if we just test "everything in proposed" once a day, and
provides, I think, reasonable coverage.
I already started running tests daily (since yesterday) on some systems
in the Montreal lab (only laptops). It's not ideal but we also need to
manually test some of the systems with 10.04.1 images, and run automatic
tests for Alpha 3.
> > * How long is it going to take for the infrastructure to be
completed
> > to an extent where you/we can start using it?
As above, I can already use the scripts necessary to start installations
and the
upgrade of these installations to the full contents of -proposed.
> > * When will QA start on the tracker / display changes and what is
an
> > estimate for having them completed?
I've started to look at how we could check the proposed component for
new
packages, but as above, if we just test "everything in proposed", it's
not necessary. Using a cron job instead takes 5 minutes to implement.
As for displaying results, once I get them in
certification.canonical.com and displaying differently than a standard
install of 10.04 or 10.04.1, then I can start to gather these results in
a report to be shared to everyone. I'm expecting this to maybe take a
week, so I can be sure the results can really be differentiated on
certification.c.c, so that I can mine them from the database to build
the report that would go on qa.u.c.
[...]
> > * Where can we expect to view test results and what permissions
will we
> > need to access the results?
> >
My take on this is the results will be available in an html report on
qa.ubuntu.com, as discussed at the Platform sprint.
[...]
>
> I think we (the kernel team) would like to get a specific date from QA
> when this testing will start. The results could be email posted to
this
> list or something else. We just want this to get going as soon as
> possible.
I think we can start testing as soon as the requirements are agreed on,
and once we know exactly what we will be testing.
Please keep in mind that testing of -proposed includes kernel uploads
but also any other packages, and needs to play nicely along with the
"normal" testing we run daily, so testing of the daily ISOs, in both
desktop and alternate form for mostly all the Ubuntu flavours; as well
as testing of the daily point release ISOs.
On 4 August 2010 16:01, Mathieu Trudel-Lapierre
<mathieu.trudel-lapierre@canonical.com> wrote:
> On lun, 2010-08-02 at 07:27 -0700, Brad Figg wrote:
>> On 07/30/2010 09:46 AM, Leann Ogasawara wrote:
>> > Hi All,
>> >
>> > A discussion regarding testing packages in the -proposed pocket has
>> > recently come up. *Obviously our own personal interest lies with
> testing
>> > of the kernels which land in -proposed before they are promoted to
>> > -updates. *Below is an initial summary of requirements that need to
> be
>> > resolved as well as some follow up questions which have been raised.
>> >
>> > Initial list of requirements from QA Team (ie Mathieu
> Trudel-Lapierre):
>> >
>> > * * an infrastructure to support adding these special tasks to the
>> > installation preseed, in other words what will add the repository
> and
>> > install/upgrade packages (mostly done)
>> > * * a tracker to kick-start installations once new packages are
> found in
>> > -proposed
>> > * * a way to display the fact, on the certification website, that
> testing
>> > was done on a release image (or point-release image), with
> additional
>> > packages from proposed, and *which* additional packages
>> >
>> > So far, the tracker and display changes have not been started yet.
> This
>> > is all development on the hardware certification side of things.
>> >
>> > Our comments/concerns from a kernel point of view are as follows:
>> > * * Are the above requirements blocking current testing of packages
> in
>> > -proposed? *It seems QA could/should be testing every kernel
> uploaded to
>> > -proposed already, even if it needs to be done manually until the
>> > automation is in place.
>
> It's certainly making the whole process very complex and labor
> intensive. Since test results currently end up in the certification
> website, we'll at least need to get those out and build a report. That's
> only for the reporting, as we still need to start installs of systems
> and the upgrade on them to what's in -proposed.
>
> On that subject, is testing -proposed once a day a reasonable way of
> taking care of the kernel uploads to -proposed and any other packages?
> That means there is much less development to be done for automation,
> especially if we just test "everything in proposed" once a day, and
> provides, I think, reasonable coverage.
>
> I already started running tests daily (since yesterday) on some systems
> in the Montreal lab (only laptops). It's not ideal but we also need to
> manually test some of the systems with 10.04.1 images, and run automatic
> tests for Alpha 3.
>
>> > * * How long is it going to take for the infrastructure to be
> completed
>> > to an extent where you/we can start using it?
>
> As above, I can already use the scripts necessary to start installations
> and the
> upgrade of these installations to the full contents of -proposed.
>
>> > * * When will QA start on the tracker / display changes and what is
> an
>> > estimate for having them completed?
>
> I've started to look at how we could check the proposed component for
> new
> packages, but as above, if we just test "everything in proposed", it's
> not necessary. Using a cron job instead takes 5 minutes to implement.
>
> As for displaying results, once I get them in
> certification.canonical.com and displaying differently than a standard
> install of 10.04 or 10.04.1, then I can start to gather these results in
> a report to be shared to everyone. I'm expecting this to maybe take a
> week, so I can be sure the results can really be differentiated on
> certification.c.c, so that I can mine them from the database to build
> the report that would go on qa.u.c.
>
> [...]
>> > * * Where can we expect to view test results and what permissions
> will we
>> > need to access the results?
>> >
>
> My take on this is the results will be available in an html report on
> qa.ubuntu.com, as discussed at the Platform sprint.
>
> [...]
>>
>> I think we (the kernel team) would like to get a specific date from QA
>> when this testing will start. The results could be email posted to
> this
>> list or something else. We just want this to get going as soon as
>> possible.
>
> I think we can start testing as soon as the requirements are agreed on,
> and once we know exactly what we will be testing.
>
> Please keep in mind that testing of -proposed includes kernel uploads
> but also any other packages, and needs to play nicely along with the
> "normal" testing we run daily, so testing of the daily ISOs, in both
> desktop and alternate form for mostly all the Ubuntu flavours; as well
> as testing of the daily point release ISOs.
>
> Does this all make sense/ seem acceptable?
Joining the discussion late, I wanted to express before that the LTP
testsuite in my opinion provides a wide gamut of exposure, which will
screen out userspace breakage, either due to eg behavioural changes or
bugs causing oopses or processes to be killed.
After eg tree merges, clearly the most common paths will have been
exposed to testing, but coverage of edge and less common cases is
needed; LTP seems to provide a lot in the way of this. Perhaps ABI
compatibility needs to be checked to some extent too?
Thanks,
Daniel
--
Daniel J Blueman
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
08-05-2010, 05:09 PM
Steve Beattie
Automate testing from -proposed
On Wed, Aug 04, 2010 at 11:01:33AM -0400, Mathieu Trudel-Lapierre wrote:
> On that subject, is testing -proposed once a day a reasonable way of
> taking care of the kernel uploads to -proposed and any other packages?
> That means there is much less development to be done for automation,
> especially if we just test "everything in proposed" once a day, and
> provides, I think, reasonable coverage.
Kernels do not go into proposed that frequently, though we do have
other boot critical packages that show up in proposed as well (grub,
mountall, plymouth, etc.). But even including those packages, daily
is overkill for the frequency that these things show up in proposed.
However, if that's the path of least resistance to getting testing of
proposed on lab hardware, I'd rather have that than no testing at all.
The concern would be if running daily proposed tests would interfere
with other daily testing and usage of those machines. But as an
easy path to development you could automate on a daily basis now,
work out the kinks with respect to the rest of process (reporting,
triaging failures, etc.), and then if it becomes too much of resource
hog in terms of lab time, then you could look to doing the work to
detect new packages in proposed and kick off test runs if something
boot-critical comes into the pocket.
> I already started running tests daily (since yesterday) on some systems
> in the Montreal lab (only laptops). It's not ideal but we also need to
> manually test some of the systems with 10.04.1 images, and run automatic
> tests for Alpha 3.
Awesome!
> > > * Where can we expect to view test results and what permissions
> will we
> > > need to access the results?
> > >
>
> My take on this is the results will be available in an html report on
> qa.ubuntu.com, as discussed at the Platform sprint.
Ultimately, the primary consumer for this data is the SRU admin team,
as they're looking to determine whether or not they should "promote"
a given package in proposed to updates, and thus are looking for
feedback as to whether packages in proposed introduce regressions.
Making it as easy as possible for an admin to find out the results
associated with a given package should be a prominent goal -- doing
the testing is not useful if the feedback never makes it to them.
The secondary consumer of this is the kernel team and other developers
looking to see why the package they put in proposed failed in some way.
They'll want detailed logs available to do failure diagnosis.
> > I think we (the kernel team) would like to get a specific date from QA
> > when this testing will start. The results could be email posted to
> this
> > list or something else. We just want this to get going as soon as
> > possible.
>
> I think we can start testing as soon as the requirements are agreed on,
> and once we know exactly what we will be testing.
>
> Please keep in mind that testing of -proposed includes kernel uploads
> but also any other packages, and needs to play nicely along with the
> "normal" testing we run daily, so testing of the daily ISOs, in both
> desktop and alternate form for mostly all the Ubuntu flavours; as well
> as testing of the daily point release ISOs.
Right, this is why you would want to scale back the frequency of test
runs, to reduce the amount of time used for this task, rather than
the above. Also, IIRC, when these machines are "idle" from a testing
perspective, they're re-purposed as launchpad PPA build hosts. There's
already complaints about not enough available builders.
--
Steve Beattie
<sbeattie@ubuntu.com>
http://NxNW.org/~steve/
--
kernel-team mailing list
kernel-team@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/kernel-team
Automate testing from -proposed
