FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS Development

 
 
LinkBack Thread Tools
 
Old 04-06-2011, 01:10 PM
Karanbir Singh
 
Default what / how could test reports be published

Hi,

On 04/06/2011 01:01 PM, Thomas Bendler wrote:
> [... automated VM install ...]
> in principle a good idea, should be (from my point of view) extended to

It works in practise as well ( over 700 automated installs during the
centos-5.6 test phase would say so .. )

> something like BFO (http://boot.fedoraproject.org/) to make it possible
> to install on physical machines as well. So something like USB boot
> stick which get standard configurations based on kickstart files, do the
> installation, report back to console and central QA server and do the
> next install on the list (the only question is how to make previous runs
> persistent so installations won't be done twice until QA system request
> a second install because of a failure).

It would be good to get boot.k.o functionality included in, but afaik
that needs httpfs etc into the kernel; so we might only be able to
achieve that for the centosplus kernel. Would you be happy to take on
the task of looking into whats involved and how feasible that might be ?

The idea of getting real-iron into the QA testing loop is very
interesting and perhaps even essential. Could a cobbler type setup,
being able to do ipmi calls to real iron be a good substitute ?

> - lshw

there is no such thing on CentOS

> - dmesg

added.

> - maybe bootchart

not sure about this. Maybe bootchart makes for a good test case in itself ?

> GIT is a good idea but you need a wrapper around that check successful
> runs and shorten this one to profile xyz on hardware abc has no problem.

the test run script does that already, so it will output something like
this : (taken from a real test ):

78 test cases complete. 73 passes, 5 fails and 0 exceptions.

> See above, an USB stick for real hardware and an ISO for virtual
> hardware will probably work best.

Ideally, we should be testing every install type ( so from boot.iso,
over pxe + ftp, pxe + http, pxe + images, cd1, dvd, diskimage etc ). At
the moment though its only usrin virt-install ( so pxe images ) into Xen
hosts using installmethod = http.

I guess if someone wants to propose automation for the various things it
would be good. Is it even possible to emulate a usb disk somewhere (
kind of like being able to use an iso image as a cd-rom in xen/kvm etc )

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 10:32 AM
Karanbir Singh
 
Default what / how could test reports be published

On 04/06/2011 11:06 AM, Karanbir Singh wrote:
> at the moment all test are run by hand, and people look at the output on
> the cli. Lets change that. Fabian and I did some work during Fosdem this
> year to get an automation script in place.
>

Must say I'm disappointed by the level of feedback here. Given all the
calls from all the people saying they wanted more info, when there is an
option to get involved and help create that info pool, its total silence.

If its because I wasn't clear enough, or what I said didn't make any
sense, please do ask and I'll try again.

Alternatively, would it help if I just put something up. Would that make
it easier for everyone to get their heads around what the intent is ?

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 11:54 AM
Athmane Madjoudj
 
Default what / how could test reports be published

On 04/07/2011 11:32 AM, Karanbir Singh wrote:
> On 04/06/2011 11:06 AM, Karanbir Singh wrote:
>> at the moment all test are run by hand, and people look at the output on
>> the cli. Lets change that. Fabian and I did some work during Fosdem this
>> year to get an automation script in place.
>>
>
> Must say I'm disappointed by the level of feedback here. Given all the
> calls from all the people saying they wanted more info, when there is an
> option to get involved and help create that info pool, its total silence.
>
> If its because I wasn't clear enough, or what I said didn't make any
> sense, please do ask and I'll try again.
>
> Alternatively, would it help if I just put something up. Would that make
> it easier for everyone to get their heads around what the intent is ?
>

I'll try to implement a temporary solution (PS. I'm not a 'real'
developer.), after writing some kickstarts and tests, I figure out that
a such solution is necessary (testing will be a pain work for CentOS-QA
team.)

I've just begin to understand how CentOS QA process works, well after
reading [1] and [2].

[1] http://lists.centos.org/pipermail/centos-devel/2011-April/007321.html

[2] http://lists.centos.org/pipermail/centos-devel/2011-April/007335.html


--
Athmane Madjoudj
RHCE
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 12:21 PM
Thomas Bendler
 
Default what / how could test reports be published

2011/4/7 Karanbir Singh <mail-lists@karan.org>


[...]

Must say I'm disappointed by the level of feedback here. Given all the

calls from all the people saying they wanted more info, when there is an

option to get involved and help create that info pool, its total silence.

What do you expect, BFO adapted for CentOS in one day completely tested and ready to roll-out? Sorry but this need a bit more work and coordination (and I guess most people here have regular jobs which had in the end more priority).


*


[...]

Alternatively, would it help if I just put something up. Would that make

it easier for everyone to get their heads around what the intent is ?

I think (my personal opinion) that the first thing that should be done is a kind of an architecture overview to see what the goal of this project is. Based on this tasks need to be defined and names need to be found to execute the tasks. With other words, this need to be coordinated in a proper project way to get people involved and doing stuff. It's not helpful to say we need something but leave the community alone with the realization and be disappointed if the thing won#t be solved in a short time frame. But again, this is my personal opinion, I'm just convinced that a project like this need a proper project management when more than two people contribute developments if the result should be valuable.



Regards, Thomas

_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 12:39 PM
Karanbir Singh
 
Default what / how could test reports be published

On 04/07/2011 01:21 PM, Thomas Bendler wrote:
> What do you expect, BFO adapted for CentOS in one day completely tested
> and ready to roll-out? Sorry but this need a bit more work and
> coordination (and I guess most people here have regular jobs which had
> in the end more priority).

As do I, have a day job. And I'm also not looking for anything being
developed or implemented. At the moment, its just a case of asking
people for enough feedback so we can start with the design stages.
Implementing stuff comes a fair bit down the road.

> I think (my personal opinion) that the first thing that should be done
> is a kind of an architecture overview to see what the goal of this
> project is. Based on this tasks need to be defined and names need to be

Right, and therefore I am looking for people's feedback on what they
would expect to see in such a tool. Or are you suggesting that you are
not clear about why we are doing tests ?

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 01:02 PM
David Hrbáč
 
Default what / how could test reports be published

Dne 7.4.2011 14:21, Thomas Bendler napsal(a):
> What do you expect, BFO adapted for CentOS in one day completely
> tested and
> ready to roll-out? Sorry but this need a bit more work and coordination (and
> I guess most people here have regular jobs which had in the end more
> priority).

Well,
I will add this. Karanbir, you can't expect to find a lot of people to
contribute testing scripts. Firstly, this is something that not many
companies/people are doing, they are just deploying. And secondly, you
can't expect positive feedback and mailbox full of scripts after many
weeks of "if you don't like it, go aways".

> I think (my personal opinion) that the first thing that should be done is a
> kind of an architecture overview to see what the goal of this project is.
> Based on this tasks need to be defined and names need to be found to execute
> the tasks. With other words, this need to be coordinated in a proper project
> way to get people involved and doing stuff. It's not helpful to say we need
> something but leave the community alone with the realization and be
> disappointed if the thing won#t be solved in a short time frame. But again,
> this is my personal opinion, I'm just convinced that a project like this
> need a proper project management when more than two people contribute
> developments if the result should be valuable.

Thomas is very right, if you want the results, you must take it
seriously and keep it as project. It really seem to me like "go and
search for RH trademarks" issue again.
DH
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 01:06 PM
Athmane Madjoudj
 
Default what / how could test reports be published

On 04/07/2011 01:39 PM, Karanbir Singh wrote:
> On 04/07/2011 01:21 PM, Thomas Bendler wrote:
>> What do you expect, BFO adapted for CentOS in one day completely tested
>> and ready to roll-out? Sorry but this need a bit more work and
>> coordination (and I guess most people here have regular jobs which had
>> in the end more priority).
>
> As do I, have a day job. And I'm also not looking for anything being
> developed or implemented. At the moment, its just a case of asking
> people for enough feedback so we can start with the design stages.
> Implementing stuff comes a fair bit down the road.
>
>> I think (my personal opinion) that the first thing that should be done
>> is a kind of an architecture overview to see what the goal of this
>> project is. Based on this tasks need to be defined and names need to be
>
> Right, and therefore I am looking for people's feedback on what they
> would expect to see in such a tool. Or are you suggesting that you are
> not clear about why we are doing tests ?
>

Hi Karanbir,

I'm thinking of implementing a small script (python/virt-install etc...)
without any external tool (BFO, cobbler)


I have some questions:

1. Does we need to get the kickstart and tests from git repo ?

2. Is there any restrictions of the QA infrastructure, like setup a
central server/vm for hosting the Centos repo, kickstarts and the tests
script ?

I need this to prepare a similar environment for testing.

Regards.

--
Athmane Madjoudj
RHCE
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 01:07 PM
Karanbir Singh
 
Default what / how could test reports be published

Hi David,

On 04/07/2011 02:02 PM, David Hrbáč wrote:
> I will add this. Karanbir, you can't expect to find a lot of people to
> contribute testing scripts. Firstly, this is something that not many
> companies/people are doing, they are just deploying. And secondly, you
> can't expect positive feedback and mailbox full of scripts after many
> weeks of "if you don't like it, go aways".

ok, I take your point onboard. However, at this stage its not a case of
test-scripts, this is mostly orthogonal.

> Thomas is very right, if you want the results, you must take it
> seriously and keep it as project. It really seem to me like "go and
> search for RH trademarks" issue again.

That is mostly what I am trying to do here. But based on these comments,
I feel it would be easier to buyinto once there is something people can
see. Let me go do that and come back.

In the mean time : the tests scripts are very welcome, keep them coming.

- KB
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 03:14 PM
Les Mikesell
 
Default what / how could test reports be published

On 4/7/2011 7:39 AM, Karanbir Singh wrote:
> On 04/07/2011 01:21 PM, Thomas Bendler wrote:
>> What do you expect, BFO adapted for CentOS in one day completely tested
>> and ready to roll-out? Sorry but this need a bit more work and
>> coordination (and I guess most people here have regular jobs which had
>> in the end more priority).
>
> As do I, have a day job. And I'm also not looking for anything being
> developed or implemented. At the moment, its just a case of asking
> people for enough feedback so we can start with the design stages.
> Implementing stuff comes a fair bit down the road.
>
>> I think (my personal opinion) that the first thing that should be done
>> is a kind of an architecture overview to see what the goal of this
>> project is. Based on this tasks need to be defined and names need to be
>
> Right, and therefore I am looking for people's feedback on what they
> would expect to see in such a tool. Or are you suggesting that you are
> not clear about why we are doing tests ?

It's not clear what the test should show. Also, I haven't seen anything
that would help to prioritize such work. That is, where is time
currently being consumed, and how can the tests be designed to predict
how to solve the problems they detect in a way that the results help
move the project forward?

--
Les Mikesell
lesmikesell@gmail.com



_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 
Old 04-07-2011, 03:41 PM
Dag Wieers
 
Default what / how could test reports be published

On Thu, 7 Apr 2011, Les Mikesell wrote:

> On 4/7/2011 7:39 AM, Karanbir Singh wrote:
>> On 04/07/2011 01:21 PM, Thomas Bendler wrote:
>>
>>> What do you expect, BFO adapted for CentOS in one day completely tested
>>> and ready to roll-out? Sorry but this need a bit more work and
>>> coordination (and I guess most people here have regular jobs which had
>>> in the end more priority).
>>
>> As do I, have a day job. And I'm also not looking for anything being
>> developed or implemented. At the moment, its just a case of asking
>> people for enough feedback so we can start with the design stages.
>> Implementing stuff comes a fair bit down the road.
>>
>>> I think (my personal opinion) that the first thing that should be done
>>> is a kind of an architecture overview to see what the goal of this
>>> project is. Based on this tasks need to be defined and names need to be
>>
>> Right, and therefore I am looking for people's feedback on what they
>> would expect to see in such a tool. Or are you suggesting that you are
>> not clear about why we are doing tests ?
>
> It's not clear what the test should show. Also, I haven't seen anything
> that would help to prioritize such work. That is, where is time
> currently being consumed, and how can the tests be designed to predict
> how to solve the problems they detect in a way that the results help
> move the project forward?

This gives also the impression that the reason releases are delayed for so
long is because of the lack of QA, while it often takes weeks for anything
to be submitted to QA. So even if the QA could be improved (which no doubt
it can), it is reasonable to assume we could also make 'going to QA'
faster by opening up the process, by feeding QA faster with new updates
and by opening up the QA process so more people can actually help test the
intermediate builds so we get more problems faster.

It seems to me a set of kickstart files is only going to achieve a small
improvement to the whole process (mostly testing anaconda) and the causes
for the delays during the 85 days CentOS 5.6 has been developed.

(The above is based on my experience with previous QA processes and what I
have heard from people involved in the CentOS 5.6 QA process.)

It would be nice to have metrics that indicate when the first build was
ready for QA, where things got blocked and how to unblock those in the
future. But that requires a transparent/open process to even discuss this
here.

--
-- dag wieers, dag@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, info@dagit.net, http://dagit.net/

[Any errors in spelling, tact or fact are transmission errors]
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
http://lists.centos.org/mailman/listinfo/centos-devel
 

Thread Tools




All times are GMT. The time now is 12:21 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org