Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Fedora Development (http://www.linux-archive.org/fedora-development/)
-   -   shortening time passed in bodhi? (http://www.linux-archive.org/fedora-development/200163-shortening-time-passed-bodhi.html)

Ralf Corsepius 11-26-2008 02:49 PM

shortening time passed in bodhi?
 
On Wed, 2008-11-26 at 16:22 +0100, Patrice Dumas wrote:
> On Wed, Nov 26, 2008 at 04:12:06PM +0100, Kevin Kofler wrote:
> > Patrice Dumas wrote:
> > > Most of the time I push to stable when reminded by a mail (nice feature)
> > > since nobody cares to give feedback on my updates and they are not
> > > urgent. However recently I had an update I wanted to have in as soon as
> > > possible since the app was broken without it. It took 8 days to have it
> > > pushed, something like 2-3 days for the push to testing and the remaining
> > > for the push and signing.
> >
> > If you want it out as soon as possible, skip testing and push directly to
> > stable.
>
> I didn't noticed that. In that case it goes straight to stable, without
> neing in a pending state at all?
Right, but the delays until a final push happens typically still is in
the order of several days, occasionally more, sometimes much more :)

Ralf


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Kevin Kofler 11-26-2008 03:04 PM

shortening time passed in bodhi?
 
Patrice Dumas wrote:
> I didn't noticed that. In that case it goes straight to stable, without
> neing in a pending state at all?

Well, it's still in pending until the next push, but it doesn't go through
testing (from where you have to request another push to stable). This means
you only have to wait for 1 push instead of 2.

Kevin Kofler

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Jesse Keating 11-26-2008 03:40 PM

shortening time passed in bodhi?
 
On Wed, 2008-11-26 at 16:04 +0100, Patrice Dumas wrote:
> Most of the time I push to stable when reminded by a mail (nice feature)
> since nobody cares to give feedback on my updates and they are not
> urgent. However recently I had an update I wanted to have in as soon as
> possible since the app was broken without it. It took 8 days to have it
> pushed, something like 2-3 days for the push to testing and the remaining
> for the push and signing.
>
> Could this be improved?

Yes. The compose process (particularly now that we're pushing out
updates for 8, 9, and 10) takes almost all of one work day, with only an
hour or two of prep time before the push, so at best you'll likely get
one per day. I've been trying to do one at least every other day,
except for on weekends. However the last couple pushes were spaced out
a bit, as we were trying to get bodhi ready for pushing F10 updates, and
then we didn't want to land a bunch more updates at the same time that
mirrors were trying to get the bitflip change to release F10.

We have a holiday week for a large number of US folks this week so the
updates pushes may get a bit delayed this week as well.

--
Jesse Keating
Fedora -- Freedomē is a feature!
identi.ca: http://identi.ca/jkeating
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Patrice Dumas 11-26-2008 04:28 PM

shortening time passed in bodhi?
 
On Wed, Nov 26, 2008 at 08:40:09AM -0800, Jesse Keating wrote:
>
> Yes. The compose process (particularly now that we're pushing out
> updates for 8, 9, and 10) takes almost all of one work day, with only an
> hour or two of prep time before the push, so at best you'll likely get
> one per day. I've been trying to do one at least every other day,

What is exactly a 'compose'? Is it what forces to go first in pending?

What about the other questions?

> We have a holiday week for a large number of US folks this week so the
> updates pushes may get a bit delayed this week as well.

That kind of delay is not problematic, since I guess it should be fixed by
having more people trusted to do the pushes and a signing server, I am
more worried about the unneeded delays.

--
Pat

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Michael Schwendt 11-26-2008 04:32 PM

shortening time passed in bodhi?
 
On Wed, 26 Nov 2008 15:25:41 +0000, Richard W.M. Jones wrote:

> On Wed, Nov 26, 2008 at 04:22:34PM +0100, Patrice Dumas wrote:
> > On Wed, Nov 26, 2008 at 04:12:06PM +0100, Kevin Kofler wrote:
> > > Patrice Dumas wrote:
> > > > Most of the time I push to stable when reminded by a mail (nice feature)
> > > > since nobody cares to give feedback on my updates and they are not
> > > > urgent. However recently I had an update I wanted to have in as soon as
> > > > possible since the app was broken without it. It took 8 days to have it
> > > > pushed, something like 2-3 days for the push to testing and the remaining
> > > > for the push and signing.
> > >
> > > If you want it out as soon as possible, skip testing and push directly to
> > > stable.
> >
> > I didn't noticed that. In that case it goes straight to stable, without
> > neing in a pending state at all?
>
> Yes - I've used it a few times.

Not right. You can skip "testing", but you cannot skip "pending".

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Jesse Keating 11-26-2008 04:47 PM

shortening time passed in bodhi?
 
On Wed, 2008-11-26 at 18:28 +0100, Patrice Dumas wrote:
> > Yes. The compose process (particularly now that we're pushing out
> > updates for 8, 9, and 10) takes almost all of one work day, with only an
> > hour or two of prep time before the push, so at best you'll likely get
> > one per day. I've been trying to do one at least every other day,
>
> What is exactly a 'compose'?

The compose is the process that:

1) gets a list of pending requests
2) signs them accordingly
3) asks bodhi to push the pending requests
4) moves packages within tags accordingly in koji
5) uses mash to create new repos for:
5.1) dist-f8-updates
5.2) dist-f8-updates-testing
5.3) dist-f9-updates
5.4) dist-f9-updates-testing
5.5) dist-f10-updates
5.6) dist-f10-updates-testing
6) makes those repos multilib (still mash here)
7) generates update metadata for the above repos (which updates are
bugfix, security, need reboot, etc...)
8) touch appropriate bugs in bugzilla based on bodhi requests
9) wait for rsync to finish
10) send mails to fedora-package-announce
11) set all states correct in bodhi itself for the pending requests

That's the rough outline, and I likely have a few of the steps out of
order, and some of that is done in parallel. The more package updates
added (that is unique packages, not unique updates for a given package),
the longer the compose process takes as there is more to consider for
multilib, more packages for createrepo to fiddle with, etc...


> Is it what forces to go first in pending?

I'm not sure what you mean by this question.

>
> What about the other questions?

I'll look again.

--
Jesse Keating
Fedora -- Freedomē is a feature!
identi.ca: http://identi.ca/jkeating
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Jesse Keating 11-26-2008 04:52 PM

shortening time passed in bodhi?
 
On Wed, 2008-11-26 at 16:04 +0100, Patrice Dumas wrote:
> What are the reasons for holding updates? Is there somebody actually
> verifying that the package works, doesn't break the distro or the like?
> What exactly do releng/QA people with updates, ie what checks?

At this point we haven't inserted any automated QA into the system. I
have some plans, but they will mostly be post-build, rather than
post-bodhi request, although we could do it both times.

The main reason we haven't inserted any automated QA is that to get a
correct picture of what the distro would look like with that update
added takes a full compose, to ensure we get the right view of multilib,
and that we don't have an older copy of the package update floating in
repos resolving deps it shouldn't, etc... The compose process itself is
extremely time and resource consuming and it unfortunately hasn't been
as high of a priority to tackle as it should have been.

I feel pretty confident that we've solved many of our other major
problems and can now focus attention on better QA through automation and
that will be one of my main focuses during the F11/F12 cycles.

--
Jesse Keating
Fedora -- Freedomē is a feature!
identi.ca: http://identi.ca/jkeating
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Patrice Dumas 11-26-2008 05:08 PM

shortening time passed in bodhi?
 
On Wed, Nov 26, 2008 at 09:47:43AM -0800, Jesse Keating wrote:
>
> The compose is the process that:
>
> 1) gets a list of pending requests
> 2) signs them accordingly
> 3) asks bodhi to push the pending requests
> 4) moves packages within tags accordingly in koji
> 5) uses mash to create new repos for:
> 5.1) dist-f8-updates
> 5.2) dist-f8-updates-testing
> 5.3) dist-f9-updates
> 5.4) dist-f9-updates-testing
> 5.5) dist-f10-updates
> 5.6) dist-f10-updates-testing
> 6) makes those repos multilib (still mash here)
> 7) generates update metadata for the above repos (which updates are
> bugfix, security, need reboot, etc...)
> 8) touch appropriate bugs in bugzilla based on bodhi requests
> 9) wait for rsync to finish
> 10) send mails to fedora-package-announce
> 11) set all states correct in bodhi itself for the pending requests
>
> That's the rough outline, and I likely have a few of the steps out of
> order, and some of that is done in parallel.

Indeed, that's a lot of steps. But what steps take so long? There are
some steps that I don't really understand, but I can't see which one
would take more than some minutes, except maybe 9) if there are a lot of
packages -- and 4) if 4) involves transferring the packages. In any case
I guess that you know what you are doing, so my questions are certainly
very naive.

Also I am not really knowledgable, but maybe 9) could be done right
after 7), if it is a long step?

> > Is it what forces to go first in pending?
>
> I'm not sure what you mean by this question.

You answered above.

--
Pat

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Patrice Dumas 11-26-2008 05:12 PM

shortening time passed in bodhi?
 
On Wed, Nov 26, 2008 at 09:52:08AM -0800, Jesse Keating wrote:
> On Wed, 2008-11-26 at 16:04 +0100, Patrice Dumas wrote:
> > What are the reasons for holding updates? Is there somebody actually
> > verifying that the package works, doesn't break the distro or the like?
> > What exactly do releng/QA people with updates, ie what checks?
>
> At this point we haven't inserted any automated QA into the system. I
> have some plans, but they will mostly be post-build, rather than
> post-bodhi request, although we could do it both times.

Ok. You still haven't explained what checks were done, especially
between pending and stable or testing.

> The main reason we haven't inserted any automated QA is that to get a
> correct picture of what the distro would look like with that update
> added takes a full compose, to ensure we get the right view of multilib,
> and that we don't have an older copy of the package update floating in
> repos resolving deps it shouldn't, etc... The compose process itself is
> extremely time and resource consuming and it unfortunately hasn't been
> as high of a priority to tackle as it should have been.

Right. This is a good reason, especially if compose takes one day. Still
it doesn't explain the other delays. I still can't see what takes time
besides composing, nor why there is a pending state.

--
Pat

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Jesse Keating 11-26-2008 06:05 PM

shortening time passed in bodhi?
 
On Wed, 2008-11-26 at 19:08 +0100, Patrice Dumas wrote:
> Indeed, that's a lot of steps. But what steps take so long? There are
> some steps that I don't really understand, but I can't see which one
> would take more than some minutes, except maybe 9) if there are a lot of
> packages -- and 4) if 4) involves transferring the packages. In any case
> I guess that you know what you are doing, so my questions are certainly
> very naive.

The parts that take the longest are sorting out the signed packages from
koji output into a directory structure that is suitable for updates via
hardlinks, and then the createrepo runs. Sorting the package takes a
little time as koji puts packages into dirs of just <arch>, which can
include noarch. debuginfo is put along side the other arch packages.
We have to sort out the debuginfo, copy around any noarch, but also
examine the srpm the noarch package came from to determine if the noarch
package has any Exclude or ExclusiveArch settings. This involves a lot
of filesystem work and tends to be a lengthy process, as is just the
mere querying of koji for the 'latest-tagged' for each of the tags.

As for repodata, for each tag (dist-f{8,9,10}-{updates,updates-testing})
we have to create repodata for:

i386, x86_64, ppc, pp64
debuginfo
source

that's 6 repos worth of packages to run createrepo on, across an NFS
mount. For two of those repos (x86_64, ppc) in order to make multilib
we first copy in all the compat arch packages (i386, ppc64), run
createrepo, run through an algorithm to select certain ones as being
"multilib" and then depsolve the rest, throwing out what's not needed,
and then running createrepo again.

So to recap, for each tag, we have 11 createrepo calls, and since we're
doing 6 tags at this point, that's 66 calls to createrepo, again all
over NFS (since we don't want to put the software that pushes updates on
the same machine that holds the filesystem that koji uses, and we want
to use hardlinks to avoid adding time to copy all those packages over
NFS).

Right now, I do believe there is a faulty switch between the machine(s)
capable of doing the repo creations and the nfs server, which is capping
bandwidth around 100mb/s rather than 1000mb/s. We've been trying to get
that fixed for nearly a year, and will continue trying, Mike McGrath is
scheduled to be onsite in the colo to do various things such as fixing
this in December.

As stated in the previous email, there are steps that are done in
parallel, like many of the createrepo calls. But there is only so much
bandwidth between the caller and the NFS share, and there is only so
much memory in the caller, and there are only so many tasks that can
truly be done in parallel.

--
Jesse Keating
Fedora -- Freedomē is a feature!
identi.ca: http://identi.ca/jkeating
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list


All times are GMT. The time now is 04:33 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.