FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-15-2009, 02:44 PM
Adeodato Simó
 
Default A hack to alleviate transitions in Britney; now what?

Hello,

this mail is to talk a bit about the current situation regarding
transitions in unstable. In my opinion, it is unfortunate that the
Release Team has had to insist on semi-serializing them, because that’s
not the kind of development you want to have in unstable right after a
release.

Executive summary
=================

We’ll manage to get Britney to be less of a PITA when dealing with
transitions, which will hopefully enable us to do more of them
concurrently if that’s what we want.

This will be done by letting the old libraries stay in testing for a
while after introducing the new one.

Please still wait for an ACK from the Release Team before starting a
transition. We’ll be handing them out more quickly now, but it’s very
important that somebody checks whether reverse dependencies will rebuild
fine or need patches, as to minimize the time core build infrastructure
remains broken.

Please find below the longer version.

Part 1: explanations
====================

Let me start by explaining why it has been that way, the semi-serializing
of transitions (you can skip the next two paragraphs if you already know
how britney handles transitions). When a library package bumps its
SONAME, it normally does so by creating a new package and dropping the
old one from the same source package. When britney tries to migrate this
change to testing, it wants to migrate at the same time both the
creation of the new package, and the removal of the old one. But removal
of the old one means rendering packages uninstallable (those depending
on the old name), so this effectively means that all packages that
depend on the old library in testing must get a new version of
themselves migrated at the same time that the library itself migrates
(they can’t migrate one at a time, because the new library would not be
in testing yet). For obvious reasons it’s a bit complicated to get all
packages in shape at the same time, but it’s certainly doable.

The problem is when you have several transitions open, because they are
bound to get tied via packages that depend on two or more of the
disappeared libraries, which means that to update package P for
transition T, you must do transition T2 at the same time, which brings
in a whole lot of more packages to migrate at the same time. And that’s
what increases the complexity of things, the ties between transitions.

I’ve been thinking what to do about this, because I sincerely hate to be
the guy that is blocking people from doing the work they’ve been waiting
for so long already. You must believe me when I say I’m very unhappy
about doing that. Should it have occurred to me earlier, I would
probably have gone for making squeeze “hidden” from the public (i.e., no
Packages.gz files), so that unstable could get very entangled and slowly
converge. The thing is that a hidden testing allows you to break it, and
a testing with permission to get broken converges much more quickly than
a testing that must remain installable.

But it’s now too late for that, and has a lot of disadvantages of its
own, so I’ve revisited a way of doing transitions that has always been
in the mind of the Release Team, but that britney does not support out
of the box: allowing the old library package to stay in testing for a
while. This means that not all packages have to migrate at once, because
the ones that do not migrate won’t be uninstallable, since the old
library stays.

Now, this has its own set of problems and caveats as well, since if you
don’t pay attention and take care of later cleanup, you end up with
packages in testing that do not belong to any source in testing, which
is bad. These partial migrations can also be problematic (AFAIK) if eg.
a recompiled library against the new transition T migrates, and a binary
depending on it and the old library of T doesn’t, whereas both the old T
and the new T will get loaded into the process namespace.

These concerns notwithstanding, I’ve decided to start using this method
as a common procedure this early in the release cycle, but only as a
means to break *ties* between transitions, and some hand-picked
exceptions. I don’t want to systematically (ab)use it, because it’d be
hard to clean afterwards.

The consequence is that individual transitions will take approximately
the same time to get ready themselves, but that once that happens it’ll
be possible to migrate them without having to wait on some other
transitions.

I’ve already done that with libdvdread, which was tied to libraw1394 and
ffmpeg via vlc and a couple others. This has been relieving, because
having a transition ready in unstable but blocked for migration means
that it will inevitably go back to “unready” after a while of waiting.

As hinted above, britney does not support this procedure out of the box,
and it has to be done with a gross hack and manual poking, but I’m
willing to do that because I think it makes things significantly better.
Additionally, britney2 does support it, and I’m hoping that we can start
using that codebase soon.

Part 2: opportunity for feedback
================================

With this procedure in place, it is now possible to hand out transition
tickets at a faster pace, and I intend to do that. This means unstable
is probably going to see more breakage, but at the end it’s going to
allow us to get the first post-release set of transitions done faster,
which I believe is important.

If somebody has good reasons why it shouldn’t be that way, or
consequences about the procedure explained above that I haven’t
foreseen, please mention them now.

However, please do not take this mail as a call for starting any pending
transtions at once. We still need to hand out tickets, but we’ll be
handing them more quickly. It is, for example, very important to
determine if the rebuilds of reverse dependencies will succeed, or if
contrary source changes are needed for them, and whether patches exist
or not. It’s one thing for a transition to take its time to migrate, but
it’s another to effectively prevent other developers from doing their
work, as it happened with poppler/texlive.

Finally, I also want to thank all the developers that have shown their
willingness to cooperate with the Release Team and wait their turn in
the transition queue, whilst seeing other developers get theirs started
by not coordinating with us. Thanks!!, it’s much appreciated.

Cheers,

--
- Are you sure we're good?
- Always.
-- Rory and Lorelai


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 01:55 AM
Steve Langasek
 
Default A hack to alleviate transitions in Britney; now what?

On Sun, Mar 15, 2009 at 04:44:08PM +0100, Adeodato Simó wrote:
> Now, this has its own set of problems and caveats as well, since if you
> don’t pay attention and take care of later cleanup, you end up with
> packages in testing that do not belong to any source in testing, which
> is bad.

Will there be reports produced on a regular basis of the stale libraries in
testing, and their reverse-dependencies, so that people can easily pitch in
to help with this later cleanup?

Cheers,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 06:48 AM
Don Armstrong
 
Default A hack to alleviate transitions in Britney; now what?

On Sun, 15 Mar 2009, Steve Langasek wrote:
> On Sun, Mar 15, 2009 at 04:44:08PM +0100, Adeodato Simó wrote:
> > Now, this has its own set of problems and caveats as well, since
> > if you don’t pay attention and take care of later cleanup, you end
> > up with packages in testing that do not belong to any source in
> > testing, which is bad.
>
> Will there be reports produced on a regular basis of the stale
> libraries in testing, and their reverse-dependencies, so that people
> can easily pitch in to help with this later cleanup?

Even better would be to turn this report into a set of bugs filed
against the set of reverse dependencies which are made RC at the time
that the transition migrates.

[I'm personally slightly concerned about relaxing britney allowing
testing to get into unreleasable states; a flag to re-enable the old
behavoir late in release would probably be good.]


Don Armstrong

--
I leave the show floor, but not before a pack of caffeinated Jolt gum
is thrust at me by a hyperactive girl screaming, "Chew more! Do more!"
The American will to consume more and produce more personified in a
stick of gum. I grab it.
-- Chad Dickerson

http://www.donarmstrong.com http://rzlab.ucr.edu


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 06:52 AM
Steve Langasek
 
Default A hack to alleviate transitions in Britney; now what?

On Mon, Mar 16, 2009 at 12:48:22AM -0700, Don Armstrong wrote:

> [I'm personally slightly concerned about relaxing britney allowing
> testing to get into unreleasable states; a flag to re-enable the old
> behavoir late in release would probably be good.]

In practice, the release team has to do this at various points in the
release cycle anyway because the transitions become so entangled that
breaking something in testing, or removing a bunch of packages that we
intend to release with, are the only options. This approach at least
ensures that testing will remain installable and (presumably) useful during
the rocky transitions, merely requiring a bit of cleanup of old packages.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 10:48 AM
Adeodato Simó
 
Default A hack to alleviate transitions in Britney; now what?

* Steve Langasek [Sun, 15 Mar 2009 19:55:50 -0700]:

Hello, Steve.

> On Sun, Mar 15, 2009 at 04:44:08PM +0100, Adeodato Simó wrote:
> > Now, this has its own set of problems and caveats as well, since if you
> > don’t pay attention and take care of later cleanup, you end up with
> > packages in testing that do not belong to any source in testing, which
> > is bad.

> Will there be reports produced on a regular basis of the stale libraries in
> testing, and their reverse-dependencies, so that people can easily pitch in
> to help with this later cleanup?

There is a web page [1] that contains information about ongoing
transitions, including those that are in need on cleanup (libdvdread at
the moment). A list of packages is provided, and they are classified in
two groups: “Pending migration” (that is, they’ve been successfully
rebuilt in unstable), and “Not fixed in unstable”.

The first group is the responsibility of the Release Team, and they’re
most likely failing to migrate because of another transition (or, could
be, because of RC bugs, but in that case removal at some point would be
warranted).

The second group (which is empty in the case of libdvdread) are the ones
we can use help with. These will have RC bugs from day 0, and will be in
the list of transition blockers (http://snipr.com/transition-blockers).
Maybe once the transition is done, they should be moved to a separate
list, I don’t know. Thoughts?

Additionally, as I said in my original mail, the purpose of this is
mainly to break ties between transitions once they are ready, and by
definition a transition is not ready if still some packages are not
rebuilt in unstable. Meaning, there will be really few packages allowed
into this “second group”, if any, and removals from testing will be
preferred in that case.

Does this address your concerns?

[1]: https://buildd.debian.org/transitions/summary.html
(This page may move to release.debian.org eventually.)

* Don Armstrong [Mon, 16 Mar 2009 00:48:22 -0700]:

Hello, Don.

> On Sun, 15 Mar 2009, Steve Langasek wrote:
> > On Sun, Mar 15, 2009 at 04:44:08PM +0100, Adeodato Simó wrote:
> > > Now, this has its own set of problems and caveats as well, since
> > > if you don’t pay attention and take care of later cleanup, you end
> > > up with packages in testing that do not belong to any source in
> > > testing, which is bad.

> > Will there be reports produced on a regular basis of the stale
> > libraries in testing, and their reverse-dependencies, so that people
> > can easily pitch in to help with this later cleanup?

> Even better would be to turn this report into a set of bugs filed
> against the set of reverse dependencies which are made RC at the time
> that the transition migrates.

As said above, failures to build against the new library are RC from day
0, and the intention is not to do transitions while those are open,
other constraints permitting.

As for packages that are rebuilt in unstable but not migrated, I don’t
think RC bugs are approriate, since they’re not bugs in the package. We
have the above mentioned list, which I think should be enough (since I
don’t expect for those packages to remain without migrating for too long
periods of time).

> [I'm personally slightly concerned about relaxing britney allowing
> testing to get into unreleasable states; a flag to re-enable the old
> behavoir late in release would probably be good.]

In addition to what Steve explained about the inevitable necessity to
bend the rules for entangled transitions, let me clear up that this is
not any flag in britney that enables the behavior permanently or
globally. This applies to a transition on a case-by-case basis, with a
conscious decision and need for manual action. Also, it is my
expectation that the need for this will mostly disappear once we get
this first batch of transitions done.

Sounds good?

Thanks for your feedback,

--
- Are you sure we're good?
- Always.
-- Rory and Lorelai


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 11:34 AM
Richard Atterer
 
Default A hack to alleviate transitions in Britney; now what?

On Mon, Mar 16, 2009 at 12:48:22AM -0700, Don Armstrong wrote:
> [I'm personally slightly concerned about relaxing britney allowing
> testing to get into unreleasable states; a flag to re-enable the old
> behavoir late in release would probably be good.]

Adeodato's proposal makes a lot of sense, but still I agree with the above.
"Always in a releasable state" was a good design decision for testing, and
this change will muddy the idea a bit further.

At the very least, there should be an auto-generated web page listing
packages in testing that are currently unreleasable!


For a cleaner separation, testing could be split it two: The normal,
releasable testing works according to the strict rules as before. A second,
add-on repository (let's call it "mouldy";-) can realize Adeodato's idea:
It is only intended to be added to sources.list _in addition to testing_,
and contains out-of-date library packages and the programs which depend on
them.

Rather than removing packages from testing to make way for a new
transition, britney would move them over to "mouldy". This way, they would
still be available. At the same time, the fact that they moved there is a
clear sign to the respective maintainers that they need to do something to
get their packages into a releasable state.

Once the problem is resolved, those packages which only indirectly depended
on the library transition can be moved back from "mouldy" to testing
without recompilation.

I can't say I'm too much of an expert with these issues, so there may be
problems with this scheme. For example, it is possible that "mouldy" would
end up containing everything and testing would be empty, which would buy
you nothing.

Cheers,

Richard

--
__ _
|_) /| Richard Atterer | GnuPG key: 888354F7
| /| http://atterer.net | 08A9 7B7D 3D13 3EF2 3D25 D157 79E6 F6DC 8883 54F7
'`


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 05:17 PM
Raphael Geissert
 
Default A hack to alleviate transitions in Britney; now what?

Steve Langasek wrote:

> On Mon, Mar 16, 2009 at 12:48:22AM -0700, Don Armstrong wrote:
>
>> [I'm personally slightly concerned about relaxing britney allowing
>> testing to get into unreleasable states; a flag to re-enable the old
>> behavoir late in release would probably be good.]
>
> In practice, the release team has to do this at various points in the
> release cycle anyway because the transitions become so entangled that
> breaking something in testing, or removing a bunch of packages that we
> intend to release with, are the only options. This approach at least
> ensures that testing will remain installable and (presumably) useful
> during the rocky transitions, merely requiring a bit of cleanup of old
> packages.
>

Wouldn't it be better to remove the packages from testing? this way if the
library and other packages are ready to go they could easily migrate
without any special hack, if my understanding of the situation is correct.
This change would not affect users of the removed packages, because unless
the new library conflicts the old one, no package would break on installed
systems.
This schema is more or less what Richard Atterer is proposing, but instead
of solving the transition on the archive, it would be happening on the
user's machines.

What do you think?

Whatever happens, thanks for your work!

Cheers,
Raphael Geissert



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 08:52 PM
Steve Langasek
 
Default A hack to alleviate transitions in Britney; now what?

On Mon, Mar 16, 2009 at 12:17:28PM -0600, Raphael Geissert wrote:
> Steve Langasek wrote:

> > On Mon, Mar 16, 2009 at 12:48:22AM -0700, Don Armstrong wrote:

> >> [I'm personally slightly concerned about relaxing britney allowing
> >> testing to get into unreleasable states; a flag to re-enable the old
> >> behavoir late in release would probably be good.]

> > In practice, the release team has to do this at various points in the
> > release cycle anyway because the transitions become so entangled that
> > breaking something in testing, or removing a bunch of packages that we
> > intend to release with, are the only options. This approach at least
> > ensures that testing will remain installable and (presumably) useful
> > during the rocky transitions, merely requiring a bit of cleanup of old
> > packages.

> Wouldn't it be better to remove the packages from testing? this way if the
> library and other packages are ready to go they could easily migrate
> without any special hack, if my understanding of the situation is correct.

In some cases, if you want a completely consistent testing you have to
remove dozens of source packages for the benefit of one "extra" binary
package that's built from the same source as something important.

Removing GNOME from testing because something depends on libfrufru1 isn't a
win for testing's usability.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slangasek@ubuntu.com vorlon@debian.org


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 09:06 PM
Don Armstrong
 
Default A hack to alleviate transitions in Britney; now what?

On Mon, 16 Mar 2009, Adeodato Simó wrote:
> As said above, failures to build against the new library are RC from
> day 0, and the intention is not to do transitions while those are
> open, other constraints permitting.

Cool.

> As for packages that are rebuilt in unstable but not migrated, I
> don’t think RC bugs are approriate, since they’re not bugs in the
> package.

Right; I really only meant the cases in which someone has to do
something to clean up the transition. (For things that only the RMs
can do, it doesn't really make a difference to me how it's tracked,
though bugs in the BTS against an appropriate psuedopckage using the
'affects'[1] mechanism to indicate which packages have a problem would
help other people besides the RMs know what was going on.)

> In addition to what Steve explained about the inevitable necessity
> to bend the rules for entangled transitions, let me clear up that
> this is not any flag in britney that enables the behavior
> permanently or globally. This applies to a transition on a
> case-by-case basis, with a conscious decision and need for manual
> action. Also, it is my expectation that the need for this will
> mostly disappear once we get this first batch of transitions done.

That's good enough for me. I didn't understand that it involved manual
action.


Don Armstrong

1: This isn't working 100% yet; I hope to have an announcement about
it and summary shortly.
--
Science is a way of trying not to fool yourself. The first principle
is that you must not fool yourself, and you are the easiest person to
fool.
-- Richard Feynman "What is and What Should be the Role of Scientific
Culture in Modern Society"; 1964

http://www.donarmstrong.com http://rzlab.ucr.edu


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-16-2009, 11:32 PM
Raphael Geissert
 
Default A hack to alleviate transitions in Britney; now what?

Steve Langasek wrote:

> On Mon, Mar 16, 2009 at 12:17:28PM -0600, Raphael Geissert wrote:
>> Wouldn't it be better to remove the packages from testing? this way if
>> the library and other packages are ready to go they could easily migrate
>> without any special hack, if my understanding of the situation is
>> correct.
>
> In some cases, if you want a completely consistent testing you have to
> remove dozens of source packages for the benefit of one "extra" binary
> package that's built from the same source as something important.
>
> Removing GNOME from testing because something depends on libfrufru1 isn't
> a win for testing's usability.
>

It would only last until it is able to migrate without breaking anything. I
think this is just a matter of deciding which way is "less broken", i.e.
broken because some packages are removed, or broken because you have
multiple versions of the same libraries. IMHO the latter approach breaks
testing more than the former, because it is a matter of availability vs
duplicates (and if something goes wrong: installability).

Cheers,
Raphael Geissert



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 10:13 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org