FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-30-2011, 02:18 PM
Stefano Zacchiroli
 
Default throw away debs and source only uploads

On Mon, Mar 28, 2011 at 03:37:05PM +0100, Mark Hymers wrote:
> Ok, the situation here is the following:

Thanks a lot for taking the time of clarifying.

> The main decision which needs to be made is whether, as a project, we
> want source only uploads or to throw away DD built non-all debs.
> There's not entire agreement amongst the ftpmasters about this (I err
> on the side of the source-only uploads to avoid making people waste
> time and bandwidth but that's not the only opinion).
<snip>
> Once a decision is made, implementation is easy, but I'd like some
> form of consensus before we go down that route.

So, let me see if I can help out in shaping the discussion around this.
First of all my gut feeling is the same as Don's [1]. I've the
impression that throw away debs is a rather uncontroversial step to take
and that we could take independently from source only uploads.

[1] http://lists.debian.org/debian-devel/2011/03/msg01063.html

Clearly that would not solve some of the issues that source only uploads
would solve, most notably the "wasted bandwidth" issue. At the same
time: 1) it won't be worse than the status quo in that respect either,
and 2) would be better than the status quo in terms of package
uniformity wrt their build environments.

I saw only one potential drawback in going forward with throw away debs,
namely the risk of doing some job to implement them and them throw it
away the day, potentially near, we further switch to source only
uploads. My interpretation of your mail is that this risk is pretty low,
as the involved work is low as well. (Yes, I've also seen the various
interesting proposals of comparing metadata of uploaded packages against
those of rebuilt packages and I understand that implementing those would
require significantly more work. But none of those proposals seem to be
*requirements* to go ahead.)

If my gut feeling will be shown to be wrong on the consensus around
throw away debs (with evidence coming as angry replies to this mail), I
propose to have a poll on throw away debs.

Bottom line #1: I propose to go ahead and deploy throw away debs, as
soon as technically feasible.

--------

> One objection to source-only is the perception that people will throw
> untested things at the buildds and see what sticks. That strikes me
> as a social problem, but as a project we probably want to think how to
> handle it.

Regarding source only upload, well, it's tricky. There is the usual
tension about the principle desire of trusting every DD to do the right
thing and the reality-check observation that enabling people to upload
only sources *will* mean that some people will upload packages without
having even built them once; let's call this latter problem "developer
sloppiness". (Shameless [academic] plug: developer sloppiness is fairly
common all over computer science. That's part of the reason why we have
developed over time programming languages which are more and more strict
in *forcing* developers to do the rightâ„¢ thing, at least by default.)

The main use case I've seen mentioned on list to favor source only
uploads over throw away debs is that of "low bandwidth" or "bandwidth
limits". Most likely, that use case applies to very few people and the
vast majority of uploads could happen without suffering of those
problems.

To address developer sloppiness, sane defaults could be enough. If this
is the case, a solution that might work is a scheme in which source only
uploads are disallowed by default, but at the same time can be enabled
if the uploader really needs to. If the needed explicit step to have a
source only upload gets through is "costly" enough, sloppy developers
won't implement it (after all they are sloppy, aren't they?) and we
should be able to avoid a good deal of gratuitous additional FTBFS.

An implementation which might cut the deal can rely on lintian. We might
for instance have a lintian error which is triggered by a changes file
missing .deb files and have such an error be valid ground for automatic
rejection by dak. A maintainer who really needs to do a source only
upload for that package will simply have to add the proper override.
Bonus features: maintainers which rely "too much" on source only uploads
(assuming that can be defined...) can be easily spotted as the overrides
are in the archive. Drawback of this solution: overrides will be per
package and will have the tendency of stick around packages; that should
not be a big deal either, assuming the current defaults and recommended
practices of dpkg-buildpackage & friend will still be about building
binary packages by default.

The above is just an idea, little more than a brain-dump, for finding a
compromise among the real needs of people with bandwidth problem and the
social issues revolving around developer sloppiness.

Once more: it's material for discussion which IMHO should not be in the
way of the implementation of the throw away debs scheme.


Cheers.

--
Stefano Zacchiroli -o- PhD in Computer Science PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela .......| ..: |.......... -- C. Adams
 
Old 03-30-2011, 11:04 PM
Paul Wise
 
Default throw away debs and source only uploads

I definitely agree we want to throw away developer-built debs (arch
all & arch any) in almost all situations.

I don't think I would want the lintian solution for source-only
uploads, I would prefer something on a per-upload basis that requires
an explicit human decision per-upload rather than something that can
be scripted away. It is also important to have an audit trail for
this.

Maybe a mail bot on ftp-master that a developer needs to mail before
the upload will be accepted. The overrides could be published on the
ftp-master website and replies to them be CCed to -devel or another
list.

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: AANLkTinjBCS0eR+oiTXVbiTG_fFt2RAWb+07QP=t=gqx@mail .gmail.com">http://lists.debian.org/AANLkTinjBCS0eR+oiTXVbiTG_fFt2RAWb+07QP=t=gqx@mail .gmail.com
 
Old 03-30-2011, 11:15 PM
Russ Allbery
 
Default throw away debs and source only uploads

Paul Wise <pabs@debian.org> writes:

> I definitely agree we want to throw away developer-built debs (arch all
> & arch any) in almost all situations.

> I don't think I would want the lintian solution for source-only uploads,
> I would prefer something on a per-upload basis that requires an explicit
> human decision per-upload rather than something that can be scripted
> away. It is also important to have an audit trail for this.

We could add another queue similar to the DELAYED queues, maybe.

--
Russ Allbery (rra@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 877hbgkxnc.fsf@windlord.stanford.edu">http://lists.debian.org/877hbgkxnc.fsf@windlord.stanford.edu
 
Old 03-31-2011, 12:17 AM
Charles Plessy
 
Default throw away debs and source only uploads

Le Wed, Mar 30, 2011 at 04:18:34PM +0200, Stefano Zacchiroli a écrit :
>
> Regarding source only upload, well, it's tricky. There is the usual
> tension about the principle desire of trusting every DD to do the right
> thing and the reality-check observation that enabling people to upload
> only sources *will* mean that some people will upload packages without
> having even built them once

Hi Stefano,

I think that one important factor is how often such errors will happen. We can
imagine all sorts of scenarios and devise counter-mesures against them, but are
they all worth the effort, and worth the damage as well ? Because it is always
frustrating to read top-down comments about simple Debian developers to be
sloppy and untrustable. Let's not make it a common assumption.

In what I have seen in the packaging teams that I follow, often when a package
fails to build on all architectures, it is because the developer has not tested
them in a minimal build environment. Making sure that binary packages have been
built together with the source package is not solving that problem.

On the binary side as well, what the maintainer can do to test the packages by
hand is also limited, not to mention that testing more than one architecture is
time consuming (so I usually never do). Build-time regression tests and
facilities to let users run the same tests on the binary packages they
downloaded (DEP8 ?) will altogether do much more for the quality of Debian than
using the presence of binary packages accompaniying the source upload as an
evidence of significant qualitative difference compared to a source-only upload.

Asking developers to publish their build logs is far more interesting, and in
the short term, does not require any infrastructure change. Currently, I store
my build logs in the git repositories where my source packages are managed, and
in the case of subversion packages, I send the logs on the maintainer mailing
list. If the uploaded package turns out to be problematic, we have a starting
point for troubleshooting.

And when developers repeatedly commit the same mistake, refuse to realise the
damage they do, and persist in ignoring the solution to the problems they
cause, let'se withdraw the trust we gave them. But does that happen that
often*? My experience is that people learn and improve their skills, not the
reverse. In that sense, occasional failures to build, while not a goal by
themselves, are an inevitable byproduct of increasing our workforce. Automated
reporting of build failures can circumvent the nuisance to the package
maintainers themselves.

Cheers,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110331001718.GA734@merveille.plessy.net">http://lists.debian.org/20110331001718.GA734@merveille.plessy.net
 
Old 03-31-2011, 12:54 PM
Ian Jackson
 
Default throw away debs and source only uploads

Lars Wirzenius writes ("Re: throw away debs and source only uploads"):
> Most uploads are done using dput or dupload. We could add code to them
> to verify that there's binaries corresponding to the source that is
> about to be built.

We could have the archive scripts insist that the .debs have to exist
in the .changes file, but not to check that the files are actually
uploaded or to accept empty files.

If we want to check the manifest we could require that the .changes
file list manifests (output of dpkg-deb --contents, or something) for
any missing .debs.

Ian.


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 19860.31129.223881.836109@chiark.greenend.org.uk"> http://lists.debian.org/19860.31129.223881.836109@chiark.greenend.org.uk
 
Old 03-31-2011, 01:28 PM
Joachim Breitner
 
Default throw away debs and source only uploads

Hi,

Am Mittwoch, den 30.03.2011, 16:18 +0200 schrieb Stefano Zacchiroli:
> The main use case I've seen mentioned on list to favor source only
> uploads over throw away debs is that of "low bandwidth" or "bandwidth
> limits". Most likely, that use case applies to very few people and the
> vast majority of uploads could happen without suffering of those
> problems.

I have another use case: Uploading packages with little (typos, changes
in debian/control) or no (aka rebuild triggering) changes, but applying
such a change to 200 packages – something that comes up with the Haskell
packaging team often. Often I am confident that my change is correct
even without manual testing. Nevertheless I have to fire up a pbuilder,
build several dozen packages in the right order (something that
wanna-build is much better in figuring out), some of which build fast
and some take quite some time. A lot of work goes into useless manual
labor here despite the fact that we (almost) have the infrastructure to
automate that.

I, for one, welcome our new alien ant^W^W source only uploads.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 
Old 04-07-2011, 02:55 PM
Stefano Zacchiroli
 
Default throw away debs and source only uploads

[ Bcc:-ing ftpmasters ]

Time to wrap up the current state of this discussion, at least as far as
I see it.

- going ahead with throw away debs seems to be largely uncontroversial;
can we haz zem please? :-)

- there seems to be no substantial objections either on the fact the
source only upload would be fine, as long as they are not the default
but can be enabled upon request


What's still largely undecided is how uploaders would require a source
only upload. Several presented use cases --- both here on list and in
private replies to me --- show that my proposal, as well as all possible
source-based solutions, are suboptimal. In other words, people seem to
really need per-upload, or even per-batch upload, white listing.

FWIW, I wouldn't like much the idea of introducing yet another list of
people which are allowed to do source only uploads as that would be a
potential bottleneck which seems unwarranted here (at least if you buy
my argument that for what concerns the risk of non-buildable-uploads,
the defaults matter more than strict controls).

How about just relying on "dpkg-buildpackage -S", maybe coupling it with
the need of using "dput -f" (extending a bit the current semantics of
-f) which would refuse to upload a source only binary by default?

Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela .......| ..: |.......... -- C. Adams
 
Old 04-17-2011, 09:20 AM
Stefano Zacchiroli
 
Default throw away debs and source only uploads

On Wed, Mar 30, 2011 at 04:18:34PM +0200, Stefano Zacchiroli wrote:
> > The main decision which needs to be made is whether, as a project, we
> > want source only uploads or to throw away DD built non-all debs.
> > There's not entire agreement amongst the ftpmasters about this (I err
> > on the side of the source-only uploads to avoid making people waste
> > time and bandwidth but that's not the only opinion).
> <snip>
> > Once a decision is made, implementation is easy, but I'd like some
> > form of consensus before we go down that route.

Further round of update on this one, after some more discussion on list
and a brief chat of mine with Mark.

- There seems to be consensus to go ahead with throw-away debs; they
require a bit of work though so either be patient or, better,
volunteer with FTP masters to help out with the implementation of the
remaining bits.

- There seems to be consensus also on going ahead with source-only
uploads, as long as they are not the default. The override method to
enable them is still undecided though. A possibility is to have an
explicit extra field in .changes file to state "yes, I'm sure I want
to do a source-only upload"; another is a simple override at dput
upload time. FTP master will discuss the various possibilities and let
us now.

Either way, we will need to document in devref that the recommended
way is to do binary-ful uploads and how to override if needed. (A bug
report is in order, but it's pointless to submit it before we have
further technical details.)

Cheers.
--
Stefano Zacchiroli -o- PhD in Computer Science PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Quando anche i santi ti voltano le spalle, | . |. I've fans everywhere
ti resta John Fante -- V. Capossela .......| ..: |.......... -- C. Adams
 
Old 04-17-2011, 10:34 AM
Michael Banck
 
Default throw away debs and source only uploads

On Thu, Apr 07, 2011 at 04:55:12PM +0200, Stefano Zacchiroli wrote:
> - going ahead with throw away debs seems to be largely uncontroversial;
> can we haz zem please? :-)

Will that throw away Arch: all packages as well? If there are no
technical issues/implementation missing with this (somebody mentioned
that), I would suggest doing that as well, for consistency (having all
build logs on buildd.d.o, e.g.), and because we've seen breakage due to
that from time to time.


Michael


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110417103406.GA31310@nighthawk.chemicalconnectio n.dyndns.org">http://lists.debian.org/20110417103406.GA31310@nighthawk.chemicalconnectio n.dyndns.org
 
Old 04-17-2011, 11:02 AM
Joachim Breitner
 
Default throw away debs and source only uploads

Hi,

Am Sonntag, den 17.04.2011, 11:20 +0200 schrieb Stefano Zacchiroli:
> - There seems to be consensus to go ahead with throw-away debs; they
> require a bit of work though so either be patient or, better,
> volunteer with FTP masters to help out with the implementation of the
> remaining bits.

will this infrastructure then also be able to do archAllBinNMUs, for
cases when problems in arch:all packages can be fixed by simply
rebuilding without source changes? That would be nice to have.

Although it would come almost for free with source-only-uploads, as then
one can mechanically fetch the latest source, add a changelog entry and
upload the otherwise unmodified source. But still, something even more
automatic is cleaner and less error prone.

In any case, I am looking forward to these changes, and kudos in advance
to anyone implementing them.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nomeata@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nomeata@joachim-breitner.de | http://people.debian.org/~nomeata
 

Thread Tools




All times are GMT. The time now is 12:55 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org