FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 03-17-2009, 03:13 PM
Neil Williams
 
Default DEP-4: The TDeb specification.

(No need to CC: me on either debian-devel or debian-i18n, thanks.)

Time to launch the debate on the Draft TDeb Specification - DEP-4.

Current HTML form: http://people.debian.org/~codehelp/tdeb/

SGML form for modification: svn://svn.debian.org/dep/web/deps/
(Draft.sgml - please notify me if the SGML is modified so that I can
update the HTML generated from the SGML.)

DEP form: http://dep.debian.net/deps/dep4/
(not yet fully automated from SVN commits, hence not yet the preferred
form for modification - can be mostly generated from the SGML.)

Date: March 2009
Drivers: Neil Williams <codehelp@debian.org>
Joerg Jaspert <joerg@debian.org>
Thomas Viehmann <tv@beamnet.de>
Mark Hymers <mhy@debian.org>
Frank Lichtenheld <djpig@debian.org>

Abstract: This document provides an overview of the TDeb format, TDeb
design and usage. This specification should be considered as a work in
progress.

Purpose: The aim of the DEP is to improve and ratify sufficient
portions of the specification to allow Squeeze to be released with
support for TDebs in core packaging tools, ready for the creation of
the first TDebs in Squeeze+1.

Primary focus: Tool support in Squeeze. Initially, only debconf
translations will be targeted for TDebs in Squeeze+1 - package
translations for native packages should also be possible in Squeeze+1
but other translations are not likely to be implemented until
Squeeze+2. Please keep the DEP concentrating on the tools, not the later
implementation. There will be time to improve support in Squeeze+1 for
corner cases arising from packages looking for TDebs in Squeeze+2.

Primary Motivations (in order):
1. Updates to translations should not require source NMU's.
2. Translation data should not be distributed in
architecture-dependent packages.
3. Translators should have a common interface for getting updates
into Debian (possibly with automated TDeb generation after i18n team
review).

Secondary Benefits:
a) users will be able to selectively install package localisation
according to the supported locales on individual machines and Debian
will keep all such selective installations updated automatically,
including adding/removing translations for existing packages when the
locale configuration is modified.
b) easier, quicker and more thoroughly
reviewed translations of debconf, native package strings and other
translations.

Resources:
i:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/slides/TDebs_draft_specification/tdeb-fosdem-2009.pdf
ii:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/low/TDebs_draft_specification.ogg
iii:
http://meetings-archive.debian.net/pub/debian-meetings/2009/fosdem/high/TDebs_draft_specification.ogg

Code: git://git.debian.org/git/users/tviehmann/dpkg
git://git.debian.org/git/users/codehelp/debhelper

ToDo: +t1.diff.gz support - i.e. allowing dpkg and apt to look for and
apply the changes made by translators - only some elements of this need
to be implemented in Squeeze. apt also needs to understand locales and
download the relevant TDeb alongside the package so that
apt-extracttemplates can offer translated debconf questions upon first
installation.

Other packages: reprepro, langupdate (in NEW), ...

Limitations: At this stage, there is no need to discuss the finer
points of TDebs as they pertain to implementations not possible until
*after* the release of Squeeze, EXCEPT where those points have a direct
bearing on how tools like dpkg and apt handle tdebs in the
Squeeze->Squeeze+1 migrations. For this reason, there doesn't need to
be any definitive agreement about how TDebs will work with upstream
translations, KDE or non-gettext translations or anything else relating
to the actual use of TDebs after Squeeze during the ratification of
DEP-4 itself.

Relation to Emdebian: Emdebian has thousands of TDebs with a slightly
different emphasis - although you are welcome to use the
http://www.emdebian.org/locale/ repository as a test case, bear in mind
that Debian TDebs will differ internally and in numbers. Note also that
reprepro needs to support TDebs - it currently forces a renaming
to .deb - a bug report will follow in due course.

Numbers: The DebConf8 and Extremadura meetings established that the
number of TDebs does need to be careully managed, therefore TDebs will
be architecture-independent and source-package based. Some TDebs will
be very, very small - others could be quite large indeed. There is
scope in the specification for multiple TDebs where source packages
have debconf support and large amounts of translated documentation.

Policy: Elements of DEP-4 can be migrated into Policy discussions and
patches in due course - how that is actually done is up to maintainers
of debian-policy.

Discuss.

:-)

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
 
Old 03-17-2009, 10:22 PM
Loïc Minier
 
Default DEP-4: The TDeb specification.

> Current HTML form: http://people.debian.org/~codehelp/tdeb/

So do I understand correctly that there might be one .tdeb per package
and per language unless maintainers merge regularly the contents into
their source packages? How many .tdebs files does that imply in the
main archive?


It seems to me fetching multiple .tdeb for perhaps every .deb to update
has the potential to slow down an update by a huge factor, not only in
size of the tdeb index but for the amount of roundtrips.


Do you prevent mixing old and new .debs and .tdebs?


How do you merge data from a new package into the tdeb data?

--
Loïc Minier


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-17-2009, 10:47 PM
Neil Williams
 
Default DEP-4: The TDeb specification.

On Wed, 18 Mar 2009 00:22:19 +0100
Loïc Minier <lool@dooz.org> wrote:

> > Current HTML form: http://people.debian.org/~codehelp/tdeb/
>
> So do I understand correctly that there might be one .tdeb per package
> and per language unless maintainers merge regularly the contents into
> their source packages? How many .tdebs files does that imply in the
> main archive?

No, updates to translations will update the existing TDeb, creating
+t2.diff.gz and +t3.diff.gz etc. All supported languages go into the
existing TDeb, organised by locale root.

Unless a package needs more than one TDeb for the debconf+large_docs
corner case, each source package should only expect to have one TDeb
for all binary packages and all locales.

Translation teams can work together to make uploads in a coordinated
manner - similar to the current method of requesting deadlines for i18n
bugs, a nominated person can collate the various translations prior to
a deadline chosen by the teams themselves, according to the needs of
that particular package.

Updated TDebs use the +t1, +t2 version suffix etc.

http://people.debian.org/~codehelp/tdeb/ch2.html

> It seems to me fetching multiple .tdeb for perhaps every .deb to update
> has the potential to slow down an update by a huge factor, not only in
> size of the tdeb index but for the amount of roundtrips.

Generally, there will be only one TDeb.

> Do you prevent mixing old and new .debs and .tdebs?

Changes to translations use +t1.diff.gz etc.

> How do you merge data from a new package into the tdeb data?

The real question is how to get apt to understand getting the
+t1.diff.gz when asked for apt-get source and for dpkg to expect to
unpack +t1.diff.gz etc.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
 
Old 03-18-2009, 04:26 AM
Manoj Srivastava
 
Default DEP-4: The TDeb specification.

On Tue, Mar 17 2009, Neil Williams wrote:


>> Do you prevent mixing old and new .debs and .tdebs?
>
> Changes to translations use +t1.diff.gz etc.
>
>> How do you merge data from a new package into the tdeb data?
>
> The real question is how to get apt to understand getting the
> +t1.diff.gz when asked for apt-get source and for dpkg to expect to
> unpack +t1.diff.gz etc.

Also, currently one may put up a new version of a package into
experimental or people.debian.org; and it carries with it the older
translations. If the translations are separated, then the people.d.o
package will be useful only for testing by the speakers of the original
langiage the debconf templates were written in. This seems like a step
back

manoj
--
Keep your eyes wide open before marriage, half shut afterwards. Benjamin
Franklin
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-18-2009, 06:25 AM
Neil Williams
 
Default DEP-4: The TDeb specification.

On Wed, 18 Mar 2009 00:26:30 -0500
Manoj Srivastava <srivasta@debian.org> wrote:

> On Tue, Mar 17 2009, Neil Williams wrote:
>
> >> Do you prevent mixing old and new .debs and .tdebs?
> >
> > Changes to translations use +t1.diff.gz etc.
> >
> >> How do you merge data from a new package into the tdeb data?
> >
> > The real question is how to get apt to understand getting the
> > +t1.diff.gz when asked for apt-get source and for dpkg to expect to
> > unpack +t1.diff.gz etc.
>
> Also, currently one may put up a new version of a package into
> experimental or people.debian.org; and it carries with it the older
> translations. If the translations are separated, then the people.d.o
> package will be useful only for testing by the speakers of the original
> langiage the debconf templates were written in. This seems like a step
> back

No, the initial TDeb will be generated by the maintainer, effectively
+t0, so the maintainer uploads the initial TDeb containing whatever
translations are currently supported, putting the TDeb wherever you
put the binary and .dsc. It is up to the maintainer to incorporate any
+t1.diff.gz containing updated or new translations that may exist
already into each new Debian version. The other repository could also
simply copy any existing TDeb that does already exist alongside the test
package. (Being Arch:all, the TDeb is only included in the maintainer
upload or subsequent translator updates, buildd's don't need to worry
about TDebs at all.)

If the new version has changed translated strings then those will only
available in English until the +t1 TDeb arrives anyway - that doesn't
change. (English is always the language in which the templates must be
written initially - en_US at that.) There remains the current advice
that changes to the templates should always seek translation updates
prior to the upload of the initial TDeb by the maintainer. If
maintainers implement a string freeze and wait for translation updates
before uploading, the chances of a +t1.diff.gz being required by time
of the next release by the maintainer are lower.

Just like the dependencies of a package in experimental, the TDeb is
still available and can be updated by translators, the +t1.diff.gz is
available to the maintainer to incorporate into any new version and
debian/rules needs to include support for generating the TDeb for each
new version.

Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
using debhelper calls and uploading TDebs each time they would
currently upload any package that contains /usr/share/locale/LC_*/ etc.
Those TDebs are, effectively, +t0 - only updates by translators start
the +t1 sequence.

Maintainer uploads:
foo_1.2.3-4_amd64.deb
foo-tdeb_1.2.3-4_all.tdeb
foo-bar_1.2.3-4_amd64.deb
foo_1.2.3-4.diff.gz
foo_1.2.3.orig.tar.gz
foo_1.2.3-4.dsc
foo_1.2.3-4_amd64.changes

The foo-tdeb package will be listed in the .changes anyway so existing
tools will simply add it to the list of files to be uploaded to
ftp-master or wherever. foo-tdeb_1.2.3-4_all.tdeb is, effectively,
foo-tdeb_1.2.3-4+t0_all.tdeb

Translator updates arrive:
foo-tdeb_1.2.3-4+t1_all.tdeb
foo_1.2.3-4+t1.diff.gz
foo_1.2.3-4+t1.dsc
foo_1.2.3-4+t1_all.changes

Maintainer makes a new release foo_1.2.3-5 which incorporates the TDeb
changes in a similar manner to how an NMU is included today. All files
matching foo*1.2.3-4* are removed by dak when the new version is
uploaded. The updated translations now exist in
foo-tdeb_1.2.3-5_all.tdeb - uploaded by the maintainer and there is no
+t1.diff.gz or +t1_all.tdeb until the package translations need to be
touched again.

The key point is that a +t1 revision can happen *during a release
freeze* without touching the source, without changing any of the
binaries. Once the release is out and unstable is accessible again, the
maintainer adds +t1.diff.gz to their next upload. Done.

(Interesting point, if the current Debian version has already migrated
to testing, the +t1 TDeb will also need to migrate with the +t1.diff.gz
as source. Whether the rules should change for a TDeb migration needs
to be decided - it could be that TDebs have a shorter time in unstable
if the relevant version of the package is already in testing or even
be allowed to migrate immediately. That's all for Squeeze+1.)

The system is quite similar to how an NMU is currently done - without
the problems of a source rebuild - and maintainers can handle the +t1
in much the same manner to fit into their workflow. Whether a changelog
entry is needed is undecided, it would be courteous but not necessarily
mandatory, especially if no bug exists to be closed.

What remains to be resolved is how best to fit that
"maintainer incorporates the changes from the TDeb" stage is handled so
that it doesn't disturb maintainer workflow too much. The changes will
only be in the po/ files and consist of simply unpacking the
+t1.diff.gz on top of your existing checkout (and committing any
changes if you use an RCS).

None of this happens until Squeeze+1.

There is no particular need for any of these stages to require bugs to
be filed or closed, unless maintainers feel that this would be the best
way to be notified. We could have something like the lowNMU system to
indicate packages where translations can be updated without needing
bugs to be opened or closed. (maybe noTBUG). This allows more automation
on the translator side.

The implementation is being staggered - all packages using debconf first
(Squeeze +1), native packages containing gettext translations second
(Squeeze +1, possibly into Squeeze +2), non-native packages with the
Debian maintainer upstream next, finally moving on to the remainder.

If you don't use debconf and your packages do not handle .debs
directly (like apt or dpkg or reprepro), there is nothing that needs to
be done in your packages until Squeeze+2.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
 
Old 03-18-2009, 09:13 AM
Charles Plessy
 
Default DEP-4: The TDeb specification.

Le Wed, Mar 18, 2009 at 07:25:40AM +0000, Neil Williams a écrit :
>
> Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
> using debhelper calls and uploading TDebs each time they would
> currently upload any package that contains /usr/share/locale/LC_*/ etc.
> Those TDebs are, effectively, +t0 - only updates by translators start
> the +t1 sequence.
>
> Maintainer uploads:
> foo_1.2.3-4_amd64.deb
> foo-tdeb_1.2.3-4_all.tdeb
> foo-bar_1.2.3-4_amd64.deb
> foo_1.2.3-4.diff.gz
> foo_1.2.3.orig.tar.gz
> foo_1.2.3-4.dsc
> foo_1.2.3-4_amd64.changes

Hi Neil,

how do you coordinate your action with the project of using the source format
'3.0 (quilt)' by default in Squeeze +1?

Have a nice day,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 03-18-2009, 09:31 AM
Neil Williams
 
Default DEP-4: The TDeb specification.

On Wed, 18 Mar 2009 19:13:03 +0900
Charles Plessy <plessy@debian.org> wrote:

> Le Wed, Mar 18, 2009 at 07:25:40AM +0000, Neil Williams a écrit :
> >
> > Maintainers will be creating TDebs in Squeeze+1, using debian/rules,
> > using debhelper calls and uploading TDebs each time they would
> > currently upload any package that contains /usr/share/locale/LC_*/ etc.
> > Those TDebs are, effectively, +t0 - only updates by translators start
> > the +t1 sequence.
> >
> > Maintainer uploads:
> > foo_1.2.3-4_amd64.deb
> > foo-tdeb_1.2.3-4_all.tdeb
> > foo-bar_1.2.3-4_amd64.deb
> > foo_1.2.3-4.diff.gz
> > foo_1.2.3.orig.tar.gz
> > foo_1.2.3-4.dsc
> > foo_1.2.3-4_amd64.changes
>
> Hi Neil,
>
> how do you coordinate your action with the project of using the source format
> '3.0 (quilt)' by default in Squeeze +1?

Why should 3.0 be any more difficult than 1.0 or anything that follows?
(Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
has to deal with incorporating patches and changes from the BTS, so
+t1.diff.gz is no different. In Squeeze+1, the changes are restricted
to debian/po or po/ for native Debian packages so there is no need to
do anything with 3.0 until Squeeze+2.

What matters is that the maintainer gets the +t1.diff.gz and applies it
onto the source package prior to the next upload. It's no different to
how the same maintainer would handle a patch or new translations
file sent to the BTS. I'm quite sure various tools and scriptage will
be devised to help with different workflow patterns before Squeeze+2.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
 
Old 03-19-2009, 08:35 AM
Jan Hauke Rahm
 
Default DEP-4: The TDeb specification.

On Wed, Mar 18, 2009 at 10:31:29AM +0000, Neil Williams wrote:
> Why should 3.0 be any more difficult than 1.0 or anything that follows?
> (Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
> has to deal with incorporating patches and changes from the BTS, so
> +t1.diff.gz is no different. In Squeeze+1, the changes are restricted
> to debian/po or po/ for native Debian packages so there is no need to
> do anything with 3.0 until Squeeze+2.

So you already see a need to change TDebs again?

I'm definitely not on your level of dpkg knowledge, so please forgive me
if I'm talking nonsense here. But I feel like introducing a new diff.gz
after finally getting rid of those with 3.0 (quilt) is a bit of a
regression. Can't dpkg-source sort out the po files [1] and put them in
an additional package-version.tdebian.tar.(gz|bz2|lzma)? That way i18n
and translators can always download all translation files without any
code, then change them if needed and upload again (the tdeb would have
to be built before, of course). That would make tdebs only work with
source format 3.0 but since it's going to be default soon I don't see a
problem (but maybe the entire suggestion is nonsense ).

> What matters is that the maintainer gets the +t1.diff.gz and applies it
> onto the source package prior to the next upload. It's no different to
> how the same maintainer would handle a patch or new translations
> file sent to the BTS. I'm quite sure various tools and scriptage will
> be devised to help with different workflow patterns before Squeeze+2.

How is the maintainer supposed to get the +t1.diff.gz? Have dget, apt
et alii have to deal with multiple +tX.diff.gz or should the maintainer
look those up at PTS? I don't see the workflow here (and again I'd see a
win in a seperate tarball as suggested above since it's always one
tdebian.tar.compr).
Huh, thinking of it... A new tdeb uploaded by a translator overwrites
the old one? Then the +tX.diff.gz can be regenerated from the tdeb,
right?

I better stop here and wait for you to rip me up.

Hauke

[1] You have a pretty specific pattern to work on, right?
${top_srcdir}/po/
${top_srcdir}/po-*/
 
Old 03-19-2009, 06:54 PM
Neil Williams
 
Default DEP-4: The TDeb specification.

On Thu, 19 Mar 2009 10:35:59 +0100
Jan Hauke Rahm <info@jhr-online.de> wrote:

> On Wed, Mar 18, 2009 at 10:31:29AM +0000, Neil Williams wrote:
> > Why should 3.0 be any more difficult than 1.0 or anything that follows?
> > (Not that I have any particular desire to use 3.0 or quilt myself.) 3.0
> > has to deal with incorporating patches and changes from the BTS, so
> > +t1.diff.gz is no different. In Squeeze+1, the changes are restricted
> > to debian/po or po/ for native Debian packages so there is no need to
> > do anything with 3.0 until Squeeze+2.
>
> So you already see a need to change TDebs again?

No, just extending the use case from an initial debconf-only to other
packages in subsequent releases.

> I'm definitely not on your level of dpkg knowledge, so please forgive me
> if I'm talking nonsense here. But I feel like introducing a new diff.gz
> after finally getting rid of those with 3.0 (quilt) is a bit of a
> regression.

The whole point of TDebs is that there is *no change* to the source
package. The generation of TDebs needs to be separate from whatever
methods are used to generate the source package or rebuild the package.
It therefore has to use a separate diff/tarball that is outside the
existing Debian source package - otherwise, the archive becomes
inconsistent.

This isn't an issue to be fixed in dpkg, it's more of an issue for
ftp-master.

> Can't dpkg-source sort out the po files [1] and put them in
> an additional package-version.tdebian.tar.(gz|bz2|lzma)?

How does that differ from +t1.diff.gz ? - bearing in mind that nothing
about generated +t1.diff.gz is allowed to modify the source package
already in Debian. Translators deserve a full diff, not some tarball of
PO files without context.

> That way i18n
> and translators can always download all translation files without any
> code, then change them if needed and upload again (the tdeb would have
> to be built before, of course).

That would happen anyway with +t1.diff.gz - however - it is not good to
only package the PO or POT file and expect translators to be happy with
their lot. Translations often need to be tested and translators often
benefit from having the source code around to understand messages that
the upstream has not made particularly clear from their perspective.
Translators always need to have the full source available. Forcing
translators to only work from the PO or POT is tantamount to assuming
that translators never have any understanding of the mysteries of source
code and shouldn't need to see it. Many translators are also upstream
developers for other projects, there is no reason to deny them the full
source code to provide context and identify bugs. (e.g. strings that
contain translatable content but were not marked for translation and
therefore never appear in the PO or POT file.)

> That would make tdebs only work with
> source format 3.0 but since it's going to be default soon I don't see a
> problem (but maybe the entire suggestion is nonsense ).

I don't see that there is a need for a change simply because of 3.0.

> > What matters is that the maintainer gets the +t1.diff.gz and applies it
> > onto the source package prior to the next upload. It's no different to
> > how the same maintainer would handle a patch or new translations
> > file sent to the BTS. I'm quite sure various tools and scriptage will
> > be devised to help with different workflow patterns before Squeeze+2.
>
> How is the maintainer supposed to get the +t1.diff.gz? Have dget, apt
> et alii have to deal with multiple +tX.diff.gz or should the maintainer
> look those up at PTS? I don't see the workflow here (and again I'd see a
> win in a seperate tarball as suggested above since it's always one
> tdebian.tar.compr).

There's no need to see the workflow for this in any detail at this stage
- it doesn't apply until after Squeeze. However, the +t1.diff.gz will be
in the archive as normal, the various tools can be extended to look for
and apply the +t1.diff.gz

> Huh, thinking of it... A new tdeb uploaded by a translator overwrites
> the old one? Then the +tX.diff.gz can be regenerated from the tdeb,
> right?

It's not regenerated, foo_1.2.3-4+t2_all.tdeb replaces foo_1.2.3-4
+t1_all.tdeb in the same upload that replaces foo_1.2.3-4+t1.diff.gz
with foo_1.2.3-4+t2.diff.gz but the +t2.diff.gz is built by the
nominated person doing the upload on behalf of one or more translation
teams.

> [1] You have a pretty specific pattern to work on, right?
> ${top_srcdir}/po/
> ${top_srcdir}/po-*/

The first target is debian/po/ for debconf but the point is moot, the
'target' to which you refer should be '-R *' for reasons explained
above.

--


Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
 
Old 04-03-2011, 07:09 PM
Helge Kreutzmann
 
Default DEP-4: The TDeb specification.

Hello Neil,
On Tue, Mar 17, 2009 at 04:13:15PM +0000, Neil Williams wrote:
> Time to launch the debate on the Draft TDeb Specification - DEP-4.
>
> Current HTML form: http://people.debian.org/~codehelp/tdeb/

...

> Discuss.

just out of curiosity, is this still an active proposal, especially
since Lenny+1 has now been released?

Greetings

Helge

--
Dr. Helge Kreutzmann debian@helgefjell.de
Dipl.-Phys. http://www.helgefjell.de/debian.php
64bit GNU powered gpg signed mail preferred
Help keep free software "libre": http://www.ffii.de/
 

Thread Tools




All times are GMT. The time now is 07:56 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org