FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian dpkg

 
 
LinkBack Thread Tools
 
Old 04-29-2011, 04:03 PM
Dustin Kirkland
 
Default dotdee: a proposal for improving conffile management in Debian

Howdy debian-dpkg!

First, please forgive that this email comes from a Debian-unknown, and
a Canonical/Ubuntu core developer at that. But I promise I've been
around Debian packaging, conffile management, and archive
administration for a few years now ;-)

BACKGROUND

Having read through some of debian-dpkg's mailing list archives, I
think the issues around conffile management are quite well understood
by this audience. For those lacking background, I have encapsulated
this same proposal in a detailed blog post (the relevant bits of which
I will duplicate here):
* http://blog.dustinkirkland.com/2011/04/dotdee-modern-proposal-for-improving.html

As Ubuntu Server Core Developer and Packager, I'm quite keen on
improving two problems we encounter frequently, with respect to
conffiles:

1. Particularly in modern, massive Debian/Ubuntu deployments (e.g.
Cloud, Grid, and Cluster computing), it's no longer humans that are
managing individual systems, but rather systems (puppet, chef,
cfengine, et al.) managing systems. (Insert your Skynet jokes
here...) As such, it's difficult (if not impossible) for a Debian
package or a distribution to make configuration changes to another
package's conffiles without violating Debian policy -- even when the
change might objectively be the "right" or the "best" thing to do, in
terms of end user experience and ensuring proper package operation
(especially when the given system is *only* ever managed by such a
configuration management system).

2. In other cases, one local package has a run-time dependency on a
second package on the local system, but requires the package it
depends on to be configured in a particular way. Again, if that
configuration lives in a conffile owned by the second package, the
first package cannot automatically make that configuration change
without violating said policy.

As an exception, rather than the rule, there are a couple of very
mature packages that provide smart frameworks enabling system
administrators, packagers, and distributions to configure them all
within policy.

A good example would be apache2's /etc/apache2 directory, which allows
for admins, packagers, and distributions to insert their own
configuration modifications as files (or symbolic links) into sourced
directories such as /etc/apache2/conf.d, /etc/apache2/mods-available,
and /etc/apache2/sites-available.

The concept of a ".d" directory in /etc is very well understood in
most Linux/UNIX circles, actually.

PROPOSAL

Here, I'm proposing a tool called "dotdee" that I think would greatly
benefit Debian and Debian-derived distributions such as Ubuntu. Its
stated goal is to turn any given flat file on your system to a
dynamically concatenated flat file generated from a unique ".d"
directory dedicated to that particular file. With such a dedicated
and well-formed directory, system administrators, Debian packagers,
and distributions could conveniently place additional configuration
snippets in particular conffile's dedicated ".d" directory.

I have written a first prototype of the tool dotdee (snapshot
attached), or to view the latest under revision control, see:
* http://bazaar.launchpad.net/~kirkland/+junk/dotdee/view/head:/dotdee

It's a very simple, straightforward shell script, inspired a bit by
Debian's incredibly useful update-alternatives tool. I've only
invested a couple of hours into it at this point; just enough to prove
the concept and demonstrate its operation. Should we agree upon a
complete design or blueprint, I would gladly rewrite it as necessary
(in shell, python, or C).

The script runs in 3 different modes:

1. sudo dotdee --setup /etc/path/to/some.conf
2. sudo dotdee --update /etc/path/to/some.conf
3. sudo dotdee --undo /etc/path/to/some.conf


SETUP MODE

First the setup mode takes a flat file as a target. Assuming the file
is not already managed by dotdee, a new directory structure is created
under /etc/dotdee. In the example above, that would be
/etc/dotdee/etc/path/to/some.conf.d. So "/etc/dotdee" is prepended,
and ".d" is appended to the path which is to be managed. It's trivial
to get back to the name of the managed file by stripping /etc/dotdee
from the head, and .d from the tail of the string.

Next, the actual managed flat file is moved from
/etc/path/to/some.conf to
/etc/dotdee/path/to/some.conf.d/50-dpkg-original. Again, this a
well-formed path, with "/etc/dotee" prepended, a ".d" appended, and
the file itself is renamed to "50-dpkg-original". This is intended to
clearly denote that this is the original, base file, as installed by
dpkg itself. The number "50" is precisely halfway between "00" and
"99", leaving plenty of remove for other file snippets to be placed in
ordered positions before and/or after the original file.

After this, we run the update function, which will concatenate in
alphanumeric order all of the files in
/etc/dotdee/etc/path/to/some.conf.d/* and write the output into
/etc/dotdee/etc/path/to/some.conf.

Finally, the update-alternatives system is used to place a symlink at
the original location, /etc/path/to/some.conf, pointing to
/etc/dotdee/etc/path/to/some.conf. Additionally, a second,
lower-priority alternative is also set, pointing to dpkg's original at
/etc/dotdee/path/to/some.conf.d/50-dpkg-original.

UPDATE MODE

As mentioned above, the update function performs the concatenation
immediately, building the targeted path from its dotdee managed
collection of snippets. This should be run anytime a file is added,
removed, or modified in the dotdee directory for a given managed file.
As a convenience, this could easily and automatically be performed by
an inotify watch of the /etc/dotdee directory. That, itself, would be
a dotdee configuration option.

UNDO MODE

The undo function is something I added for my own development and
debugging while working on the tool, however I quickly realized that
it might be an important tool for other system administrators and
packagers (think, postrm maintenance scripts!).

DPKG INTEGRATION

This approach would require some integration with dpkg itself. On
package upgrade/installation, dpkg would need to need to detect when
the target path of a file it wants to create/update is in fact a
symbolic link referencing an /etc/dotdee path. It would need to drill
down into that path and place the file it wants to write on top of the
50-dpkg-original file instead.

IN PRACTICE

So what would this look like in practice?

Once integrated with dpkg, I'd like dotdee to be a utility that human
system administrators could run to manually turn a generic conffile
into a ".d" style configuration directory, such that they could append
their own changes to some numbered file in the dotdee directory, avoid
the interactive dpkg-conffile-changed prompts.

More importantly, I would like one package's postinst maintainer
script to be able take another package that it depends upon and turn
its conffile into a dotdee managed file, such that it could append or
prepend configuration information necessary for proper operation.
This, of course, would require significant buy-in from Debian, and
entail various appropriate policy updates.

In terms of supported configuration file types, dotdee would
eventually need some code for handling some particular configuration
file types. The current, basic implementation works well for
sequentially evaluated file types (like sourced shell scripts), python
config parser, and even windows ini file syntax. On the other hand,
something like XML would not immediately work in the current dotdee
implementation. For that, I've thought a bit about a similar
approach, constructing the conffile from a quilt directory of numbered
patch files. If you're interested in this approach, I can describe
that in more detail, too, and perhaps implement another prototype.

COMMENTS?

I plan to lead a session on this topic at the Ubuntu Developer Summit
in May 2011 in Budapest, and will certainly capture that feedback and
summarize it here for this audience.

But in the mean time, what do you think? Have you encountered similar
problems before? What other approaches have been taken to try and
solve this? What parts of this proposal do you think are reasonable?
Are any parts completely unreasonable? Are there extensions or
changes you propose?

Cheers,
--
:-Dustin

Dustin Kirkland
Ubuntu Core Developer
 
Old 04-29-2011, 08:04 PM
Jonathan Nieder
 
Default dotdee: a proposal for improving conffile management in Debian

Hi,

Dustin Kirkland wrote:

> DPKG INTEGRATION
>
> This approach would require some integration with dpkg itself. On
> package upgrade/installation, dpkg would need to need to detect when
> the target path of a file it wants to create/update is in fact a
> symbolic link referencing an /etc/dotdee path. It would need to drill
> down into that path and place the file it wants to write on top of the
> 50-dpkg-original file instead.

It sounds like this needs a hook to rename (divert) conffiles, plus a
hook to regenerate the combined conffile after "dpkg --configure"
finishes and before the package is considered configured.

(Of course, when I say "divert" this of course has nothing to do with
dpkg-divert. I am imagining something like

--conffile-name-filter=command
Set a command to be run via “sh -c” that maps
standard names of conffiles to the names of conffiles
dpkg should manage on the filesystem. This command
would receive a newline-separated [or NUL-separated?
-jn] list of filenames on its standard input and
should write a list of modified names to standard
input.

The intended use is by tools that want the pristine
conffiles managed by dpkg to be diverted to some other
place and used to write the actual conffiles in place.
Goes nicely with the --post-deferred-configure hook.

--post-deferred-configure=command
Set a command to be run via “sh -c” after conffiles
have been committed to disk but before the package
has entered the half-configured state. The intended
use is by tools that want to make some modifications
to changed conffiles before “postinst configure” runs.

Please do not take the details too seriously, of course.)

What do you think? Would you be interested in working on those
things? I imagine the result could be useful for other tools layered
on top of dpkg --- for example, it could be used to provide a nicer
interface for merging changes when /etc is kept in a VCS.

Hope that helps,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110429200458.GB32488@elie">http://lists.debian.org/20110429200458.GB32488@elie
 
Old 05-03-2011, 06:30 AM
Guillem Jover
 
Default dotdee: a proposal for improving conffile management in Debian

Hi!

On Fri, 2011-04-29 at 11:03:39 -0500, Dustin Kirkland wrote:
> IN PRACTICE

> Once integrated with dpkg, I'd like dotdee to be a utility that human
> system administrators could run to manually turn a generic conffile
> into a ".d" style configuration directory, such that they could append
> their own changes to some numbered file in the dotdee directory, avoid
> the interactive dpkg-conffile-changed prompts.

This does not take into account admins dropping the file under the
directory and then the package picking up the new file automatically,
they have to generate it manually afterwards. You mention apache, and
its framework (another good example is pam), but I don't think that
much is usually needed, just having the application support .d config
directories should be enough, and that usually implies less than around
50 lines of C. This also benefits upstream and all its downstreams,
and people building directly from sources. Also this implies the
configuration file cannot be directly edited, and if done so the
changes will get lost on the next regeneration.

As a counter example, a package that provides something similar to
dotdee, there's exim4. And while I think the efforts by their
maintainers to provide a more manageable way to configure it are
worthwhile, it's also one of the reasons I always use the user managed
single concatenated file. It's too annoying modifying a file and
forgetting to run update-exim4.conf. At some point in the future I
should just sit down and propose a patch for native .d support.

In addition the combination of "diverting" the conffile, merging it
back by a tool, and then using alternatives, seems pretty convoluted
and a quite possible source of confusion and fragility.

> More importantly, I would like one package's postinst maintainer
> script to be able take another package that it depends upon and turn
> its conffile into a dotdee managed file, such that it could append or
> prepend configuration information necessary for proper operation.
> This, of course, would require significant buy-in from Debian, and
> entail various appropriate policy updates.

That in essence is just diverting conffiles, and I think it's at least
for distributions a bad idea, for personal use I don't see any problem
with it though. If a package needs to modify the behaviour of another
one it should do it in concordance with the other package, and not
under its feet. Usually in that case the package provides an interface
to modify the configuration file, or provides a .d config directory.
That should currenty be described already in policy.

> In terms of supported configuration file types, dotdee would
> eventually need some code for handling some particular configuration
> file types. The current, basic implementation works well for
> sequentially evaluated file types (like sourced shell scripts), python
> config parser, and even windows ini file syntax. On the other hand,
> something like XML would not immediately work in the current dotdee
> implementation. For that, I've thought a bit about a similar
> approach, constructing the conffile from a quilt directory of numbered
> patch files. If you're interested in this approach, I can describe
> that in more detail, too, and perhaps implement another prototype.

To be honest, that sounds like a hack to me. But if you are
interested to still go this route, then maybe Config::Model might be
useful to you? In any case, the application handling the configuration
files is in the best place to know its internal layout, and how to
handle it.


So all in all, I don't think this is the right way to solve the problem
you describe. And while I also think the conffile handling in dpkg can
be improved, in this case the correct solution seems to me, to be to
add .d support to the needed applications.


thanks,
guillem


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110503063000.GA9447@gaara.hadrons.org">http://lists.debian.org/20110503063000.GA9447@gaara.hadrons.org
 
Old 05-04-2011, 04:49 PM
Dustin Kirkland
 
Default dotdee: a proposal for improving conffile management in Debian

On Fri, Apr 29, 2011 at 3:04 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Dustin Kirkland wrote:
>
>> DPKG INTEGRATION
>>
>> This approach would require some integration with dpkg itself. *On
>> package upgrade/installation, dpkg would need to need to detect when
>> the target path of a file it wants to create/update is in fact a
>> symbolic link referencing an /etc/dotdee path. *It would need to drill
>> down into that path and place the file it wants to write on top of the
>> 50-dpkg-original file instead.
>
> It sounds like this needs a hook to rename (divert) conffiles, plus a
> hook to regenerate the combined conffile after "dpkg --configure"
> finishes and before the package is considered configured.

Right. That's more or less my thinking ;-) A couple of triggers that
are called by dpkg at the right time (or perhaps even happen
automagically via inotify).

> (Of course, when I say "divert" this of course has nothing to do with
> dpkg-divert. *I am imagining something like
>
> * * * *--conffile-name-filter=command
> * * * * * * * *Set a command to be run via “sh -c” that maps
> * * * * * * * *standard names of conffiles to the names of conffiles
> * * * * * * * *dpkg should manage on the filesystem. *This command
> * * * * * * * *would receive a newline-separated [or NUL-separated?
> * * * * * * * *-jn] list of filenames on its standard input and
> * * * * * * * *should write a list of modified names to standard
> * * * * * * * *input.
>
> * * * * * * * *The intended use is by tools that want the pristine
> * * * * * * * *conffiles managed by dpkg to be diverted to some other
> * * * * * * * *place and used to write the actual conffiles in place.
> * * * * * * * *Goes nicely with the --post-deferred-configure hook.
>
> * * * *--post-deferred-configure=command
> * * * * * * * *Set a command to be run via “sh -c” after conffiles
> * * * * * * * *have been committed to disk but before the package
> * * * * * * * *has entered the half-configured state. *The intended
> * * * * * * * *use is by tools that want to make some modifications
> * * * * * * * *to changed conffiles before “postinst configure” runs.
>
> Please do not take the details too seriously, of course.)
>
> What do you think? *Would you be interested in working on those
> things? *I imagine the result could be useful for other tools layered
> on top of dpkg --- for example, it could be used to provide a nicer
> interface for merging changes when /etc is kept in a VCS.

Hi Jonathan,

As you said the details aren't entirely flushed out, but yes, these
are definitely along the lines of what I'm thinking -- a conffile map,
and a trigger that "builds" conffiles by intelligently combining the
original packaged conffile, and inserting (via cat, patch, VCS, or
something else) both local user changes, and allowable changes by
other packages that require this package to be configured in a
particular way for proper operation.

Would you be willing to join us via IRC and/or live internet audio
stream for 1 hour next week, as we discuss this at the Ubuntu
Developer Summit in Budapest? If you let me know what times (UTC) and
days are good for you, I can try to get this session schedule
appropriately.

I am absolutely, keenly interested in seeing this solved for both
Debian and Ubuntu. I'm willing and able to work on this myself, and
I'm willing and able to delegate the work to capable members on my
team. I'd like to hold off on both of those until we have an agreed
upon design, but I'm very much hoping to see this work through in the
coming months, if in any way possible.

Cheers,
--
:-Dustin

Dustin Kirkland
Ubuntu Core Developer


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTinSTRVV_Jib+vh0_uAZRx8MiYbWzQ@mail.gmail.com ">http://lists.debian.org/BANLkTinSTRVV_Jib+vh0_uAZRx8MiYbWzQ@mail.gmail.com
 
Old 05-04-2011, 05:30 PM
Dustin Kirkland
 
Default dotdee: a proposal for improving conffile management in Debian

On Tue, May 3, 2011 at 1:30 AM, Guillem Jover <guillem@debian.org> wrote:
> Hi!
>
> On Fri, 2011-04-29 at 11:03:39 -0500, Dustin Kirkland wrote:
>> IN PRACTICE
>
>> Once integrated with dpkg, I'd like dotdee to be a utility that human
>> system administrators could run to manually turn a generic conffile
>> into a ".d" style configuration directory, such that they could append
>> their own changes to some numbered file in the dotdee directory, avoid
>> the interactive dpkg-conffile-changed prompts.
>
> This does not take into account admins dropping the file under the
> directory and then the package picking up the new file automatically,
> they have to generate it manually afterwards. You mention apache, and
> its framework (another good example is pam), but I don't think that
> much is usually needed, just having the application support .d config
> directories should be enough, and that usually implies less than around
> 50 lines of C. This also benefits upstream and all its downstreams,
> and people building directly from sources. Also this implies the
> configuration file cannot be directly edited, and if done so the
> changes will get lost on the next regeneration.
>
> As a counter example, a package that provides something similar to
> dotdee, there's exim4. And while I think the efforts by their
> maintainers to provide a more manageable way to configure it are
> worthwhile, it's also one of the reasons I always use the user managed
> single concatenated file. It's too annoying modifying a file and
> forgetting to run update-exim4.conf.

Postfix, too. I am certainly familiar with this problem :-)

> At some point in the future I
> should just sit down and propose a patch for native .d support.

I've had that feeling, I know. And I'm feeling that way about more
and more and more utilities. I just don't think I, you, or any of us
is going to actually getting around to "fixing" every service to
support a .d style configuration structure.

And that's at the crux of this discussion (whether it's dotdee that
solves this, or something else completely different). I'm
wondering/hoping/thinking about other ways we could approach this
problem, as a distribution, and talking to you, the maintainers of the
dpkg system itself. The landscape has changed quite a bit since the
original dpkg configuration file management policies were written.
a) there's what, 20K or 30K packages in Debian now?
b) people are deploying 10s of thousands of "managed" Debian and
Ubuntu servers in clouds, grids, and other data centers
c) these systems are managed by things like puppet, chef, and other
management systems, above and beyond individual package management

> In addition the combination of "diverting" the conffile, merging it
> back by a tool, and then using alternatives, seems pretty convoluted
> and a quite possible source of confusion and fragility.

Agreed, especially in the near term, it will be confusing to some.

The update-alternative part of dotdee's implementation is not entirely
necessary. It was simply used to make the alternatives discoverable,
and to manage the symlinks themselves in a well defined manner.

Further out, though, I think there's ample opportunity for
documentation and education.

>> More importantly, I would like one package's postinst maintainer
>> script to be able take another package that it depends upon and turn
>> its conffile into a dotdee managed file, such that it could append or
>> prepend configuration information necessary for proper operation.
>> This, of course, would require significant buy-in from Debian, and
>> entail various appropriate policy updates.
>
> That in essence is just diverting conffiles, and I think it's at least
> for distributions a bad idea, for personal use I don't see any problem
> with it though. If a package needs to modify the behaviour of another
> one it should do it in concordance with the other package, and not
> under its feet. Usually in that case the package provides an interface
> to modify the configuration file, or provides a .d config directory.
> That should currenty be described already in policy.

Understood. It's that part of the policy that I would like to see a
tool designed to work within the spirit of, but perhaps modernized and
extended a bit.

>> In terms of supported configuration file types, dotdee would
>> eventually need some code for handling some particular configuration
>> file types. *The current, basic implementation works well for
>> sequentially evaluated file types (like sourced shell scripts), python
>> config parser, and even windows ini file syntax. *On the other hand,
>> something like XML would not immediately work in the current dotdee
>> implementation. *For that, I've thought a bit about a similar
>> approach, constructing the conffile from a quilt directory of numbered
>> patch files. *If you're interested in this approach, I can describe
>> that in more detail, too, and perhaps implement another prototype.
>
> To be honest, that sounds like a hack to me. But if you are
> interested to still go this route, then maybe Config::Model might be
> useful to you? In any case, the application handling the configuration
> files is in the best place to know its internal layout, and how to
> handle it.

Oh, it's no doubt a hack! The current implementation of dotdee was
less than a day's worth of work and smoke testing. The goal was
simply to put together enough of a functional hack to generate some
discussion about it, and propose the idea as a potential approach.

Thanks for the Config::Model reference, as yes, this would be a useful library.

> So all in all, I don't think this is the right way to solve the problem
> you describe. And while I also think the conffile handling in dpkg can
> be improved, in this case the correct solution seems to me, to be to
> add .d support to the needed applications.

I'm a little de-motivated by that assessment, and I'd like to think
that we could do better in Debian, Ubuntu, and dpkg, but in any case,
thank you for your careful review!

--
:-Dustin

Dustin Kirkland
Ubuntu Core Developer


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTim78VGE4KEO+U+PPsrhiXn1tPoiGg@mail.gmail.com ">http://lists.debian.org/BANLkTim78VGE4KEO+U+PPsrhiXn1tPoiGg@mail.gmail.com
 
Old 05-04-2011, 07:50 PM
Jonathan Nieder
 
Default dotdee: a proposal for improving conffile management in Debian

Hi,

Dustin Kirkland wrote:

> I've had that feeling, I know. And I'm feeling that way about more
> and more and more utilities. I just don't think I, you, or any of us
> is going to actually getting around to "fixing" every service to
> support a .d style configuration structure.
>
> And that's at the crux of this discussion (whether it's dotdee that
> solves this, or something else completely different). I'm
> wondering/hoping/thinking about other ways we could approach this
> problem, as a distribution

Here's some lower-hanging fruit:

http://bugs.debian.org/32877
http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11409

It needs someone willing to go through the patches, resubmit them,
and take or give feedback from the list to simplify and polish them.
The result would, in my humble opinion, already be much, much better
than what we have today.

Would you be interested in that?


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110504195030.GB11280@elie">http://lists.debian.org/20110504195030.GB11280@elie
 
Old 05-04-2011, 08:12 PM
Dustin Kirkland
 
Default dotdee: a proposal for improving conffile management in Debian

On Wed, May 4, 2011 at 2:50 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Dustin Kirkland wrote:
>
>> I've had that feeling, I know. *And I'm feeling that way about more
>> and more and more utilities. *I just don't think I, you, or any of us
>> is going to actually getting around to "fixing" every service to
>> support a .d style configuration structure.
>>
>> And that's at the crux of this discussion (whether it's dotdee that
>> solves this, or something else completely different). *I'm
>> wondering/hoping/thinking about other ways we could approach this
>> problem, as a distribution
>
> Here's some lower-hanging fruit:
>
> * * * *http://bugs.debian.org/32877

Heh :-) Can any 12-year-old bug be considered low hanging fruit? :-)

In any case, I see that bug 32877 has an updated patch as of Wed, 23
Mar 2011, from Vasily i. Redkin, but I don't see any feedback yet on
that latest patch.

> * * * *http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11409

I read through this thread too. Very interesting, thoughtful layout
of conffile tracking, if a bit complex. But it looks to me like the
developer trying to solve this problem is left hanging since February
2010.

> It needs someone willing to go through the patches, resubmit them,
> and take or give feedback from the list to simplify and polish them.
> The result would, in my humble opinion, already be much, much better
> than what we have today.

Hmm, well, neither of these itches are quite identical to our
particular itch, and the proposed/pending/abandoned(?) solutions don't
quite solve the problem I need to solve. And I'd hate to spend time
simplifying and polishing them entirely in vain. If the maintainers
have a punch list of exactly what needs to be solved to get such
solutions committed, I'd be glad to help out. But unfortunately I
don't have an open ended schedule on this.

Cheers,
--
:-Dustin

Dustin Kirkland
Ubuntu Core Developer


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: BANLkTinp5aq6EWdw1=q4yjvG44WH15CzSQ@mail.gmail.com ">http://lists.debian.org/BANLkTinp5aq6EWdw1=q4yjvG44WH15CzSQ@mail.gmail.com
 
Old 05-04-2011, 08:14 PM
Jonathan Nieder
 
Default dotdee: a proposal for improving conffile management in Debian

Dustin Kirkland wrote:

> Would you be willing to join us via IRC and/or live internet audio
> stream for 1 hour next week, as we discuss this at the Ubuntu
> Developer Summit in Budapest?

No, not really. I wish I had time but email works much better for me.

> I am absolutely, keenly interested in seeing this solved for both
> Debian and Ubuntu. I'm willing and able to work on this myself, and
> I'm willing and able to delegate the work to capable members on my
> team. I'd like to hold off on both of those until we have an agreed
> upon design

Ah, then we have different philosophies. I like to see people
experimenting with future changes for dpkg, which involves

* people playing around with their private copies of dpkg;
* submitting experiments when they look good, with full
knowledge that if something better comes around then their code
might not be used;
* public development and public review of work to make it easier to
see how to get started and avoid duplication of effort.

Which is not to say that early thoughts about a design are a bad
thing, just that commiting to a design before there's code seems like
a bad idea.

FWIW my reactions to the dotdee concept were similar to Guillem's. On
the other hand the idea of hooks to support a custom strategy for
updating conffiles seems appealing to me. Saving pristine conffiles
to some fixed place ( la Sean's conffiledb) would probably work
better than the filename mapping idea I mentioned. All this is to
say, you chose an interesting topic and I'd be very happy if you move
it forward, even if your goal is this dotdee thing I'm not convinced
about.

Hope that helps,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110504201430.GC11280@elie">http://lists.debian.org/20110504201430.GC11280@elie
 
Old 05-04-2011, 08:22 PM
Raphael Hertzog
 
Default dotdee: a proposal for improving conffile management in Debian

Hi,

On Fri, 29 Apr 2011, Dustin Kirkland wrote:
> As Ubuntu Server Core Developer and Packager, I'm quite keen on
> improving two problems we encounter frequently, with respect to
> conffiles:

I would also like to solve those problems.

> PROPOSAL
>
> Here, I'm proposing a tool called "dotdee" that I think would greatly
> benefit Debian and Debian-derived distributions such as Ubuntu. Its
> stated goal is to turn any given flat file on your system to a
> dynamically concatenated flat file generated from a unique ".d"
> directory dedicated to that particular file. With such a dedicated
> and well-formed directory, system administrators, Debian packagers,
> and distributions could conveniently place additional configuration
> snippets in particular conffile's dedicated ".d" directory.

I don't want to go into details but I also do not believe that dotdee is
a complete answer to the problem. It just doesn't scale to all the
configuration file format that we can encounter and it's not clean enough
to be able to use it in other packages.

Let me explain what I would suggest to solve the problem. The root problem
is that dpkg is not flexible enough to manage the configuration files, and
it probably never will be. It just doesn't have the required knowledge.
However it knows when the package tries to upgrade the configuration file
and it installs the file itself.

I propose that we take out of dpkg the knowledge required to install a
configuration file. dpkg would still provide the default configuration
handler that implements the prompt that we know today but we should be
able to replace it easily.

By default, dpkg would use a program named dpkg-conffile-handler to
install new configuration files. That program would in fact be an
alternatives pointing by default to the default configuration file handler
provided by dpkg (the one we have now and that prompts us). Other
configuration handlers could be registered in the alternative... for
example one using a VCS to manage the configuration files, etc.

It would also be possible to override the choice of the default
configuration file handler for specific configuration files using
configuration snippets put in .d directories:
- /lib/dpkg/conffile-handling.d/ could be used by packages that want
to generate their configuration files or use something more complicated
than what dpkg offers (for example with ucf, or a script using Config::Model)
- /etc/dpkg/conffile-handling.d/ could be used by the local admin to
further override everything

dotdee could be one tool that the administrator could use as configuration
file handler but it doesn't need to be the only one. And you don't need to
divert the original file, dpkg takes care of all that part implicitly. The
admin would just do something like this:
# echo "handler /etc/foo.conf dotdee" >/etc/dpkg/conffile-handling.d/foo
# dotdee --setup /etc/foo.conf

With this solution it's also easy for a package to setup a custom conffile
handler to take over another conffile or to pre/post-process it.

There's quite some work left to define the interface that the
dpkg-conffile-handler programs must implement, and to the way the override
would work but I think this would be a clean solution to all the problems
that annoy you.

What do you think ?

> But in the mean time, what do you think? Have you encountered similar
> problems before? What other approaches have been taken to try and
> solve this? What parts of this proposal do you think are reasonable?
> Are any parts completely unreasonable? Are there extensions or
> changes you propose?

While dotdee could be acceptable for local admins to use, I think it's far
from being acceptable for official Debian packages. It really looks like
taking over other files without any way for the local admin to keep
control on his configuration files.

I think my proposal is way better in this regard.

Up to now, all the people who have had those problems dealt with them with
tools that overwrite entirely the configuration files and never took care
to think about proper integration in packages.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
▶ http://RaphaelHertzog.fr (Français)


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110504202214.GB27336@rivendell.home.ouaza.com">h ttp://lists.debian.org/20110504202214.GB27336@rivendell.home.ouaza.com
 
Old 05-04-2011, 08:33 PM
Jonathan Nieder
 
Default dotdee: a proposal for improving conffile management in Debian

Hi,

The following contains guesses about human nature; please keep in mind
that these are just guesses I am very, very likely to be wrong.

Dustin Kirkland wrote:

> Hmm, well, neither of these itches are quite identical to our
> particular itch, and the proposed/pending/abandoned(?) solutions don't
> quite solve the problem I need to solve. And I'd hate to spend time
> simplifying and polishing them entirely in vain.

Okay, but can you take inspiration from them? I explained the
relationship a little more in a separate message.

I can promise to be very responsive re something like that patch if
you are willing to listen and work with me (and I imagine the same
might go for some other people).

It's petty of me, but the previous time I sent review on that series
I was ignored. That's demotivating, so I stopped reviewing it.

http://thread.gmane.org/gmane.linux.debian.devel.dpkg.general/11100

I don't think that's Sean's fault --- all around, the process just
didn't go well because (I guess) of mismatched expectations:

- Sean expected a show of good faith from the maintainer (e.g.,
merging some new APIs that would make development of the later
patches easier);

- I expect that after I review a patch, the next iteration will
either address the problems I pointed out or give some comments on
the subject;

- Guillem expected --- well, I don't know what Guillem expected, but
in general his practice seems to be to wait for conversation to
settle and pick up what looks good, giving input when things have
gone in a wrong direction. Which works well for Linux subsystem
maintainers most of the time, but doesn't seem to have worked here.

> If the maintainers
> have a punch list of exactly what needs to be solved to get such
> solutions committed, I'd be glad to help out.

As I mentioned in another reply, I think that's a bad way to write
software. Unforeseen problems can come up after writing a patch.

But I'm not the maintainer --- I'm just someone who wants dpkg's
conffile handling to be a little better. So feel free to ignore me
if you're only interested in getting your code committed.

Hope that helps,
Jonathan


--
To UNSUBSCRIBE, email to debian-dpkg-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110504203352.GD11280@elie">http://lists.debian.org/20110504203352.GD11280@elie
 

Thread Tools




All times are GMT. The time now is 07:47 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org