FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 12-06-2008, 04:02 AM
Stefan Monnier
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

>>> Q: How are we going to do that?
>>> A: It's not possible in general.
>> Of course it is, since you can always fall back on the current code in
>> those cases where you don't know how else to do it.
> No, it's not.
> Falling back to the old behavior (not merging changes) is not a technique
> for automatically merging local changes to conffiles. It is refusing to
> solve the problem, not a solution to the problem.

When an upgrade is installed, local changes *have* to be merged with the
changes brought in from the upgrade. That's just an unvoidable need.
Currently, Debian provides very little support for that, mostly
detecting some of the conflicts, showing a `diff' output for those, and
asking the user to choose between one of the two (and leaving it up to
him to refine the merge later using the .dpkg-* files left around).

My proposal is not "Debian should always do it 100% automatically" since
we know trivially it's not possible (unless you find wrong answers
generally acceptable). I just suggest that Debian should work harder,
e.g. using diff3 rather than diff, so that some of the merges can be
done automatically.

>>> When something is impossible, it's impractical to think about how it
>>> would be done.
>> Solving NP-hard problems in a reasonable amount of time is considered
>> (currently and maybe for ever) impossible in general. Yet, people write
>> programs that do that every day.
> No, they don't.
> Instead, they make take the general problem and make it more specific
> through a set of assumptions. This altered problem is no longer NP-hard.
> The provided solution no longer solves the problem in general.
> Alternatively, they make successive approximations of the solution and stop
> when the approximation is "good enough", never actually, exactly solving
> the problem.

Right. And that's exactly what I'm proposing: redefine the problem from
"merge 100% automatically in all cases", to "do it in those cases where
we know how to do it". Suddenly your "impossible in general" becomes
quite doable.


Stefan


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-06-2008, 05:45 AM
"Boyd Stephen Smith Jr."
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

On Friday 2008 December 05 23:02, Stefan Monnier wrote:
> >>> Q: How are we going to do that?
> >>> A: It's not possible in general.
> >>
> >> Of course it is, since you can always fall back on the current code in
> >> those cases where you don't know how else to do it.
> >
> > No, it's not.
> > Falling back to the old behavior (not merging changes) is not a technique
> > for automatically merging local changes to conffiles. It is refusing to
> > solve the problem, not a solution to the problem.
>
> When an upgrade is installed, local changes *have* to be merged with the
> changes brought in from the upgrade. That's just an unvoidable need.

I disagree with this. It should be possible to establish "hooks" so that the
administrator should not ever have to edit an installed file, but instead
place additional or overriding instructions in a separate file which the
packages manager would not read or modify.

Even if upstream packaging does not have this capability, Debian-specific
patches can add it in.

> >>> When something is impossible, it's impractical to think about how it
> >>> would be done.
> >>
> >> Solving NP-hard problems in a reasonable amount of time is considered
> >> (currently and maybe for ever) impossible in general. Yet, people write
> >> programs that do that every day.
> >
> > No, they don't.
> > Instead, they make take the general problem and make it more specific
> > through a set of assumptions.
>
> Right. And that's exactly what I'm proposing: redefine the problem from
> "merge 100% automatically in all cases", to "do it in those cases where
> we know how to do it". Suddenly your "impossible in general" becomes
> quite doable.

Using that definition, I'd say have the individual package maintainers do it.
They'd be able to add more conffile-specific information to the process and
be best equipped to judge when a merge will cause less problems than the
alternative. I'm not sure how this would tie into the standard conffile
process, as conffiles aren't (IIRC) supposed to be modified/generated by the
package, but rather a single conffile to to be shipped that is an acceptable
default.

It would be nice to have a choice of diffing tools, but diff3 and the like
prefer having 3 copies: "orig", "yours", and "mine". IIRC, by the time dpkg
is prompting you, the original conffile is gone. You've modified the
installed version, and the old version of the package you are updating is
generally not available. So, you only have "mine" (current installed)
and "yours" (version from package currently being installed).

I'm not sure the best way to provide an "orig". I suppose you could install
conffiles under something like /usr/.orig/<path> or just /.orig/<path>,
with .orig and directories under it with 200 permissions and files under it
having 400 permissions. It probably wouldn't take up that much space and
could be done automatically. It would be a bit a annoying to have files on
my / device that mirror those on my /usr or /var device, but not entirely
unreasonable as long as the size is kept down. This should, of course, be
cleaned during a purge, and updated *after* the conflict(s), if any, are
resolved.

Why not write some patches against dpkg and float them out on debian-devel?
I've got too much on my plate as is.
--
Boyd Stephen Smith Jr. * * * * * * * * * * ,= ,-_-. =.
bss03@volumehost.net * * * * * * * * * * *((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy * * * * * `-'(. .)`-'
http://iguanasuicide.org/ * * * * * * * * * * *\_/ * *
 
Old 12-06-2008, 08:42 AM
"Boyd Stephen Smith Jr."
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

On Saturday 2008 December 06 02:03, lee wrote:
> On Sat, Dec 06, 2008 at 12:45:28AM -0600, Boyd Stephen Smith Jr. wrote:
> > On Friday 2008 December 05 23:02, Stefan Monnier wrote:
> > > When an upgrade is installed, local changes *have* to be merged with
> > > the changes brought in from the upgrade. That's just an unvoidable
> > > need.
> >
> > I disagree with this. It should be possible to establish "hooks" so that
> > the administrator should not ever have to edit an installed file, but
> > instead place additional or overriding instructions in a separate file
> > which the packages manager would not read or modify.
>
> How exactly would that work?

For example, a configuration file that was sourced as part of the startup
script would have "[ -r /etc/package/user.conf ] && . /etc/package/user.conf"
near the end of the file. Another example is the configuration for gdm, the
defaults are in /usr/share/gdm/defaults.conf, and the administrator and
supplement or override those with /etc/gdm/gdm.conf. There are lots of ways
to do this, but it basically boils down to having a distribution/upstream
provided configuration and locally provided configuration. This is actually
ALWAYS the case, as the source code has some default behavior (might be:
error out of otherwise "break") which is under control of
upstream/distribution and configuration files change that.

The ideal would be that files installed by packages would not be touched by
the administrator. It would make auditing a system easier and would also
allow for identifying and repairing packages that were corrupted for whatever
reason. However, packages should also work (or some definition of work)
immediately after they are installed, so that other packages can use them in
their install/remove scripts. For programs that cannot provide reasonable
defaults themselves, they generally load settings from *a* configuration
file, or *a* system configuration file and a user file. When the program
only reads from a single file, it's difficult to separate distribution
settings from locally administered settings, even though those are two
different things.

Thus, we have conffiles -- a half-way solution between having separate files
for distribution and locally-administered settings. When/where the defaults
work administrators have no worries, the package maintainer updates the
conffiles as needed. When the defaults don't work, you get the dpkg prompt,
which is usually enough; administrators that have made changes keep their old
file, until they see an incompatable change (e.g. in the conffile format) and
then have to rebuild their configuration. At this point they'd generally
have to rebuild their configuration anyway.

Anyway, the point is that most users of F(L)OSS software don't get their
software directly from the original authors, so it makes sense to have at
least 3 configuration files/directories (distribution, in /usr/share mostly;
system, in /etc mostly; and user, in ~ mostly) for any user application, and
2 (the first 2) for non-user applications. [It would also be nice to
have "site" configuration in /usr/local/share or somesuch.] Unfortunately,
this doesn't happen often and we get half-way solutions like conffiles (or
Gentoo's equivalent).
--
Boyd Stephen Smith Jr. * * * * * * * * * * ,= ,-_-. =.
bss03@volumehost.net * * * * * * * * * * *((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy * * * * * `-'(. .)`-'
http://iguanasuicide.org/ * * * * * * * * * * *\_/ * *
 
Old 12-07-2008, 09:24 PM
"Eduardo M KALINOWSKI"
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

From: lee <lee@yun.yagibdah.de>
> When a program uses a number of different configuration files, it's
> much more difficult for the administrator to configure it.

I'd say it's a matter of preference. I like exim's split configuration, it
makes upgrades easier as I only have to deal with the parts that I've
changed. And perhaps not even that, if I can make a change by adding a new
snippet file instead of changing an existing one.

For those who prefer a single file, debconf allows that, satisfying all
kinds of people. You can even dump debconf altogether and do all the
configuration yourself.

> Well, that already has been achieved to a great deal, hasn't it? Just
> packages like exim4 or apache2 that use an approach which makes it
> very difficult to impossible for the administrator to configure them
> break it.

That's your opinion. Do not take it as absolute.

I don't know about apache, but as I said, with exim it's quite easy to use
a single file, if you prefer it that way.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 12-07-2008, 10:10 PM
"Boyd Stephen Smith Jr."
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

On Sunday 07 December 2008, lee <lee@yun.yagibdah.de> wrote about 'Re:
Better support for merging local and upstream (was: Erase cache, clean
registry in Linux)':
>On Sat, Dec 06, 2008 at 03:42:42AM -0600, Boyd Stephen Smith Jr. wrote:
>> On Saturday 2008 December 06 02:03, lee wrote:
>> > On Sat, Dec 06, 2008 at 12:45:28AM -0600, Boyd Stephen Smith Jr.
wrote:
>> > > I disagree with this. It should be possible to establish "hooks"
>> > > so that the administrator should not ever have to edit an installed
>> > > file, but instead place additional or overriding instructions in a
>> > > separate file which the packages manager would not read or modify.
>> >
>> > How exactly would that work?
>>
>> There are lots of ways to do this, but it basically boils down to
>> having a distribution/upstream provided configuration and locally
>> provided configuration. This is actually ALWAYS the case, as the
>> source code has some default behavior
>
>Yes, I guess you have to have something in the source to compile it. I
>don't really consider that as "configuration", though.

But it is. It's the underlying "defaults" that 1) your configuration
alters and 2) could change the same way a distribution-provided
configuration might.

>> When the program only reads from a single file, it's difficult to
>> separate distribution settings from locally administered settings,
>> even though those are two different things.
>
>It's the configuration of a program, not two different things.

It is. They are maintained by two separate entities. Just like the
default behavior is maintained by the original author and the
configuration is maintained by others.

>When a program uses a number of different configuration files, it's
>much more difficult for the administrator to configure it.

I completely disagree here. Your specific examples, apache2 and exim4
simply convince me further that you are wrong. I maintain a exim4
installation and multiple apache2 installations and I greatly prefer
separated files to a single file.

It's easier to work with that way, not harder.

>As you want or need to
>distinguish the administrators configuration from the package
>managers, that could (better) be done by having different sections in
>the configuration file or by some other way to have and keep the
>package managers configuration within the file together with what the
>administrator has set up.

No, separate files is better, since files are already a natural unit. 'rm
my.conf' (and leaving debian.conf) is easier than editing a single file to
remove local changes.

>> Thus, we have conffiles -- a half-way solution between having
>> separate files for distribution and locally-administered settings.
>> When/where the defaults work administrators have no worries, the
>> package maintainer updates the conffiles as needed. When the
>> defaults don't work, you get the dpkg prompt, which is usually
>> enough; administrators that have made changes keep their old file,
>> until they see an incompatable change (e.g. in the conffile format)
>> and then have to rebuild their configuration. At this point they'd
>> generally have to rebuild their configuration anyway.
>
>Well, that already has been achieved to a great deal, hasn't it?

Well, it's achieved less than the alternative would, but with arguably less
work.

>Just
>packages like exim4 or apache2 that use an approach which makes it
>very difficult to impossible for the administrator to configure them
>break it.

I find the debian way of configuration apache2 superior to other
distributions I've used. I've never installed exim4 on a different
distributions, but I do think it would be troublesome making changes to a
single, large file then the logically separated small files in well-named
directories used by Debian.

>> Anyway, the point is that most users of F(L)OSS software don't get
>> their software directly from the original authors, so it makes sense to
>> have at least 3 configuration files/directories (distribution, in
>> /usr/share mostly; system, in /etc mostly; and user, in ~ mostly) for
>> any user application, and 2 (the first 2) for non-user applications.
>
>Hm, when I switched from Suse to Debian, one of the advantages of
>Debian was that they stayed closer to what the original authors
>did.

I guess. There are certainly changes in Ubuntu/openSuse that I don't see
in Debian packages, but Debian (particularly the security team) is very
adamant that Debian must be able to make changes without prior approval
from the original authors to better serve the needs of it's users. (E.g.
firefox vs. iceweasel)

Also, some upstream programs may not need their source changed, but still
need to be "configured" to work immediately after the package as been
installed on a Debian system. This might mean looking somewhere other
than upstream expected for libraries, reading configurations
from /etc/<package>/<package>.conf instead of /etc/<package>.conf, or
simply having some defaults provided so the program doesn't "give up"
until the user manually fiddles with it.

>That made it a lot easier to, for example, get a newer version of
>a software and use that instead of what came in the distribution ---
>not only with configuring it, but also with compiling it and keeping
>that version installed.

Down that road lies danger, unless you properly segregate "packages" not
under control of the package manager OR properly inform the package
manager of their existence.

>Too much automatic doing is a bad thing;

Then throw away your computer and get out your pen and paper. The only
things computers do is automate tasks, all of it can be done manually.
The more things that are automated, the less labor is required, and the
more time can be spent focusing on less menial tasks.

Caveat: Automating something that is not menial is generally a dangerous
path. Since it is not menial, it needs some sort of conscious, generally
considered and informed, input which computers (and other non-sentient
machines) cannot provide.
--
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss03@volumehost.net ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
http://iguanasuicide.org/ \_/
 
Old 12-09-2008, 07:23 PM
"Boyd Stephen Smith Jr."
 
Default Better support for merging local and upstream (was: Erase cache, clean registry in Linux)

On Tuesday 2008 December 09 13:33:19 lee wrote:
>On Sun, Dec 07, 2008 at 08:24:45PM -0200, Eduardo M KALINOWSKI wrote:
>> From: lee <lee@yun.yagibdah.de>
>>
>> > When a program uses a number of different configuration files, it's
>> > much more difficult for the administrator to configure it.
>>
>> I'd say it's a matter of preference. I like exim's split configuration.
>
>Well, I don't like it.

So, you clearly agree with Eduardo. It's a matter of preference. Oddly
enough, exim4-config provides the option of split or single.

>Using a single configuration file, you as well
>only need to deal with the parts that need to be changed. But you
>would know what is configured and what needs to be changed which is
>impossible with the automatic configuration.

It's also impossible for exim4 for work immediately after installation without
an automatic configuration. Packages need to work immediate after
installation so that other packages can use them during their installation.

Distribution-shipped configuration files aren't going away. Perhaps they
should be moved to somewhere you don't see them -- like /usr/share -- and
then overridden with your configuration in /etc, but they aren't going away.
If you want to use your version of the configuration file, but tell dpkg not
to use the maintainer's version; it'll ask you before replacing files marked
as conffiles.

>> For those who prefer a single file, debconf allows that, satisfying all
>> kinds of people. You can even dump debconf altogether and do all the
>> configuration yourself.
>
>Yes, it's good to have a choice. The problem is that the choice
>eventually gets taken away.

How so? While I would don't see myself using single-file exim4 configurations
in the future, I certainly think it should be an option. I used to
use /etc/apt/sources.list.d, but now I think I prefer
the /etc/apt/sources.list approach. It's supposed by more programs and my
list never gets over 24 lines anyway.

>> > Just
>> > packages like exim4 or apache2 that use an approach which makes it
>> > very difficult to impossible for the administrator to configure them
>> > break it.
>>
>> That's your opinion. Do not take it as absolute.
>
>It's a matter of fact.

No, it's not. It's opinion. I found configuring apache2 on Debian easier
than Gentoo or FC particularly because it was split up *well*. I even split
up my configuration across multiple files. It's nice to be able to disable
just a single site with a single command (okay, two; apache2 has to reload
the configuration) rather than commenting out multiple set of lines and
hoping I've got it all.

>You can try it by, for example, installing php
>for apache2. You're not being told what needs to be done with the
>apache2 config, it gets modified automatically. Then you find that it
>doesn't work, not before restarting apache2 manually.

Done it, for running Drupal. Yes, I had to ask apache to re-read it's
configuration files. That's about it, and I was expecting that.

>Exim4 is another example. I can configure it easily bypassing the
>automatic configuration, but I'm unable to configure it using the
>auomatic configuration. What about changes if you use the automatic
>configuration? Would you be told what is being changed? Or would your
>mailserver suddenly be configured differently without you even knowing
>about it?

You are basically worried about the behavior you haven't specified changing.

Unfortunately, even with a single file that can happen due to changes in the
source code. If you need specific behavior, don't default to it, explicitly
specify it.

Personally, I only needed to tweak things in the exim4 configuration, so most
(maybe all) of my changes were adding files to the existing directory
structure. The split in to multiple files made it easier to work with.

>> I don't know about apache, but as I said, with exim it's quite easy
>> to use a single file, if you prefer it that way.
>
>Well, where is the option to have the automatic configuration create a
>single configuration file that you can easily check and then use to
>configure your program if you want to?

Do you mean specifically for exim4? I think the option to use a single file
is the first or second question asked by dpkg-reconfigure exim4-config.
--
Boyd Stephen Smith Jr. * * * * * * * * * * ,= ,-_-. =.
bss03@volumehost.net * * * * * * * * * * *((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy * * * * * `-'(. .)`-'
http://iguanasuicide.org/ * * * * * * * * * * *\_/ * *
 

Thread Tools




All times are GMT. The time now is 11:19 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org