FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo Development

 
 
LinkBack Thread Tools
 
Old 04-30-2012, 08:39 AM
Ulrich Mueller
 
Default RFC: new feature to disable merging into stray locations

>>>>> On Mon, 30 Apr 2012, Krzysztof Pawlik wrote:

> On 30/04/12 10:12, Ulrich Mueller wrote:
>> If the eclass doesn't work with FEATURES="collision-protect" then it
>> needs to be fixed.

> Long story short: older eclass compiled Python byte code in live
> file system, new one does it in src_install so .pyo/.pyc gets
> properly recorded,

That's a good change.

> so the package *has to overwrite* files that are not owned by anyone
> (no package owns them).

The usual approach to this problem was to have some cleanup code in
pkg_setup or pkg_preinst that would remove the old files.

Ulrich
 
Old 04-30-2012, 09:28 AM
Michał Górny
 
Default RFC: new feature to disable merging into stray locations

On Mon, 30 Apr 2012 10:23:03 +0200
Krzysztof Pawlik <nelchael@gentoo.org> wrote:

> On 30/04/12 10:12, Ulrich Mueller wrote:
> >>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
> >> Since lately Gentoo devs force you to replace collision-protect
> >> with protect-owned [1] and sometimes packages just spit out files
> >> randomly on the filesystem due to random errors, I thought it may
> >> be a good idea to provide a new feature limiting the locations
> >> where packages can install.
> >
> > If the eclass doesn't work with FEATURES="collision-protect" then it
> > needs to be fixed.
>
> Long story short: older eclass compiled Python byte code in live file
> system, new one does it in src_install so .pyo/.pyc gets properly
> recorded, so the package *has to overwrite* files that are not owned
> by anyone (no package owns them). I've talked to Zac and:
>
> >>>
> There's not much else you can do. However, FEATURES="protect-owned" is
> enabled by default, and it will work fine if the .pyo and .pyc files
> are not owned by anything. I don't know why people use
> FEATURES="collision-protect", but at least you can tell them that
> they'll still have a reasonable level of protection from
> FEATURES="protect-owned" (it protect against file collisions between
> packages).
> <<<
>
> If you want to help feel free to suggest *how* to fix it, patches are
> especially welcomed

Just make it warn or something like that. Or even fail but only when
such files exist. There is no point in forcing me to disable
collision-protect when the ebuilds were migrated already.

--
Best regards,
Michał Górny
 
Old 04-30-2012, 09:30 AM
Michał Górny
 
Default RFC: new feature to disable merging into stray locations

On Mon, 30 Apr 2012 10:24:58 +0200
Krzysztof Pawlik <nelchael@gentoo.org> wrote:

> On 30/04/12 08:57, Michał Górny wrote:
> > Hello,
> >
> > Since lately Gentoo devs force you to replace collision-protect with
> > protect-owned [1] and sometimes packages just spit out files
> > randomly on the filesystem due to random errors, I thought it may
> > be a good idea to provide a new feature limiting the locations
> > where packages can install.
>
> You're not forced to do anything.

I am not? How come random ebuilds die in pkg_pretend() for me then?

>
> > In order to do that, we should first compose a complete
> > include/exclude list where packages can install. I'd suggest the
> > following:
> >
> > + /bin
> > + /boot (but maybe just subdirectories so packages can't overwrite
> > kernels?)
> > [potentially + /dev? but that's useful only when tmpfs isn't
> > mounted]
> > + /etc
> > + /lib, /lib32, /lib64
> > + /opt
> > + /sbin
> > [potentially + /service for ugly daemontools]
> > + /usr
> > + /var
> > - /usr/local
> > - /usr/portage
> >
> > What are your thoughts on this?
> >
> > [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>
> I think it's feature creep - you can just set INSTALL_MASK="/usr/local
> /usr/portage", no need for new features.
>
> PS. I (and few other folks) don't have /usr/portage.

INSTALL_MASK won't make committing such an ebuild a fatal error. It
will just hide problems.

--
Best regards,
Michał Górny
 
Old 04-30-2012, 09:37 AM
Krzysztof Pawlik
 
Default RFC: new feature to disable merging into stray locations

On 30/04/12 11:28, Michał Górny wrote:
> On Mon, 30 Apr 2012 10:23:03 +0200
> Krzysztof Pawlik <nelchael@gentoo.org> wrote:
>
>> On 30/04/12 10:12, Ulrich Mueller wrote:
>>>>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
>>>> Since lately Gentoo devs force you to replace collision-protect
>>>> with protect-owned [1] and sometimes packages just spit out files
>>>> randomly on the filesystem due to random errors, I thought it may
>>>> be a good idea to provide a new feature limiting the locations
>>>> where packages can install.
>>>
>>> If the eclass doesn't work with FEATURES="collision-protect" then it
>>> needs to be fixed.
>>
>> Long story short: older eclass compiled Python byte code in live file
>> system, new one does it in src_install so .pyo/.pyc gets properly
>> recorded, so the package *has to overwrite* files that are not owned
>> by anyone (no package owns them). I've talked to Zac and:
>>
>>>>>
>> There's not much else you can do. However, FEATURES="protect-owned" is
>> enabled by default, and it will work fine if the .pyo and .pyc files
>> are not owned by anything. I don't know why people use
>> FEATURES="collision-protect", but at least you can tell them that
>> they'll still have a reasonable level of protection from
>> FEATURES="protect-owned" (it protect against file collisions between
>> packages).
>> <<<
>>
>> If you want to help feel free to suggest *how* to fix it, patches are
>> especially welcomed
>
> Just make it warn or something like that. Or even fail but only when
> such files exist. There is no point in forcing me to disable
> collision-protect when the ebuilds were migrated already.

Does this work for you:

$ cvs diff python-distutils-ng.eclass
Index: python-distutils-ng.eclass
================================================== =================
RCS file: /var/cvsroot/gentoo-x86/eclass/python-distutils-ng.eclass,v
retrieving revision 1.12
diff -u -r1.12 python-distutils-ng.eclass
--- python-distutils-ng.eclass 30 Apr 2012 08:25:31 -0000 1.12
+++ python-distutils-ng.eclass 30 Apr 2012 09:36:48 -0000
@@ -304,11 +304,10 @@
# Phase function: pkg_pretend
python-distutils-ng_pkg_pretend() {
if has "collision-protect" ${FEATURES}; then
- eerror "Due to previous eclass compiling Python files outside of
src_install"
- eerror "(and not recording resulting .pyc and .pyo files as
owned by any package)"
- eerror "merging this package with "collision-protect" in
FEATURES will result"
- eerror "in an error, please switch to using "protect-owned"
instead."
- die ""collision-protect" in FEATURES detected"
+ ewarn "Due to previous eclass compiling Python files outside of
src_install"
+ ewarn "(and not recording resulting .pyc and .pyo files as owned
by any package)"
+ ewarn "merging this package with "collision-protect" in
FEATURES may result"
+ ewarn "in an error, please switch to using "protect-owned"
instead."
fi
}

(it's s/eerror/ewarn/ & die removal).

I'm still wondering how many people will read this warning when they hit
collision error.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...
 
Old 04-30-2012, 09:40 AM
Krzysztof Pawlik
 
Default RFC: new feature to disable merging into stray locations

On 30/04/12 11:30, Michał Górny wrote:
> On Mon, 30 Apr 2012 10:24:58 +0200
> Krzysztof Pawlik <nelchael@gentoo.org> wrote:
>
>> On 30/04/12 08:57, Michał Górny wrote:
>>> Hello,
>>>
>>> Since lately Gentoo devs force you to replace collision-protect with
>>> protect-owned [1] and sometimes packages just spit out files
>>> randomly on the filesystem due to random errors, I thought it may
>>> be a good idea to provide a new feature limiting the locations
>>> where packages can install.
>>
>> You're not forced to do anything.
>
> I am not? How come random ebuilds die in pkg_pretend() for me then?
>
>>
>>> In order to do that, we should first compose a complete
>>> include/exclude list where packages can install. I'd suggest the
>>> following:
>>>
>>> + /bin
>>> + /boot (but maybe just subdirectories so packages can't overwrite
>>> kernels?)
>>> [potentially + /dev? but that's useful only when tmpfs isn't
>>> mounted]
>>> + /etc
>>> + /lib, /lib32, /lib64
>>> + /opt
>>> + /sbin
>>> [potentially + /service for ugly daemontools]
>>> + /usr
>>> + /var
>>> - /usr/local
>>> - /usr/portage
>>>
>>> What are your thoughts on this?
>>>
>>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>>
>> I think it's feature creep - you can just set INSTALL_MASK="/usr/local
>> /usr/portage", no need for new features.
>>
>> PS. I (and few other folks) don't have /usr/portage.
>
> INSTALL_MASK won't make committing such an ebuild a fatal error. It
> will just hide problems.

How would your proposed feature prevent ebuilds that install something to
/usr/local from being committed?

Other valid case are private repos - I have a few ebuilds that install stuff to
/<some_random_dir> (proprietary stuff).

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...
 
Old 04-30-2012, 09:42 AM
Krzysztof Pawlik
 
Default RFC: new feature to disable merging into stray locations

On 30/04/12 10:39, Ulrich Mueller wrote:
>>>>>> On Mon, 30 Apr 2012, Krzysztof Pawlik wrote:
>
>> On 30/04/12 10:12, Ulrich Mueller wrote:
>>> If the eclass doesn't work with FEATURES="collision-protect" then it
>>> needs to be fixed.
>
>> Long story short: older eclass compiled Python byte code in live
>> file system, new one does it in src_install so .pyo/.pyc gets
>> properly recorded,
>
> That's a good change.



>> so the package *has to overwrite* files that are not owned by anyone
>> (no package owns them).
>
> The usual approach to this problem was to have some cleanup code in
> pkg_setup or pkg_preinst that would remove the old files.

Interesting idea - I see two problems:
- how I would get the list of files for already installed version (the file
list can (and will) differ between versions)
- PM has a manifest file that lists all installed file, it can check against
it, not against live file system, so just removing the files may not be enough
(just my speculations - I'm not sure of this)

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...
 
Old 04-30-2012, 09:43 AM
Michał Górny
 
Default RFC: new feature to disable merging into stray locations

On Mon, 30 Apr 2012 11:37:53 +0200
Krzysztof Pawlik <nelchael@gentoo.org> wrote:

> On 30/04/12 11:28, Michał Górny wrote:
> > On Mon, 30 Apr 2012 10:23:03 +0200
> > Krzysztof Pawlik <nelchael@gentoo.org> wrote:
> >
> >> On 30/04/12 10:12, Ulrich Mueller wrote:
> >>>>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
> >>>> Since lately Gentoo devs force you to replace collision-protect
> >>>> with protect-owned [1] and sometimes packages just spit out files
> >>>> randomly on the filesystem due to random errors, I thought it may
> >>>> be a good idea to provide a new feature limiting the locations
> >>>> where packages can install.
> >>>
> >>> If the eclass doesn't work with FEATURES="collision-protect" then
> >>> it needs to be fixed.
> >>
> >> Long story short: older eclass compiled Python byte code in live
> >> file system, new one does it in src_install so .pyo/.pyc gets
> >> properly recorded, so the package *has to overwrite* files that
> >> are not owned by anyone (no package owns them). I've talked to Zac
> >> and:
> >>
> >>>>>
> >> There's not much else you can do. However,
> >> FEATURES="protect-owned" is enabled by default, and it will work
> >> fine if the .pyo and .pyc files are not owned by anything. I don't
> >> know why people use FEATURES="collision-protect", but at least you
> >> can tell them that they'll still have a reasonable level of
> >> protection from FEATURES="protect-owned" (it protect against file
> >> collisions between packages).
> >> <<<
> >>
> >> If you want to help feel free to suggest *how* to fix it, patches
> >> are especially welcomed
> >
> > Just make it warn or something like that. Or even fail but only when
> > such files exist. There is no point in forcing me to disable
> > collision-protect when the ebuilds were migrated already.
>
> Does this work for you:
>
> $ cvs diff python-distutils-ng.eclass
> Index: python-distutils-ng.eclass
> ================================================== =================
> RCS file: /var/cvsroot/gentoo-x86/eclass/python-distutils-ng.eclass,v
> retrieving revision 1.12
> diff -u -r1.12 python-distutils-ng.eclass
> --- python-distutils-ng.eclass 30 Apr 2012 08:25:31 -0000 1.12
> +++ python-distutils-ng.eclass 30 Apr 2012 09:36:48 -0000
> @@ -304,11 +304,10 @@
> # Phase function: pkg_pretend
> python-distutils-ng_pkg_pretend() {
> if has "collision-protect" ${FEATURES}; then
> - eerror "Due to previous eclass compiling Python files
> outside of src_install"
> - eerror "(and not recording resulting .pyc and .pyo
> files as owned by any package)"
> - eerror "merging this package with
> "collision-protect" in FEATURES will result"
> - eerror "in an error, please switch to using
> "protect-owned" instead."
> - die ""collision-protect" in FEATURES detected"
> + ewarn "Due to previous eclass compiling Python files
> outside of src_install"
> + ewarn "(and not recording resulting .pyc and .pyo
> files as owned by any package)"
> + ewarn "merging this package with
> "collision-protect" in FEATURES may result"
> + ewarn "in an error, please switch to using
> "protect-owned" instead."
> fi
> }
>
> (it's s/eerror/ewarn/ & die removal).

Yes, it will let me use the eclass at all.

> I'm still wondering how many people will read this warning when they
> hit collision error.

You can make it fatal if you detect collisions yourself but that's
repeating a lot of work.

--
Best regards,
Michał Górny
 
Old 04-30-2012, 09:47 AM
Michał Górny
 
Default RFC: new feature to disable merging into stray locations

On Mon, 30 Apr 2012 11:40:26 +0200
Krzysztof Pawlik <nelchael@gentoo.org> wrote:

> On 30/04/12 11:30, Michał Górny wrote:
> > On Mon, 30 Apr 2012 10:24:58 +0200
> > Krzysztof Pawlik <nelchael@gentoo.org> wrote:
> >
> >> On 30/04/12 08:57, Michał Górny wrote:
> >>> Hello,
> >>>
> >>> Since lately Gentoo devs force you to replace collision-protect
> >>> with protect-owned [1] and sometimes packages just spit out files
> >>> randomly on the filesystem due to random errors, I thought it may
> >>> be a good idea to provide a new feature limiting the locations
> >>> where packages can install.
> >>
> >> You're not forced to do anything.
> >
> > I am not? How come random ebuilds die in pkg_pretend() for me then?
> >
> >>
> >>> In order to do that, we should first compose a complete
> >>> include/exclude list where packages can install. I'd suggest the
> >>> following:
> >>>
> >>> + /bin
> >>> + /boot (but maybe just subdirectories so packages can't overwrite
> >>> kernels?)
> >>> [potentially + /dev? but that's useful only when tmpfs isn't
> >>> mounted]
> >>> + /etc
> >>> + /lib, /lib32, /lib64
> >>> + /opt
> >>> + /sbin
> >>> [potentially + /service for ugly daemontools]
> >>> + /usr
> >>> + /var
> >>> - /usr/local
> >>> - /usr/portage
> >>>
> >>> What are your thoughts on this?
> >>>
> >>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
> >>
> >> I think it's feature creep - you can just set
> >> INSTALL_MASK="/usr/local /usr/portage", no need for new features.
> >>
> >> PS. I (and few other folks) don't have /usr/portage.
> >
> > INSTALL_MASK won't make committing such an ebuild a fatal error. It
> > will just hide problems.
>
> How would your proposed feature prevent ebuilds that install
> something to /usr/local from being committed?

If we enable this by default, devs will get errors for that.

> Other valid case are private repos - I have a few ebuilds that
> install stuff to /<some_random_dir> (proprietary stuff).

Feel free to disable it. That's what FEATURES are for.

--
Best regards,
Michał Górny
 
Old 04-30-2012, 09:50 AM
Michał Górny
 
Default RFC: new feature to disable merging into stray locations

On Mon, 30 Apr 2012 11:42:56 +0200
Krzysztof Pawlik <nelchael@gentoo.org> wrote:

> >> so the package *has to overwrite* files that are not owned by
> >> anyone (no package owns them).
> >
> > The usual approach to this problem was to have some cleanup code in
> > pkg_setup or pkg_preinst that would remove the old files.
>
> Interesting idea - I see two problems:
> - how I would get the list of files for already installed version
> (the file list can (and will) differ between versions)

As PMS doesn't give us any better tools, you can just look at ${ROOT}
in pkg_preinst().

> - PM has a manifest file that lists all installed file, it can check
> against it, not against live file system, so just removing the files
> may not be enough (just my speculations - I'm not sure of this)

It is enough in case of Portage. Especially that those files aren't
listed in those manifests.

--
Best regards,
Michał Górny
 
Old 04-30-2012, 10:00 AM
Krzysztof Pawlik
 
Default RFC: new feature to disable merging into stray locations

On 30/04/12 11:47, Michał Górny wrote:
> On Mon, 30 Apr 2012 11:40:26 +0200
> Krzysztof Pawlik <nelchael@gentoo.org> wrote:
>
>> On 30/04/12 11:30, Michał Górny wrote:
>>> On Mon, 30 Apr 2012 10:24:58 +0200
>>> Krzysztof Pawlik <nelchael@gentoo.org> wrote:
>>>
>>>> On 30/04/12 08:57, Michał Górny wrote:
>>>>> Hello,
>>>>>
>>>>> Since lately Gentoo devs force you to replace collision-protect
>>>>> with protect-owned [1] and sometimes packages just spit out files
>>>>> randomly on the filesystem due to random errors, I thought it may
>>>>> be a good idea to provide a new feature limiting the locations
>>>>> where packages can install.
>>>>
>>>> You're not forced to do anything.
>>>
>>> I am not? How come random ebuilds die in pkg_pretend() for me then?
>>>
>>>>
>>>>> In order to do that, we should first compose a complete
>>>>> include/exclude list where packages can install. I'd suggest the
>>>>> following:
>>>>>
>>>>> + /bin
>>>>> + /boot (but maybe just subdirectories so packages can't overwrite
>>>>> kernels?)
>>>>> [potentially + /dev? but that's useful only when tmpfs isn't
>>>>> mounted]
>>>>> + /etc
>>>>> + /lib, /lib32, /lib64
>>>>> + /opt
>>>>> + /sbin
>>>>> [potentially + /service for ugly daemontools]
>>>>> + /usr
>>>>> + /var
>>>>> - /usr/local
>>>>> - /usr/portage
>>>>>
>>>>> What are your thoughts on this?
>>>>>
>>>>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>>>>
>>>> I think it's feature creep - you can just set
>>>> INSTALL_MASK="/usr/local /usr/portage", no need for new features.
>>>>
>>>> PS. I (and few other folks) don't have /usr/portage.
>>>
>>> INSTALL_MASK won't make committing such an ebuild a fatal error. It
>>> will just hide problems.
>>
>> How would your proposed feature prevent ebuilds that install
>> something to /usr/local from being committed?
>
> If we enable this by default, devs will get errors for that.

Could it be added instead to repoman? I'm testing all ebuilds before committing
like this:

$ ebuild xhtml2pdf-0.0.3.ebuild manifest clean install

repoman could look at build.log or ${D} and warn if it can't do those checks.

>> Other valid case are private repos - I have a few ebuilds that
>> install stuff to /<some_random_dir> (proprietary stuff).
>
> Feel free to disable it. That's what FEATURES are for.

Yes, I could.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...
 

Thread Tools




All times are GMT. The time now is 09:47 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org