FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Desktop

 
 
LinkBack Thread Tools
 
Old 09-14-2011, 09:23 PM
Barry Fishman
 
Default tracker in F16

On 2011-09-13 23:41:01 EDT, Cosimo Cecchi wrote:
> Yeah, tracker is a required dependency of gnome-documents now. As
> Bastien says, we should try to identify and fix possible bugs and
> resource issues upstream.

I don't see why tracker or any such daemon should ever be a required
package, or that one should not be able to uninstall it without taking
much of Gnome with it.

I have no problems with it being used in the default gnome-shell setup,

On a related note, I had thought the concerns of people that gnome-shell
did not have a simple to find way of shutting down the computer as an
over reaction. Now that it has also been removed from the GDM setup
has made me change my mind. Is the prefered method of shutting down the
computer pressing the hardware's power button? What if I just want to
just reboot another OS?

--
Barry Fishman

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-14-2011, 09:52 PM
Cosimo Cecchi
 
Default tracker in F16

On Wed, 2011-09-14 at 17:23 -0400, Barry Fishman wrote:

> I don't see why tracker or any such daemon should ever be a required
> package, or that one should not be able to uninstall it without taking
> much of Gnome with it.
>
> I have no problems with it being used in the default gnome-shell setup,

Tracker provides both a store process (holding the actual DB) and a
shared library for applications with APIs to access and modify the
database. gnome-documents depends on the library, which of course is
written under the assumption that the daemon is available.
Of course you can always uninstall tracker, but that will remove core
parts of the GNOME desktop as well (at least gnome-documents).

Cosimo

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-14-2011, 10:25 PM
Evandro Giovanini
 
Default tracker in F16

On Wed, Sep 14, 2011 at 6:23 PM, Barry Fishman <barry_fishman@acm.org> wrote:
>
> On 2011-09-13 23:41:01 EDT, Cosimo Cecchi wrote:
>> Yeah, tracker is a required dependency of gnome-documents now. As
>> Bastien says, we should try to identify and fix possible bugs and
>> resource issues upstream.
>
> I don't see why tracker or any such daemon should ever be a required
> package, or that one should not be able to uninstall it without taking
> much of Gnome with it.
>
> I have no problems with it being used in the default gnome-shell setup,
>
> On a related note, I had thought the concerns of people that gnome-shell
> did not have a simple to find way of shutting down the computer as an
> over reaction. *Now that it has also been removed from the GDM setup
> has made me change my mind. *Is the prefered method of shutting down the
> computer pressing the hardware's power button? *What if I just want to
> just reboot another OS?
>
>

The menu for Shutdown/Restart will be added back to GDM.

--
Evandro
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-15-2011, 04:04 PM
Cosimo Cecchi
 
Default tracker in F16

On Tue, 2011-09-13 at 23:41 -0400, Cosimo Cecchi wrote:

> I also think tracker should use a different default configuration; these
> are the changes I think we should make:

Just a quick update: both my proposed changes made their way into the
default tracker configuration upstream, so they will eventually reach
Fedora when the new version gets released and packaged.

Cosimo

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-18-2011, 10:38 PM
Lennart Poettering
 
Default tracker in F16

On Tue, 13.09.11 23:41, Cosimo Cecchi (ccecchi@redhat.com) wrote:

>
> Hi Adam,
>
> On Mon, 2011-09-12 at 14:01 -0700, Adam Williamson wrote:
>
> > oop, good catch. I did suspect it's going to become more a part of the
> > GNOME Experience from now on, though I think I missed the bit where
> > tracker beat zeitgeist =)
>
> Yeah, tracker is a required dependency of gnome-documents now. As
> Bastien says, we should try to identify and fix possible bugs and
> resource issues upstream.
>
> I also think tracker should use a different default configuration; these
> are the changes I think we should make:
> - removable media indexing should be off by default - I don't know if
> this is used by the totem grilo integration, but it's not by
> gnome-documents. Anyway, I think the best way to approach this is
> tracker should just be aware of removable media and applications who
> make use of them should be able to have them crawled on-demand, but that
> needs fixing in tracker first.
> - indexing should be enabled on battery - right now it's disabled. If we
> disable indexing on battery, applications like gnome-documents will be
> "locked" into the last view of the local file entries in the tracker
> database before you unplugged the cable, which is very ugly.
>
> Having used tracker on by default for a bit now it seems that after the
> initial crawling, which is an expensive operation, I didn't notice any
> particular increase in resource usage. With removable devices indexing
> off, this should ideally be a one-shot operation.

I'd be thankful if the impact the crawling has on the running system
could be minimized via SCHED_IDLE and IOPRIO_CLASS_IDLE. gnome bz
#659422.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-19-2011, 04:00 PM
Lennart Poettering
 
Default tracker in F16

On Mon, 19.09.11 00:38, Lennart Poettering (mzerqung@0pointer.de) wrote:

> > Having used tracker on by default for a bit now it seems that after the
> > initial crawling, which is an expensive operation, I didn't notice any
> > particular increase in resource usage. With removable devices indexing
> > off, this should ideally be a one-shot operation.
>
> I'd be thankful if the impact the crawling has on the running system
> could be minimized via SCHED_IDLE and IOPRIO_CLASS_IDLE. gnome bz
> #659422.

Hmm, so investigating this further with strace it appears to me that
tracker is trying to make use of the kernel in a way it shouldn't. Or to
put this another way: our infrastructure (the Linux kernel) isn't ready
for tracker yet.

The problems here have been known for a long time, but afaik there still
hasn't been done anything about them to make things ready for tracker. I
am not sure we should enable tracker before these fundamental issues
aren't fixed. I am completely fine with enabling stuff that has known
bugs because that's the way how you get them fixed. But in this case
here I fear that the basics just don't exist and hence tracker is
fundamentally built on infrastructure that is borked. Everytime people
have looked into enabling tracker (or beagle for the matter) these
issues showed up, and every single time nothing happened about them, and
I am not really seeing why these issues stopped mattering now.

To be more explicit:

a) tracker uses inotify recursively and creates a massive number of
watches due to that. That is both ugly and doesn't scale. Tracker
apparently tries to not take up the full pool of inotfy handles the
system provides, but that won't help if you have more than one user on
the system. The solution here should probably be fanotify, which allows
proper recursive file system watches. So far fanotify has been
accessible to root only, which is presumably why tracker doesn't use
it. However, the solution here cannot be to work around that fact by
using inotify, but must be to invest the necessary kernel work to make
fanotify useful from unprivileged processes.

b) We still don't have a way to detect offline modification of
directories. That means detecting changes to the home directory made
offline is very expensive. btrfs now has hooks to improve the situation,
but ext4 still hasn't. Does tracker at least use the btrfs hooks? (btrfs
provides a log of changes to userspace, which can be used for that. Another
solution are recursive directory change timestamps).

I'd really prefer if we could fix these fundamental issues before we
enable tracker. To me it appears here as if we are trying to make the
second step before the first.

[ And there are acouple of other things I'd like to see changed. For
example, I am pretty sure that tracker's open() calls to files should
not be considered accesses in regards to access time. O_NOATIME should
be used here, which would reduce the amount of disk writes
substantially. Also, tracker appears to BSD lock all files it
accesses. That looks quite borked. Which other tool is it synchronizing
against here? This looks unsecure to do (because the files are often
accessible to others), and since these locks are advisory only there
needs to be a strict protocol followed by everybody else accessing these
files, which I guarantee you there isn't since these are basically all
the user's files. Moreover it appears tracker is mixing BSD and POSIX
locks, which is dangerous due to ABBA, in particular when used on NFS
directories, which will just end up in total chaos since Linux is so
stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
way. In any case you should NEVER EVER use POSIX locking, since it is
compltely borked anyway. The locking must go. I also see a massive
amount of futex calls in strace, i.e. probably some mutexes thrown in
the mix to make the locking problems even more interesting, which makes
my fingernails roll up, since they apparently are congested all the
time? ]

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-19-2011, 04:43 PM
Matthias Clasen
 
Default tracker in F16

On Mon, 2011-09-19 at 18:00 +0200, Lennart Poettering wrote:
> On Mon, 19.09.11 00:38, Lennart Poettering (mzerqung@0pointer.de) wrote:
>
> > > Having used tracker on by default for a bit now it seems that after the
> > > initial crawling, which is an expensive operation, I didn't notice any
> > > particular increase in resource usage. With removable devices indexing
> > > off, this should ideally be a one-shot operation.
> >
> > I'd be thankful if the impact the crawling has on the running system
> > could be minimized via SCHED_IDLE and IOPRIO_CLASS_IDLE. gnome bz
> > #659422.
>
> Hmm, so investigating this further with strace it appears to me that
> tracker is trying to make use of the kernel in a way it shouldn't. Or to
> put this another way: our infrastructure (the Linux kernel) isn't ready
> for tracker yet.
>
> The problems here have been known for a long time, but afaik there still
> hasn't been done anything about them to make things ready for tracker. I
> am not sure we should enable tracker before these fundamental issues
> aren't fixed. I am completely fine with enabling stuff that has known
> bugs because that's the way how you get them fixed. But in this case
> here I fear that the basics just don't exist and hence tracker is
> fundamentally built on infrastructure that is borked. Everytime people
> have looked into enabling tracker (or beagle for the matter) these
> issues showed up, and every single time nothing happened about them, and
> I am not really seeing why these issues stopped mattering now.
>
> To be more explicit:
>
> a) tracker uses inotify recursively and creates a massive number of
> watches due to that. That is both ugly and doesn't scale. Tracker
> apparently tries to not take up the full pool of inotfy handles the
> system provides, but that won't help if you have more than one user on
> the system. The solution here should probably be fanotify, which allows
> proper recursive file system watches. So far fanotify has been
> accessible to root only, which is presumably why tracker doesn't use
> it. However, the solution here cannot be to work around that fact by
> using inotify, but must be to invest the necessary kernel work to make
> fanotify useful from unprivileged processes.
>
> b) We still don't have a way to detect offline modification of
> directories. That means detecting changes to the home directory made
> offline is very expensive. btrfs now has hooks to improve the situation,
> but ext4 still hasn't. Does tracker at least use the btrfs hooks? (btrfs
> provides a log of changes to userspace, which can be used for that. Another
> solution are recursive directory change timestamps).
>
> I'd really prefer if we could fix these fundamental issues before we
> enable tracker. To me it appears here as if we are trying to make the
> second step before the first.
>
> [ And there are acouple of other things I'd like to see changed. For
> example, I am pretty sure that tracker's open() calls to files should
> not be considered accesses in regards to access time. O_NOATIME should
> be used here, which would reduce the amount of disk writes
> substantially. Also, tracker appears to BSD lock all files it
> accesses. That looks quite borked. Which other tool is it synchronizing
> against here? This looks unsecure to do (because the files are often
> accessible to others), and since these locks are advisory only there
> needs to be a strict protocol followed by everybody else accessing these
> files, which I guarantee you there isn't since these are basically all
> the user's files. Moreover it appears tracker is mixing BSD and POSIX
> locks, which is dangerous due to ABBA, in particular when used on NFS
> directories, which will just end up in total chaos since Linux is so
> stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
> way. In any case you should NEVER EVER use POSIX locking, since it is
> compltely borked anyway. The locking must go. I also see a massive
> amount of futex calls in strace, i.e. probably some mutexes thrown in
> the mix to make the locking problems even more interesting, which makes
> my fingernails roll up, since they apparently are congested all the
> time? ]
>

Good stuff, Lennart.

But it would probably have much more effect in
https://bugzilla.gnome.org/browse.cgi?product=tracker
or
https://mail.gnome.org/archives/tracker-list/

Can you put it there ?

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-19-2011, 06:58 PM
Lennart Poettering
 
Default tracker in F16

On Mon, 19.09.11 12:43, Matthias Clasen (mclasen@redhat.com) wrote:

> > a) tracker uses inotify recursively and creates a massive number of
> > watches due to that. That is both ugly and doesn't scale. Tracker
> > apparently tries to not take up the full pool of inotfy handles the
> > system provides, but that won't help if you have more than one user on
> > the system. The solution here should probably be fanotify, which allows
> > proper recursive file system watches. So far fanotify has been
> > accessible to root only, which is presumably why tracker doesn't use
> > it. However, the solution here cannot be to work around that fact by
> > using inotify, but must be to invest the necessary kernel work to make
> > fanotify useful from unprivileged processes.
> >
> > b) We still don't have a way to detect offline modification of
> > directories. That means detecting changes to the home directory made
> > offline is very expensive. btrfs now has hooks to improve the situation,
> > but ext4 still hasn't. Does tracker at least use the btrfs hooks? (btrfs
> > provides a log of changes to userspace, which can be used for that. Another
> > solution are recursive directory change timestamps).
> >
> > I'd really prefer if we could fix these fundamental issues before we
> > enable tracker. To me it appears here as if we are trying to make the
> > second step before the first.
> >
> > [ And there are acouple of other things I'd like to see changed. For
> > example, I am pretty sure that tracker's open() calls to files should
> > not be considered accesses in regards to access time. O_NOATIME should
> > be used here, which would reduce the amount of disk writes
> > substantially. Also, tracker appears to BSD lock all files it
> > accesses. That looks quite borked. Which other tool is it synchronizing
> > against here? This looks unsecure to do (because the files are often
> > accessible to others), and since these locks are advisory only there
> > needs to be a strict protocol followed by everybody else accessing these
> > files, which I guarantee you there isn't since these are basically all
> > the user's files. Moreover it appears tracker is mixing BSD and POSIX
> > locks, which is dangerous due to ABBA, in particular when used on NFS
> > directories, which will just end up in total chaos since Linux is so
> > stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
> > way. In any case you should NEVER EVER use POSIX locking, since it is
> > compltely borked anyway. The locking must go. I also see a massive
> > amount of futex calls in strace, i.e. probably some mutexes thrown in
> > the mix to make the locking problems even more interesting, which makes
> > my fingernails roll up, since they apparently are congested all the
> > time? ]
> >
>
> Good stuff, Lennart.
>
> But it would probably have much more effect in
> https://bugzilla.gnome.org/browse.cgi?product=tracker
> or
> https://mail.gnome.org/archives/tracker-list/
>
> Can you put it there ?

I already filed two bugs there today. And the inotify/offline
modification issues the tracker folks are well aware of, because people
(including me) told them that over and over again over the years.

This mail was mostly intended as a summary for fedora, and for pointing
out that these core issues have not been addressed in the last
years. There's no news in all of this. Maybe except for the fact that
Fedora appears willing to ignore the issues that previously
mattered.

Anyway, I will forward this to Jürg an Martyn, maybe they have something
to say something new about this.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-19-2011, 10:37 PM
Matthias Clasen
 
Default tracker in F16

On Mon, 2011-09-19 at 20:58 +0200, Lennart Poettering wrote:

> Maybe except for the fact that
> Fedora appears willing to ignore the issues that previously
> mattered.

It is not a Fedora issue, really. GNOME is moving to depend on tracker.
Letting it linger as an optional dependency for another few years is
realistically not going help things along. Do you have any better idea
for how to make progress on getting the necessary infrastructure fixes
in place, other than using the stuff ?

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-19-2011, 10:49 PM
Lennart Poettering
 
Default tracker in F16

On Mon, 19.09.11 18:37, Matthias Clasen (mclasen@redhat.com) wrote:

>
> On Mon, 2011-09-19 at 20:58 +0200, Lennart Poettering wrote:
>
> > Maybe except for the fact that
> > Fedora appears willing to ignore the issues that previously
> > mattered.
>
> It is not a Fedora issue, really. GNOME is moving to depend on tracker.
> Letting it linger as an optional dependency for another few years is
> realistically not going help things along. Do you have any better idea
> for how to make progress on getting the necessary infrastructure fixes
> in place, other than using the stuff ?

Work with Eric to get fanotify fixed up for unprivileged use. fanotify
is already in the kernel, so the big changes have been made. It's just a
matter of building on that, and Eric is a very responsive and friendly
person. The privilege issues could probably be dealt with in a minimal
way already by simply adding some kind of tail-drop logic to the
fanotify queues.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 

Thread Tools




All times are GMT. The time now is 04:31 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org