FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora Desktop

 
 
LinkBack Thread Tools
 
Old 09-20-2011, 12:11 AM
Peter Robinson
 
Default tracker in F16

On Mon, Sep 19, 2011 at 11:37 PM, Matthias Clasen <mclasen@redhat.com> wrote:
> On Mon, 2011-09-19 at 20:58 +0200, Lennart Poettering wrote:
>
>> Maybe except for the fact that
>> Fedora appears willing to ignore the issues that previously
>> mattered.
>
> It is not a Fedora issue, really. GNOME is moving to depend on tracker.
> Letting it linger as an optional dependency for another few years is
> realistically not going help things along. Do you have any better idea
> for how to make progress on getting the necessary infrastructure fixes
> in place, other than using the stuff ?

Correct, its not a Fedora issue but its not going to help Fedora's
cause if on most new installs it burns CPU and makes the end users
experience horrible. Having initially dealt with tracker when
packaging both rygel and moblin it has the same problems back in
Fedora 12 and I got hate mail, and from memory bug reports from
Lennart (the bug report was Lennart, not the mail) about the issues
and upstream weren't overly interested in fixing them so the solution
was to make the dependency optional. I like the principal behind
tracker (and beagle before it) but if in the short term the problem
isn't going to result in Fedora being usable (and yes, I'm aware
initial indexing is always going to be a problem, but it needs to
detect a fresh install and not grind the machine for 3 hours to know
the current user has nothing in their home directory external media -
that's what it did on my fresh install no media netbook) for the vast
majority of users we need to make it optional for F-16 and re-review
it because ultimately end users won't tolerate it and will walk with
their feet elsewhere, just like they have from Ubuntu to Fedora
because they don't like Unity. We don't want all the new Power Users
to leave 6 months after they've arrived.

Peter
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-20-2011, 08:46 AM
Jürg Billeter
 
Default tracker in F16

On Mon, 2011-09-19 at 18:00 +0200, Lennart Poettering wrote:
> a) tracker uses inotify recursively and creates a massive number of
> watches due to that. That is both ugly and doesn't scale. Tracker
> apparently tries to not take up the full pool of inotfy handles the
> system provides, but that won't help if you have more than one user on
> the system. The solution here should probably be fanotify, which allows
> proper recursive file system watches. So far fanotify has been
> accessible to root only, which is presumably why tracker doesn't use
> it. However, the solution here cannot be to work around that fact by
> using inotify, but must be to invest the necessary kernel work to make
> fanotify useful from unprivileged processes.

The current recursive inotify watches are definitely far from ideal. We
were looking forward to fanotify entering mainline, however, last time I
checked it was still missing too much functionality to even match
inotify for our purposes. In addition to permission issues it did not
support notification of directory events at all. If I remember
correctly, this was all planned for future iterations but not considered
a priority use case for fanotify. Does anyone know Eric's current plans?

The max_user_watches limit is per inotify instance and each user can
have max_user_instances inotify instances, as far as I know. I don't see
how this would be an issue on multi-user systems. The default limit
should be sufficient for a large part of the user base with the default
tracker configuration. However, it might make sense to increase the
default on distributions using tracker to be on the safe side.

> b) We still don't have a way to detect offline modification of
> directories. That means detecting changes to the home directory made
> offline is very expensive. btrfs now has hooks to improve the situation,
> but ext4 still hasn't. Does tracker at least use the btrfs hooks? (btrfs
> provides a log of changes to userspace, which can be used for that. Another
> solution are recursive directory change timestamps).

No, tracker does not have any btrfs-specific code at the moment. I'm
still waiting for a btrfs fsck release to start using btrfs myself, but
I'm looking forward to learning more about the btrfs facilities and how
we could use them to improve tracker.

> I'd really prefer if we could fix these fundamental issues before we
> enable tracker. To me it appears here as if we are trying to make the
> second step before the first.

I think you were in the room when we discussed these issues in Gran
Canaria. At that point the plan was to wait for fanotify and try to
convince filesystem developers of recursive directory timestamps. If I
remember correctly, Matthew Garrett volunteered to talk to other kernel
developers as a next step. Unfortunately, I don't think we had any
follow-up discussions about recursive directory timestamps.

fanotify was delayed quite a bit, and we were told that there was
nothing we can do to help and it was way too early to start
experimenting with it for tracker. Now that it is in mainline, it would
probably be easier to help out, but given that it took a long time for a
kernel developer to get a subset of the planned features into mainline,
I didn't attempt to work on the missing features myself so far.

> [ And there are acouple of other things I'd like to see changed. For
> example, I am pretty sure that tracker's open() calls to files should
> not be considered accesses in regards to access time. O_NOATIME should
> be used here, which would reduce the amount of disk writes

We already use O_NOATIME in a few extractors, however, certain libraries
don't make this very easy. Not even GIO allows to open a file for
reading with O_NOATIME set, as far as I can tell. Also, I don't expect
the amount of atime-related writes to be very high with relatime, but I
haven't measured this and could be mistaken.

> substantially. Also, tracker appears to BSD lock all files it
> accesses. That looks quite borked. Which other tool is it synchronizing
> against here? This looks unsecure to do (because the files are often
> accessible to others), and since these locks are advisory only there
> needs to be a strict protocol followed by everybody else accessing these
> files, which I guarantee you there isn't since these are basically all
> the user's files. Moreover it appears tracker is mixing BSD and POSIX
> locks, which is dangerous due to ABBA, in particular when used on NFS
> directories, which will just end up in total chaos since Linux is so
> stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
> way. In any case you should NEVER EVER use POSIX locking, since it is
> compltely borked anyway. The locking must go. I also see a massive
> amount of futex calls in strace, i.e. probably some mutexes thrown in
> the mix to make the locking problems even more interesting, which makes
> my fingernails roll up, since they apparently are congested all the
> time? ]

Can you please share your findings in a bug report? As far as I know,
tracker itself doesn't use BSD or POSIX locks at all. SQLite is using
file locks, but if there are issues with how SQLite is using locks, this
should probably be discussed with SQLite upstream.

As a side-note, I myself would like to see more radical changes in how
user files will be stored in the future. Ideally, we would stop storing
them in traditional directory hierarchies. Among other things, this
would completely avoid the need for recursive directory monitoring,
recursive directory timestamps, and crawling on startup. On the
downside, this would require changes in many applications, although FUSE
could certainly help providing a compatibility layer. If anyone is
interested in discussing or working on this, let me know.

Regards,
Jürg


--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-20-2011, 11:27 AM
Lennart Poettering
 
Default tracker in F16

On Tue, 20.09.11 10:46, Jürg Billeter (j@bitron.ch) wrote:

>
> On Mon, 2011-09-19 at 18:00 +0200, Lennart Poettering wrote:
> > a) tracker uses inotify recursively and creates a massive number of
> > watches due to that. That is both ugly and doesn't scale. Tracker
> > apparently tries to not take up the full pool of inotfy handles the
> > system provides, but that won't help if you have more than one user on
> > the system. The solution here should probably be fanotify, which allows
> > proper recursive file system watches. So far fanotify has been
> > accessible to root only, which is presumably why tracker doesn't use
> > it. However, the solution here cannot be to work around that fact by
> > using inotify, but must be to invest the necessary kernel work to make
> > fanotify useful from unprivileged processes.
>
> The current recursive inotify watches are definitely far from ideal. We
> were looking forward to fanotify entering mainline, however, last time I
> checked it was still missing too much functionality to even match
> inotify for our purposes. In addition to permission issues it did not
> support notification of directory events at all. If I remember
> correctly, this was all planned for future iterations but not considered
> a priority use case for fanotify. Does anyone know Eric's current
> plans?

Eric is an awesome, very responsive guy. You can catch him on IRC
easily. Talk to him directly!

> > I'd really prefer if we could fix these fundamental issues before we
> > enable tracker. To me it appears here as if we are trying to make the
> > second step before the first.
>
> I think you were in the room when we discussed these issues in Gran
> Canaria. At that point the plan was to wait for fanotify and try to
> convince filesystem developers of recursive directory timestamps. If I
> remember correctly, Matthew Garrett volunteered to talk to other kernel
> developers as a next step. Unfortunately, I don't think we had any
> follow-up discussions about recursive directory timestamps.
>
> fanotify was delayed quite a bit, and we were told that there was
> nothing we can do to help and it was way too early to start
> experimenting with it for tracker. Now that it is in mainline, it would
> probably be easier to help out, but given that it took a long time for a
> kernel developer to get a subset of the planned features into mainline,
> I didn't attempt to work on the missing features myself so far.

GC is three years ago. a lot of time lost by not getting the
kernel fixed.

> > [ And there are acouple of other things I'd like to see changed. For
> > example, I am pretty sure that tracker's open() calls to files should
> > not be considered accesses in regards to access time. O_NOATIME should
> > be used here, which would reduce the amount of disk writes
>
> We already use O_NOATIME in a few extractors, however, certain libraries
> don't make this very easy. Not even GIO allows to open a file for
> reading with O_NOATIME set, as far as I can tell. Also, I don't expect
> the amount of atime-related writes to be very high with relatime, but I
> haven't measured this and could be mistaken.

GIO and all the libs you are using are open source, so prepare patches!
I am quite sure you even have commit access to gio, right?

> > substantially. Also, tracker appears to BSD lock all files it
> > accesses. That looks quite borked. Which other tool is it synchronizing
> > against here? This looks unsecure to do (because the files are often
> > accessible to others), and since these locks are advisory only there
> > needs to be a strict protocol followed by everybody else accessing these
> > files, which I guarantee you there isn't since these are basically all
> > the user's files. Moreover it appears tracker is mixing BSD and POSIX
> > locks, which is dangerous due to ABBA, in particular when used on NFS
> > directories, which will just end up in total chaos since Linux is so
> > stupid to "upgrade" BSD locks on NFS shares to POSIX locks on the
> > way. In any case you should NEVER EVER use POSIX locking, since it is
> > compltely borked anyway. The locking must go. I also see a massive
> > amount of futex calls in strace, i.e. probably some mutexes thrown in
> > the mix to make the locking problems even more interesting, which makes
> > my fingernails roll up, since they apparently are congested all the
> > time? ]
>
> Can you please share your findings in a bug report? As far as I know,
> tracker itself doesn't use BSD or POSIX locks at all. SQLite is using
> file locks, but if there are issues with how SQLite is using locks, this
> should probably be discussed with SQLite upstream.

My findings are basically a straight-forward reading of "strace -p
$(pidof tracker-miner-fs)" which I ran to track down that .desktop file
looping issue i reported.

> As a side-note, I myself would like to see more radical changes in how
> user files will be stored in the future. Ideally, we would stop storing
> them in traditional directory hierarchies. Among other things, this
> would completely avoid the need for recursive directory monitoring,
> recursive directory timestamps, and crawling on startup. On the
> downside, this would require changes in many applications, although FUSE
> could certainly help providing a compatibility layer. If anyone is
> interested in discussing or working on this, let me know.

Personally I must say I see big value in supporting all kinds of files,
not just files generated by GNOME apps. For me personally having tracker
index my source code sounds like the most exciting use of tracker at all.

Lennart

--
Lennart Poettering - Red Hat, Inc.
--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 
Old 09-20-2011, 12:46 PM
Matthias Clasen
 
Default tracker in F16

On Tue, 2011-09-20 at 10:46 +0200, Jürg Billeter wrote:

> fanotify was delayed quite a bit, and we were told that there was
> nothing we can do to help and it was way too early to start
> experimenting with it for tracker. Now that it is in mainline, it would
> probably be easier to help out, but given that it took a long time for a
> kernel developer to get a subset of the planned features into mainline,
> I didn't attempt to work on the missing features myself so far.

Looks like the 'wait for the kernel to spontaneously grow the features
we need' approach is not working great here. You should make your case
for what you want to see in fanotify, and push for it.

> We already use O_NOATIME in a few extractors, however, certain libraries
> don't make this very easy. Not even GIO allows to open a file for
> reading with O_NOATIME set, as far as I can tell. Also, I don't expect
> the amount of atime-related writes to be very high with relatime, but I
> haven't measured this and could be mistaken.

A GIO feature request for this has been filed ?

> As a side-note, I myself would like to see more radical changes in how
> user files will be stored in the future. Ideally, we would stop storing
> them in traditional directory hierarchies. Among other things, this
> would completely avoid the need for recursive directory monitoring,
> recursive directory timestamps, and crawling on startup. On the
> downside, this would require changes in many applications, although FUSE
> could certainly help providing a compatibility layer. If anyone is
> interested in discussing or working on this, let me know.

I fear that this kind of 'radical' vision is not going to help making
tracker successful in the short to medium term. I would really like to
see tracker be successful in the use cases that it currently claims to
cover before I would trust all my data to it...if ever.

Matthias

--
desktop mailing list
desktop@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/desktop
 

Thread Tools




All times are GMT. The time now is 10:32 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org