FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 07-22-2012, 12:00 PM
Nilesh Govindrajan
 
Default Eix takes ridiculously long to update cache of overlays

I guess I'm missing some settings specific to this? I have 3 overlays
installed via layman, and this eix takes ridiculously long to index
through them, I don't know why.


The portage tree is indexed quickly.

--
Nilesh Govindrajan
http://nileshgr.com
 
Old 07-22-2012, 01:15 PM
Florian Philipp
 
Default Eix takes ridiculously long to update cache of overlays

Am 22.07.2012 14:00, schrieb Nilesh Govindrajan:
> I guess I'm missing some settings specific to this? I have 3 overlays
> installed via layman, and this eix takes ridiculously long to index
> through them, I don't know why.
>
> The portage tree is indexed quickly.
>

There is usually not much you can do there. This typically happens with
overlays that contain no metadata cache.

Look at your overlay directory into metadata/layout.conf. If it doesn't
contain a "cache-formats" line you are probably out of luck since eix
has to parse each ebuild.

Read the section about CACHE_METHOD in `man eix`

The default method for overlays is "parse|ebuild*" meaning that if there
is a cache, it will be used and double-checked with a fast parser
("parse"). If there is no cache, the parser will be used alone. If that
does not give satisfying results (e.g. because variables are set in
eclasses instead of ebuilds), the real ebuild parser (i.e. bash) will be
used to parse them ("ebuild*").

The last step is probably what causes the delays. It also poses a
security risk if you don't trust the ebuilds since you are basically
executing them (although with limited permissions).

Things might get better if they ever get libbash [1] finished. But don't
hold your breath.

If you need to update the eix cache on several systems, it might be
faster to generate a cache on one machine and distribute it to all
other. See `man egencache`.

[1] http://qiaomuf.wordpress.com/2011/05/05/introduction-to-libbash/

Regards,
Florian Philipp
 
Old 07-22-2012, 04:17 PM
Vaeth
 
Default Eix takes ridiculously long to update cache of overlays

I guess I'm missing some settings specific to this? I have 3 overlays
installed via layman, and this eix takes ridiculously long to index
through them, I don't know why.

The portage tree is indexed quickly.



There is usually not much you can do there. This typically happens with
overlays that contain no metadata cache.

Look at your overlay directory into metadata/layout.conf. If it doesn't
contain a "cache-formats" line you are probably out of luck since eix
has to parse each ebuild.


It is not the "cache-formats" line which is crucial but whether
there is an (up-to-date) metadata/cache or metadata/md5-cache
subdireectory.
If the overlay does not have such a directory (or it is not
updated regularly), You can let portage generate this directory
by using "egencache --repo=... update".
You can eix-sync automatically call this at every think
(see the example lines for eix-sync in the eix manpage).


The default method for overlays is "parse|ebuild*" meaning that if there
is a cache, it will be used and double-checked with a fast parser
("parse"). If there is no cache, the parser will be used alone. If that
does not give satisfying results (e.g. because variables are set in
eclasses instead of ebuilds), the real ebuild parser (i.e. bash) will be
used to parse them ("ebuild*").


Almost: CACHE_METHOD_PARSE is appended to the default method.
Thus, the actual default method for overlays is something like
"parse|ebuild*#metadata-md5#metadata-flat#assign"
The choices after "#" are various cache methods; if the corresponding
cache is available, this method is used (in the specified order),
if they fail then "parse" is used (which is a fast parser, as you
mentioned, but not reliable) and if that does not give a satisfying
result then the real ebuild parser is used (which is very slow).
An alternative way to speed up eix is to remove the "|ebuild*"
in the overlay cache method: This means that eix will take
the result of "parse" even if it is not perfect. Of course, this
gives some "wrong" information (e.g. wrong SLOT name, missing
dependency information, etc.)


Things might get better if they ever get libbash [1] finished.


I do not think that eix will be using it - it would be a lot
of work and only slightly improve the heuristics (main issue
are that inheriting eclasses currently requires some "wrapper"
interaction, so this cannot be solved with libbash alone.)
Perhaps some day portage might use libbash in "egencache", but
egencache is already surprisingly fast:


If you need to update the eix cache on several systems, it might be
faster to generate a cache on one machine and distribute it to all
other. See `man egencache`.


Actually, egencache is already *much* faster than parsing each ebuild
separately. Moreover, I guess at least with cache-format=md5-dict
it parses really only those ebuilds for which the md5 checksum has
changed (i.e. the first execution might be slow, but after updates
there is not so much delay, usually).
 
Old 07-22-2012, 04:47 PM
Florian Philipp
 
Default Eix takes ridiculously long to update cache of overlays

Am 22.07.2012 18:17, schrieb Vaeth:
>>> I guess I'm missing some settings specific to this? I have 3 overlays
>>> installed via layman, and this eix takes ridiculously long to index
>>> through them, I don't know why.
>>>
>>> The portage tree is indexed quickly.
>>>
>>
>> There is usually not much you can do there. This typically happens with
>> overlays that contain no metadata cache.
>>
>> Look at your overlay directory into metadata/layout.conf. If it doesn't
>> contain a "cache-formats" line you are probably out of luck since eix
>> has to parse each ebuild.
>
> It is not the "cache-formats" line which is crucial but whether
> there is an (up-to-date) metadata/cache or metadata/md5-cache
> subdireectory.
> If the overlay does not have such a directory (or it is not
> updated regularly), You can let portage generate this directory
> by using "egencache --repo=... update".
> You can eix-sync automatically call this at every think
> (see the example lines for eix-sync in the eix manpage).
>
[...]
>
> Actually, egencache is already *much* faster than parsing each ebuild
> separately. Moreover, I guess at least with cache-format=md5-dict
> it parses really only those ebuilds for which the md5 checksum has
> changed (i.e. the first execution might be slow, but after updates
> there is not so much delay, usually).
>

Interesting. Does this interact nicely with layman or do the different
VCSs barf when they find new metadata caches in their overlay directories?

Regards,
Florian Philipp
 

Thread Tools




All times are GMT. The time now is 09:25 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org