FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > ArchLinux > ArchLinux Pacman Development

 
 
LinkBack Thread Tools
 
Old 09-01-2011, 05:33 PM
Philipp
 
Default pacman -Qs first-run performance

Hi there,
pacman works very well for me, with one exception: searching for
installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
seconds. I also noticed that this seems to be true for the first search
run only, the following ones are less than a second, no matter whether
it's a repo or local search. But if you want to just quickly find out a
thing, 25 seconds or so is a long time.

Anyway, I had a very brief look at the code and am far from
understanding it, but I think libalpm/db.c handles the search, the
package names, descriptions, etc. are stored in a linked list of
structs, the whole thing is cached in memory only and regex wizardry
is used for the search. If that's true, the bottleneck I experience is
the caching.

5 seconds for -Ss is acceptable for me, but I wonder whether there's a
reasonably easy way to improve the 25 seconds of -Qs.

My knowledge of pacman internals is non-existant and my C skills are
minimal, so I don't think I can be a lot of help with this one, but
maybe there's something else I can be of help with instead.

Best regards,
Philipp
 
Old 09-01-2011, 05:38 PM
Karol Blazewicz
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 7:33 PM, Philipp <hollunder@lavabit.com> wrote:
> Hi there,
> pacman works very well for me, with one exception: searching for
> installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
> seconds. I also noticed that this seems to be true for the first search
> run only, the following ones are less than a second, no matter whether
> it's a repo or local search. But if you want to just quickly find out a
> thing, 25 seconds or so is a long time.
>
> Anyway, I had a very brief look at the code and am far from
> understanding it, but I think libalpm/db.c handles the search, the
> package names, descriptions, etc. are stored in a linked list of
> structs, the whole thing is cached in memory only and regex wizardry
> is used for the search. If that's true, the bottleneck I experience is
> the caching.
>
> 5 seconds for -Ss is acceptable for me, but I wonder whether there's a
> reasonably easy way to improve the 25 seconds of -Qs.
>
> My knowledge of pacman internals is non-existant and my C skills are
> minimal, so I don't think I can be a lot of help with this one, but
> maybe there's something else I can be of help with instead.
>
> Best regards,
> Philipp
>
>
>

I'm getting 10s -Qs first-time searches, what harddrive are you using??
 
Old 09-01-2011, 06:14 PM
Denis A. Altoé Falqueto
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher
<hollunder@lavabit.com> wrote:
> Excerpts from Karol Blazewicz's message of 2011-09-01 19:38:32 +0200:
>> On Thu, Sep 1, 2011 at 7:33 PM, Philipp <hollunder@lavabit.com> wrote:
>> > Hi there,
>> > pacman works very well for me, with one exception: searching for
>> > installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
>> > seconds. I also noticed that this seems to be true for the first search
>> > run only, the following ones are less than a second, no matter whether
>> > it's a repo or local search. But if you want to just quickly find out a
>> > thing, 25 seconds or so is a long time.
>> >
>> > Anyway, I had a very brief look at the code and am far from
>> > understanding it, but I think libalpm/db.c handles the search, the
>> > package names, descriptions, etc. are stored in a linked list of
>> > structs, the whole thing is cached in memory only and regex wizardry
>> > is used for the search. If that's true, the bottleneck I experience is
>> > the caching.
>> >
>> > 5 seconds for -Ss is acceptable for me, but I wonder whether there's a
>> > reasonably easy way to improve the 25 seconds of -Qs.
>> >
>> > My knowledge of pacman internals is non-existant and my C skills are
>> > minimal, so I don't think I can be a lot of help with this one, but
>> > maybe there's something else I can be of help with instead.
>> >
>> > Best regards,
>> > Philipp
>> >
>> >
>> >
>>
>> I'm getting 10s -Qs first-time searches, what harddrive are you using??
>
> It's a normal laptop HD, Hitachi HTS54321, 160GB, 5400 rpm.
> It's really kind of weird that the web search takes longer than the
> local search, intuition dictates it should be the other way around.

It is not a web search, though. -Ss searches in /var/lib/pacman/sync
and -Qs searches in /var/lib/pacman/local. Both are limited by the
same I/O bus. Theoretically, you'll have less packages installed than
not installed, so the size of /var/lib/pacman/local should be smaller
than /var/lib/pacman/sync. Could you compare the sizes of those two
directories?

--
A: Because it obfuscates the reading.
Q: Why is top posting so bad?

-------------------------------------------
Denis A. Altoe Falqueto
Linux user #524555
-------------------------------------------
 
Old 09-01-2011, 06:17 PM
Dan McGee
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 1:14 PM, Denis A. Altoé Falqueto
<denisfalqueto@gmail.com> wrote:
> On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher <hollunder@lavabit.com> wrote:
>> It's really kind of weird that the web search takes longer than the
>> local search, intuition dictates it should be the other way around.
>
> It is not a web search, though. -Ss searches in /var/lib/pacman/sync
> and -Qs searches in /var/lib/pacman/local.
Yes, this is definitely not a web search, I'm not sure where that idea
came from? Is our documentation unclear?

> Both are limited by the
> same I/O bus. Theoretically, you'll have less packages installed than
> not installed, so the size of /var/lib/pacman/local should be smaller
> than /var/lib/pacman/sync. Could you compare the sizes of those two
> directories?
This won't really be indicative of anything- the sync databases are
both single-file and compressed, meaning there is no 4K filesystem
block rounding per file and text compresses well, among other things.
More useful numbers would be a lot of the commands I dropped in my
last email.

-Dan
 
Old 09-01-2011, 06:19 PM
Denis A. Altoé Falqueto
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 3:14 PM, Denis A. Altoé Falqueto
<denisfalqueto@gmail.com> wrote:
> On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher
> <hollunder@lavabit.com> wrote:
>> Excerpts from Karol Blazewicz's message of 2011-09-01 19:38:32 +0200:
>>> On Thu, Sep 1, 2011 at 7:33 PM, Philipp <hollunder@lavabit.com> wrote:
>>> > Hi there,
>>> > pacman works very well for me, with one exception: searching for
>>> > installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
>>> > seconds. I also noticed that this seems to be true for the first search
>>> > run only, the following ones are less than a second, no matter whether
>>> > it's a repo or local search. But if you want to just quickly find out a
>>> > thing, 25 seconds or so is a long time.
>>> >
>>> > Anyway, I had a very brief look at the code and am far from
>>> > understanding it, but I think libalpm/db.c handles the search, the
>>> > package names, descriptions, etc. are stored in a linked list of
>>> > structs, the whole thing is cached in memory only and regex wizardry
>>> > is used for the search. If that's true, the bottleneck I experience is
>>> > the caching.
>>> >
>>> > 5 seconds for -Ss is acceptable for me, but I wonder whether there's a
>>> > reasonably easy way to improve the 25 seconds of -Qs.
>>> >
>>> > My knowledge of pacman internals is non-existant and my C skills are
>>> > minimal, so I don't think I can be a lot of help with this one, but
>>> > maybe there's something else I can be of help with instead.
>>> >
>>> > Best regards,
>>> > Philipp
>>> >
>>> >
>>> >
>>>
>>> I'm getting 10s -Qs first-time searches, what harddrive are you using??
>>
>> It's a normal laptop HD, Hitachi HTS54321, 160GB, 5400 rpm.
>> It's really kind of weird that the web search takes longer than the
>> local search, intuition dictates it should be the other way around.
>
> It is not a web search, though. -Ss searches in /var/lib/pacman/sync
> and -Qs searches in /var/lib/pacman/local. Both are limited by the
> same I/O bus. Theoretically, you'll have less packages installed than
> not installed, so the size of /var/lib/pacman/local should be smaller
> than /var/lib/pacman/sync. Could you compare the sizes of those two
> directories?

Replying to myself.... /var/lib/pacman/sync is using tar.gz's to store
the database repositories, so what I said is not true anymore (it was
for old versions os pacman). /var/lib/pacman/sync is smaller than
/var/lib/pacman/local, because local is made from lots of small
files.

Should have tested it before sending the email

--
A: Because it obfuscates the reading.
Q: Why is top posting so bad?

-------------------------------------------
Denis A. Altoe Falqueto
Linux user #524555
-------------------------------------------
 
Old 09-01-2011, 06:22 PM
Philipp Überbacher
 
Default pacman -Qs first-run performance

Excerpts from Dan McGee's message of 2011-09-01 19:54:34 +0200:
> On Thu, Sep 1, 2011 at 12:33 PM, Philipp <hollunder@lavabit.com> wrote:
> > Hi there,
> > pacman works very well for me, with one exception: searching for
> > installed packages. For me a -Ss takes about 5 seconds, a -Qs 25
> > seconds. I also noticed that this seems to be true for the first search
> > run only, the following ones are less than a second, no matter whether
> > it's a repo or local search. But if you want to just quickly find out a
> > thing, 25 seconds or so is a long time.
>
> Of course subsequent runs are faster- the page cache comes into play.

I didn't expect them to help this much.

> 25 seconds is beyond slow. I have some results from my slowest machine
> (HDDs are running using PIO4 right now, not even DMA), and I can't top
> 5 seconds on -Qs or 10 seconds on -Ss *with* caches dropped [1]. The
> first timing in each command is uncached; the second is then the
> cached time.
>
> > Anyway, I had a very brief look at the code and am far from
> > understanding it, but I think libalpm/db.c handles the search, the
> > package names, descriptions, etc. are stored in a linked list of
> > structs, the whole thing is cached in memory only and regex wizardry
> > is used for the search. If that's true, the bottleneck I experience is
> > the caching.
> >
> > 5 seconds for -Ss is acceptable for me, but I wonder whether there's a
> > reasonably easy way to improve the 25 seconds of -Qs.
> No, the bottleneck is your slow hard drive, your lack of RAM, and/or
> your slow CPU. Notice that on a cached search, my times don't exceed
> 0.3 seconds for any operation.
>
> > My knowledge of pacman internals is non-existant and my C skills are
> > minimal, so I don't think I can be a lot of help with this one, but
> > maybe there's something else I can be of help with instead.
> If you could tell us more about your setup, that would be very
> helpful. CPU info from /proc/cpuinfo, memory info from /proc/meminfo
> and `free -m` output, `hdparm -Tt` numbers, etc.

cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 22
model name : Intel(R) Celeron(R) CPU 560 @ 2.13GHz
stepping : 1
cpu MHz : 2128.233
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm pbe syscall nx lm constant_tsc up arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm lahf_lm dts
bogomips : 4258.81
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:


cat /proc/meminfo
MemTotal: 2050332 kB
MemFree: 964024 kB
Buffers: 17888 kB
Cached: 287340 kB
SwapCached: 10868 kB
Active: 487996 kB
Inactive: 276964 kB
Active(anon): 448104 kB
Inactive(anon): 176348 kB
Active(file): 39892 kB
Inactive(file): 100616 kB
Unevictable: 224952 kB
Mlocked: 130928 kB
SwapTotal: 2104508 kB
SwapFree: 2003180 kB
Dirty: 1432 kB
Writeback: 0 kB
AnonPages: 675560 kB
Mapped: 134016 kB
Shmem: 73536 kB
Slab: 46608 kB
SReclaimable: 14012 kB
SUnreclaim: 32596 kB
KernelStack: 1472 kB
PageTables: 8620 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 3129672 kB
Committed_AS: 1367008 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 542528 kB
VmallocChunk: 34359167180 kB
HardwareCorrupted: 0 kB
AnonHugePages: 241664 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 13120 kB
DirectMap2M: 2074624 kB



free -m
total used free shared buffers cached
Mem: 2002 1080 922 0 17 299
-/+ buffers/cache: 763 1239
Swap: 2055 98 1956



hdparm -Tt /dev/sda

/dev/sda:
Timing cached reads: 1426 MB in 2.00 seconds = 712.73 MB/sec
Timing buffered disk reads: 166 MB in 3.01 seconds = 55.09 MB/sec


> We've massively sped up pacman database reading for both hot and cold
> cases over what it was a year or so ago, so most of us consider this a
> "solved" problem by now. None of us see anywhere near the times you
> are seeing for operations, so anything you can provide us would be
> helpful.
>
> The output from these two commands might also be useful:
>
> echo 3 > /proc/sys/vm/drop_caches; /usr/bin/time pacman -Qs foobarbaz;
> /usr/bin/time pacman -Qs foobarbaz
> echo 3 > /proc/sys/vm/drop_caches; /usr/bin/time pacman -Ss foobarbaz;
> /usr/bin/time pacman -Ss foobarbaz
>
> -Dan


echo 3 > /proc/sys/vm/drop_caches; time pacman -Qs foobarbaz; time pacman -Qs foobarbaz

real 1m45.235s
user 0m0.127s
sys 0m0.590s

real 0m0.097s
user 0m0.043s
sys 0m0.023s

echo 3 > /proc/sys/vm/drop_caches; time pacman -Ss foobarbaz; time pacman -Ss foobarbaz

real 0m1.299s
user 0m0.200s
sys 0m0.023s

real 0m0.252s
user 0m0.200s
sys 0m0.010s



My 25 seconds weren't measured, it seems like it's even worse in reality.


Best regards,
Philipp

> [1]
> [root@dublin ~]# echo 3 > /proc/sys/vm/drop_caches; time pacman -Ss
> foobarbaz; time pacman -Ss foobarbaz
>
> real 0m5.533s
> user 0m0.360s
> sys 0m0.553s
>
> real 0m0.349s
> user 0m0.343s
> sys 0m0.007s
>
> [root@dublin ~]# echo 3 > /proc/sys/vm/drop_caches; time pacman -Qs
> foobarbaz; time pacman -Qs foobarbaz
>
> real 0m10.030s
> user 0m0.050s
> sys 0m0.290s
>
> real 0m0.063s
> user 0m0.040s
> sys 0m0.020s
>
 
Old 09-01-2011, 06:37 PM
Philipp Überbacher
 
Default pacman -Qs first-run performance

Excerpts from Dan McGee's message of 2011-09-01 20:17:47 +0200:
> On Thu, Sep 1, 2011 at 1:14 PM, Denis A. Altoé Falqueto
> <denisfalqueto@gmail.com> wrote:
> > On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher <hollunder@lavabit.com> wrote:
> >> It's really kind of weird that the web search takes longer than the
> >> local search, intuition dictates it should be the other way around.
> >
> > It is not a web search, though. -Ss searches in /var/lib/pacman/sync
> > and -Qs searches in /var/lib/pacman/local.
> Yes, this is definitely not a web search, I'm not sure where that idea
> came from? Is our documentation unclear?

It simply arose because it searches for packages that aren't necessarily
installed. If it only searches everything it knows from the last 'y'
that's fine too.

> > Both are limited by the
> > same I/O bus. Theoretically, you'll have less packages installed than
> > not installed, so the size of /var/lib/pacman/local should be smaller
> > than /var/lib/pacman/sync. Could you compare the sizes of those two
> > directories?
> This won't really be indicative of anything- the sync databases are
> both single-file and compressed, meaning there is no 4K filesystem
> block rounding per file and text compresses well, among other things.
> More useful numbers would be a lot of the commands I dropped in my
> last email.
>
> -Dan

I ran ncdu anyway, local has 42MiB, sync 1.2MiB. The more interesting
numbers are: Items: 4606.
I don't know whether Items includes directories. sync accounts for 6
files, so from the file structure I assume I have either about 1500 or
2300 packages installed, but there's probably a better way to find out.

Regards,
Philipp
 
Old 09-01-2011, 06:40 PM
Karol Blazewicz
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 8:37 PM, Philipp Überbacher
<hollunder@lavabit.com> wrote:
> Excerpts from Dan McGee's message of 2011-09-01 20:17:47 +0200:
>> On Thu, Sep 1, 2011 at 1:14 PM, Denis A. Altoé Falqueto
>> <denisfalqueto@gmail.com> wrote:
>> > On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher <hollunder@lavabit.com> wrote:
>> >> It's really kind of weird that the web search takes longer than the
>> >> local search, intuition dictates it should be the other way around.
>> >
>> > It is not a web search, though. -Ss searches in /var/lib/pacman/sync
>> > and -Qs searches in /var/lib/pacman/local.
>> Yes, this is definitely not a web search, I'm not sure where that idea
>> came from? Is our documentation unclear?
>
> It simply arose because it searches for packages that aren't necessarily
> installed. If it only searches everything it knows from the last 'y'
> that's fine too.
>
>> > Both are limited by the
>> > same I/O bus. Theoretically, you'll have less packages installed than
>> > not installed, so the size of /var/lib/pacman/local should be smaller
>> > than /var/lib/pacman/sync. Could you compare the sizes of those two
>> > directories?
>> This won't really be indicative of anything- the sync databases are
>> both single-file and compressed, meaning there is no 4K filesystem
>> block rounding per file and text compresses well, among other things.
>> More useful numbers would be a lot of the commands I dropped in my
>> last email.
>>
>> -Dan
>
> I ran ncdu anyway, local has 42MiB, sync 1.2MiB. The more interesting
> numbers are: Items: 4606.
> I don't know whether Items includes directories. sync accounts for 6
> files, so from the file structure I assume I have either about 1500 or
> 2300 packages installed, but there's probably a better way to find out.
>
> Regards,
> Philipp
>
>
>

'pacman -Qq | wc -l' will tell you how many packages have you installed.
My sync is 1.3 MB, local 8.1 MB, 1375 items.
 
Old 09-01-2011, 06:41 PM
Dan McGee
 
Default pacman -Qs first-run performance

On Thu, Sep 1, 2011 at 1:22 PM, Philipp Überbacher
<hollunder@lavabit.com> wrote:
> Excerpts from Dan McGee's message of 2011-09-01 19:54:34 +0200:
> *free -m
> * * * * * * total * * * used * * * free * * shared * *buffers * * cached
> Mem: * * * * *2002 * * * 1080 * * * *922 * * * * *0 * * * * 17 * * * *299
> -/+ buffers/cache: * * * *763 * * * 1239
> Swap: * * * * 2055 * * * * 98 * * * 1956
Wow- what are you running on a laptop that is keeping 763 MB of RAM
pegged? Although there appears to be something else in play here.

> *hdparm -Tt /dev/sda
>
> /dev/sda:
> *Timing cached reads: * 1426 MB in *2.00 seconds = 712.73 MB/sec
> *Timing buffered disk reads: 166 MB in *3.01 seconds = *55.09 MB/sec
This is a bit slow, but nothing that should impact your performance that much.

> *echo 3 > /proc/sys/vm/drop_caches; time pacman -Qs foobarbaz; time pacman -Qs foobarbaz
>
> real * *1m45.235s
> user * *0m0.127s
> sys * * 0m0.590s
>
> real * *0m0.097s
> user * *0m0.043s
> sys * * 0m0.023s
Can you repeat this and use /usr/bin/time instead of just plain
"time"? You may have to install the "time" package.
echo 3 > /proc/sys/vm/drop_caches; /usr/bin/time pacman -Qs foobarbaz;
/usr/bin/time pacman -Qs foobarbaz

Once you do that (but please get the numbers first!), try running
`pacman-optimize` then rerunning the above test.

Finally, what is the output of df -h for whatever drive /var/lib is
on? Is it near capacity? And what filesystem with what mount options?

> *echo 3 > /proc/sys/vm/drop_caches; time pacman -Ss foobarbaz; time pacman -Ss foobarbaz
>
> real * *0m1.299s
> user * *0m0.200s
> sys * * 0m0.023s
>
> real * *0m0.252s
> user * *0m0.200s
> sys * * 0m0.010s

At least this one is OK.

-Dan
 
Old 09-01-2011, 06:50 PM
Philipp Überbacher
 
Default pacman -Qs first-run performance

Excerpts from Karol Blazewicz's message of 2011-09-01 20:40:52 +0200:
> On Thu, Sep 1, 2011 at 8:37 PM, Philipp Überbacher
> <hollunder@lavabit.com> wrote:
> > Excerpts from Dan McGee's message of 2011-09-01 20:17:47 +0200:
> >> On Thu, Sep 1, 2011 at 1:14 PM, Denis A. Altoé Falqueto
> >> <denisfalqueto@gmail.com> wrote:
> >> > On Thu, Sep 1, 2011 at 2:57 PM, Philipp Überbacher <hollunder@lavabit.com> wrote:
> >> >> It's really kind of weird that the web search takes longer than the
> >> >> local search, intuition dictates it should be the other way around.
> >> >
> >> > It is not a web search, though. -Ss searches in /var/lib/pacman/sync
> >> > and -Qs searches in /var/lib/pacman/local.
> >> Yes, this is definitely not a web search, I'm not sure where that idea
> >> came from? Is our documentation unclear?
> >
> > It simply arose because it searches for packages that aren't necessarily
> > installed. If it only searches everything it knows from the last 'y'
> > that's fine too.
> >
> >> > Both are limited by the
> >> > same I/O bus. Theoretically, you'll have less packages installed than
> >> > not installed, so the size of /var/lib/pacman/local should be smaller
> >> > than /var/lib/pacman/sync. Could you compare the sizes of those two
> >> > directories?
> >> This won't really be indicative of anything- the sync databases are
> >> both single-file and compressed, meaning there is no 4K filesystem
> >> block rounding per file and text compresses well, among other things.
> >> More useful numbers would be a lot of the commands I dropped in my
> >> last email.
> >>
> >> -Dan
> >
> > I ran ncdu anyway, local has 42MiB, sync 1.2MiB. The more interesting
> > numbers are: Items: 4606.
> > I don't know whether Items includes directories. sync accounts for 6
> > files, so from the file structure I assume I have either about 1500 or
> > 2300 packages installed, but there's probably a better way to find out.
> >
> > Regards,
> > Philipp
> >
> >
> >
>
> 'pacman -Qq | wc -l' will tell you how many packages have you installed.
> My sync is 1.3 MB, local 8.1 MB, 1375 items.

$ pacman -Qq | wc -l
1429

Lots installed, few programs running.
 

Thread Tools




All times are GMT. The time now is 09:27 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org