FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian Development

 
 
LinkBack Thread Tools
 
Old 08-27-2008, 10:50 AM
"Gustaf Räntilä"
 
Default Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs

Package: general
Version: AMD64

This is basically a debian AMD64-version of the bug report for ubuntu AMD64 bug 24691 [1].

The problem is (seems to be) that a lot of libraries are built with alignment above 2**3. Most of these cases are actually 2**20 in ubuntu AMD64 and 2**21 in debian AMD64. In other words, 1 and 2 MB correspondingly!


I often see loose and vague arguments such as "if 99MB of that is shared, the calculator is really only 'using' 1MB of ram -- and that's fine" [2].
It's not fine. And it's certainly incorrect. Just because a library is shared doesn't mean it's fine that it consumes megabytes(!) of memory in vain. Especially libraries that are "shared" "between" 1 process.

Now, I can't figure out why such huge amounts of memory is hogged on my computer. But I need to restart X about once a week. If I don't, my 4 GB of RAM is quickly filled and my 4 GB swap starts to work (hurray, 15 second delay when changing virtual desktop). It's been like this since I bought this machine (soon 2 years ago), and I frequently update my X driver (-radeon, -radeonhd, fglrx, etc), so I doubt they are to blame for stealing my memory, even though it could've been a good guess.


memstat reports lots and lots of libraries which consumes slightly more than 2 mb each:
$ memstat | grep '.so' | grep -v PID
gives me 551 lines, and by just grasping the result, easily 90% of them are slightly more than 2 mb (2**21 + small stuff).


If these libraries could be built with 2**3 (8 byte) alignment, instead of 2**21 (2 megabytes), I assume, just like the discussion in [1], that at least some memory wouldn't be wasted in vain.
So how many libraries (on my system) are built with 2**21 alignment?


/lib: $ for file in `ls *.so.*` ; do if objdump -x $file | grep -q -e '2**21' ; then echo $file ; fi ; done | wc
99
/usr/lib: $ for file in `ls *.so.*` ; do if objdump -x $file | grep -q -e '2**21' ; then echo $file ; fi ; done | wc

2777
/usr/lib/*: $ for file in `ls */*.so.*` ; do if objdump -x $file | grep -q -e '2**21' ; then echo $file ; fi ; done | wc
396

99 + 2777 + 396 = 3272.
Quite a lot of libraries. Loading them all would require roughly 7 GB. Remember: loading. Not using. And these are just on my system, it's not even close to "all" libraries in debian.


Just looking at how much memory pidgin-specific plugins consumes is frightening:
memstat | grep -E '(purple|pidgin)+.*.so.*'
Returns 88 libraries, ALL consuming slightly more than 2 mb. How many of them are "shared" with any other program than pidgin? I'd say none. But that's just a guess.

So this means at least 200 mb memory usage for pidgin alone? Could this really be the case?!
$ ps aux | grep pidgin
gustaf 30432 0.2 1.7 631196 68532 ? S Aug26 2:36 pidgin
Oh yeah, 631192 kB virtual and 68532 kB resident.


I say this again, some people argues; "but most of that 616 MB is shared so it doesn't matter". It matters, because Linux prefers to swap it, to give place for IO buffers, and when things are being swapped, holy moses, Ctrl+Alt+Backspace is thy saviour.

Pidgin consuming 616 MB virtual memory is just.. Well.. Messed up. To put it lightly.

It could be reasonable to see the rest of the memory hoggers on my system, to make it clear that this really is a big problem.


Please note that pidgin is only on 8th place! This is 'top' sorted by 'M' (memory usage):

31095 gustaf 20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin
32197 gustaf 20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse

30134 root 20 0 625m 256m 14m S 8 6.5 99:41.51 Xorg
2680 gustaf 20 0 479m 103m 25m S 0 2.6 4:07.31 banshee-1
30305 gustaf 20 0 438m 82m 17m S 0 2.1 0:31.87 /usr/lib/ontv/o

1621 gustaf 20 0 304m 76m 14m S 0 1.9 0:42.91 gnome-terminal
6070 clamav 20 0 92524 76m 380 S 0 1.9 0:00.00 clamd
30432 gustaf 20 0 616m 66m 28m S 1 1.7 2:37.08 pidgin


$ free -m
total used free shared buffers cached
Mem: 3934 3891 43 0 331 1186
-/+ buffers/cache: 2373 1561
Swap: 3859 5 3853


Real memory usage of my very recently booted machine (I haven't started even a small subset of the apps I usually run): 2373 MB!

$ uptime
12:27:24 up 21:44, 13 users, load average: 1.09, 1.05, 0.99


I'm not sure my mobo supports much more than what I have, but I'm seriously thinking of spending (too much) money on 8 or 16 GB RAM, just to have a system that is usable.

Oh yeah, I'm using 'sid', maybe that's important to know, by I certainly doubt it.


I haven't seen many discussions about this in debian, and just 1 about ubuntu [1]. It could be that I have gotten things completely the wrong way (then please tell me so), but it could also be that this has been overseen for too long, and now needs serious attention.

Memory is cheap nowadays, but it isn't free. And if we're expected to buy dozens of GB to get a usable system, then, well, debian is getting rather expensive I'm afraid.

Regards,
Gustaf

[1] https://bugs.launchpad.net/ubuntu/+bug/24692

[2] https://lists.ubuntu.com/archives/ubuntu-devel/2005-October/012653.html
 
Old 08-27-2008, 11:21 AM
Samuel Thibault
 
Default Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs

Gustaf Räntilä, le Wed 27 Aug 2008 12:50:45 +0200, a écrit :
> I haven't seen many discussions about this in debian, and just 1 about ubuntu [1]. It could be that I have gotten things completely the wrong way (then please tell me so), but it could also be that this has been overseen for too long, and now needs serious attention.

Do not mistake virtual memory for resident memory. Aligning stuff in
the address space is not a problem, it does _not_ consume any memory
(even less if the library is not even used).

> 31095 gustaf 20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin
> 32197 gustaf 20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse

Here are the suckers, not pidgin. Do not have a look at virtual
numbers, as the name says these are virtual. Have a look at the %MEM or
RES columns instead.

Samuel



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 08-27-2008, 01:50 PM
 
Default Bug#496771: Deb AMD64 eats huge amounts of memory (and babies?) because of badly built libs

On Wed, Aug 27, 2008 at 12:50:45PM +0200, Gustaf R??ntil?? wrote:
> Package: general
> Version: AMD64
>
> This is basically a debian AMD64-version of the bug report for ubuntu
> AMD64 bug 24691 [1].
>
> The problem is (seems to be) that a lot of libraries are built with
> alignment above 2**3. Most of these cases are actually 2**20 in ubuntu
> AMD64 and 2**21 in debian AMD64. In other words, 1 and 2 MB
> correspondingly!
>
> I often see loose and vague arguments such as "if 99MB of that is
> shared, the calculator is really only 'using' 1MB of ram -- and that's
> fine" [2].
> It's not fine. And it's certainly incorrect. Just because a library is
> shared doesn't mean it's fine that it consumes megabytes(!) of memory
> in vain. Especially libraries that are "shared" "between" 1 process.
> Now, I can't figure out why such huge amounts of memory is hogged on
> my computer. But I need to restart X about once a week. If I don't, my
> 4 GB of RAM is quickly filled and my 4 GB swap starts to work (hurray,
> 15 second delay when changing virtual desktop). It's been like this
> since I bought this machine (soon 2 years ago), and I frequently
> update my X driver (-radeon, -radeonhd, fglrx, etc), so I doubt they
> are to blame for stealing my memory, even though it could've been a
> good guess.
>
> memstat reports lots and lots of libraries which consumes slightly
> more than 2 mb each:
> $ memstat | grep '.so' | grep -v PID
> gives me 551 lines, and by just grasping the result, easily 90% of
> them are slightly more than 2 mb (2**21 + small stuff).
>
> If these libraries could be built with 2**3 (8 byte) alignment,
> instead of 2**21 (2 megabytes), I assume, just like the discussion in
> [1], that at least some memory wouldn't be wasted in vain.
> So how many libraries (on my system) are built with 2**21 alignment?
>
> /lib: $ for file in `ls *.so.*` ; do if objdump -x $file | grep -q -e
> '2**21' ; then echo $file ; fi ; done | wc
> 99
> /usr/lib: $ for file in `ls *.so.*` ; do if objdump -x $file | grep
> -q -e '2**21' ; then echo $file ; fi ; done | wc
> 2777
> /usr/lib/*: $ for file in `ls */*.so.*` ; do if objdump -x $file |
> grep -q -e '2**21' ; then echo $file ; fi ; done | wc
> 396
>
> 99 + 2777 + 396 = 3272.
> Quite a lot of libraries. Loading them all would require roughly 7 GB.
> Remember: loading. Not using. And these are just on my system, it's
> not even close to "all" libraries in debian.
>
> Just looking at how much memory pidgin-specific plugins consumes is frightening:
> memstat | grep -E '(purple|pidgin)+.*.so.*'
> Returns 88 libraries, ALL consuming slightly more than 2 mb. How many
> of them are "shared" with any other program than pidgin? I'd say none.
> But that's just a guess.
> So this means at least 200 mb memory usage for pidgin alone? Could
> this really be the case?!
> $ ps aux | grep pidgin
> gustaf 30432 0.2 1.7 631196 68532 ? S Aug26 2:36 pidgin
> Oh yeah, 631192 kB virtual and 68532 kB resident.
>
> I say this again, some people argues; "but most of that 616 MB is
> shared so it doesn't matter". It matters, because Linux prefers to
> swap it, to give place for IO buffers, and when things are being
> swapped, holy moses, Ctrl+Alt+Backspace is thy saviour.
> Pidgin consuming 616 MB virtual memory is just.. Well.. Messed up. To
> put it lightly.
>
> It could be reasonable to see the rest of the memory hoggers on my
> system, to make it clear that this really is a big problem.
>
> Please note that pidgin is only on 8th place! This is 'top' sorted by
> 'M' (memory usage):
>
> 31095 gustaf 20 0 1314m 700m 35m R 30 17.8 252:06.36 firefox-bin
> 32197 gustaf 20 0 1006m 439m 39m S 12 11.2 167:03.50 epiphany-browse
> 30134 root 20 0 625m 256m 14m S 8 6.5 99:41.51 Xorg
> 2680 gustaf 20 0 479m 103m 25m S 0 2.6 4:07.31 banshee-1
> 30305 gustaf 20 0 438m 82m 17m S 0 2.1 0:31.87 /usr/lib/ontv/o
> 1621 gustaf 20 0 304m 76m 14m S 0 1.9 0:42.91 gnome-terminal
> 6070 clamav 20 0 92524 76m 380 S 0 1.9 0:00.00 clamd
> 30432 gustaf 20 0 616m 66m 28m S 1 1.7 2:37.08 pidgin

Firefox and epiphany are complete pigs when it comes to memory use.

On amd64 machines not running those things look absolutely fine.

Firefox causes plenty of swapping on i386 as well.

> $ free -m
> total used free shared buffers cached
> Mem: 3934 3891 43 0 331 1186
> -/+ buffers/cache: 2373 1561
> Swap: 3859 5 3853
>
> Real memory usage of my very recently booted machine (I haven't
> started even a small subset of the apps I usually run): 2373 MB!

Firefox probably consumes over 50% of the ram of all the applications
you are likely to run.

Memory is allocated in 4KB pages (since that is what the hardware
supports unless you think 2MB pages are a good idea, or 1GB pages).
What memory address the pages are mapped to is completely irrelevant.
Hence alignment should not affect memory use.

--
Len Sorensen



--
To UNSUBSCRIBE, email to debian-devel-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 

Thread Tools




All times are GMT. The time now is 01:31 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org