Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   CentOS (http://www.linux-archive.org/centos/)
-   -   High memory needs (http://www.linux-archive.org/centos/707575-high-memory-needs.html)

Jérémie Dubois-Lacoste 09-26-2012 04:14 PM

High memory needs
 
Dear All,

We recently reinstalled our computing cluster. We were using CentOS
5.3 (32 bits). It is now CentOS 6.3 (64 bits), installed from the
CentOS 6.2 x64 CD, then upgraded to 6.3.

We have some issues with the memory needs of our running jobs. They
require much more than before, it may be due to the switch from 32 to
64 bits, but to me this cannot explain the whole difference.

Here are our investigations.

We used the following simple benchmark:

1. Run a python script and check the memory that
it requires (field "VIRT" of the "top" command).
This script is:
----
import time
time.sleep(30)
print("done")
----

2. Similarly, run and check the memory of a simple
bash script:
----
#!/bin/bash
sleep 30
echo "done"
----

3. Open a R session and check the memory used


I asked 10 of our users to run these three things on their personal
PCs. They are running different distributions (mainly ubuntu,
slackware), half of them use a 32 bits system, the other half a 64
one. Here is a summary of the results:

Bash script:
Avg Min Max
32 bits 5400 4192 9024
64 bits 12900 10000 16528

Python script:
Avg Min Max
32 bits 8500 5004 11132
64 bits 32800 30000 36336

R:
Avg Min Max
32 bits 26900 21000 33452
64 bits 100200 93008 97496

(as a side remark, the difference between 32 and 64 is surprisingly
big to me...).

Then we ran the same things on our CentOS cluster, getting
surprisingly high results. I installed a machine from scratch with the
CentOS CD (6.2 x64) to be sure another component of the cluster was
not playing a role. On this freshly installed machine I get the
following results:
SH: 103MB
PYTHON: 114MB
R: 200MB

So, compared to the highest of our users (among the 64 bits ones), we
have a ratio of ~7, ~3, ~2, respectively.


It is very problematic for us because many jobs now cannot run
properly, because they lack memory on most of our computing nodes.
So we really cannot stand the situation...

Do you see any reason for this? Do you have suggestions?

Sincerely,

Jérémie
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Adrian Sevcenco 09-26-2012 04:37 PM

High memory needs
 
On 09/26/12 19:14, Jérémie Dubois-Lacoste wrote:

Dear All,

Hi!


We recently reinstalled our computing cluster. We were using CentOS
5.3 (32 bits). It is now CentOS 6.3 (64 bits), installed from the
CentOS 6.2 x64 CD, then upgraded to 6.3.

We have some issues with the memory needs of our running jobs. They
require much more than before, it may be due to the switch from 32 to
64 bits, but to me this cannot explain the whole difference.

it would seem that there is a malloc(glibc) behaviour ...
i seen in other list an advice to use :
export MALLOC_ARENA_MAX=1
export MALLOC_MMAP_THRESHOLD=131072

in order to decrease the used memory ..

HTH,
Adrian



Here are our investigations.

We used the following simple benchmark:

1. Run a python script and check the memory that
it requires (field "VIRT" of the "top" command).
This script is:
----
import time
time.sleep(30)
print("done")
----

2. Similarly, run and check the memory of a simple
bash script:
----
#!/bin/bash
sleep 30
echo "done"
----

3. Open a R session and check the memory used


I asked 10 of our users to run these three things on their personal
PCs. They are running different distributions (mainly ubuntu,
slackware), half of them use a 32 bits system, the other half a 64
one. Here is a summary of the results:

Bash script:
Avg Min Max
32 bits 5400 4192 9024
64 bits 12900 10000 16528

Python script:
Avg Min Max
32 bits 8500 5004 11132
64 bits 32800 30000 36336

R:
Avg Min Max
32 bits 26900 21000 33452
64 bits 100200 93008 97496

(as a side remark, the difference between 32 and 64 is surprisingly
big to me...).

Then we ran the same things on our CentOS cluster, getting
surprisingly high results. I installed a machine from scratch with the
CentOS CD (6.2 x64) to be sure another component of the cluster was
not playing a role. On this freshly installed machine I get the
following results:
SH: 103MB
PYTHON: 114MB
R: 200MB

So, compared to the highest of our users (among the 64 bits ones), we
have a ratio of ~7, ~3, ~2, respectively.


It is very problematic for us because many jobs now cannot run
properly, because they lack memory on most of our computing nodes.
So we really cannot stand the situation...

Do you see any reason for this? Do you have suggestions?

Sincerely,

Jérémie
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos




_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Jérémie Dubois-Lacoste 09-26-2012 04:50 PM

High memory needs
 
Hm, interesting suggestion. But didn't change anything. :(
Thanks anyway,

Jérémie



2012/9/26 Adrian Sevcenco <Adrian.Sevcenco@cern.ch>:
> On 09/26/12 19:14, Jérémie Dubois-Lacoste wrote:
>>
>> Dear All,
>
> Hi!
>
>
>> We recently reinstalled our computing cluster. We were using CentOS
>> 5.3 (32 bits). It is now CentOS 6.3 (64 bits), installed from the
>> CentOS 6.2 x64 CD, then upgraded to 6.3.
>>
>> We have some issues with the memory needs of our running jobs. They
>> require much more than before, it may be due to the switch from 32 to
>> 64 bits, but to me this cannot explain the whole difference.
>
> it would seem that there is a malloc(glibc) behaviour ...
> i seen in other list an advice to use :
> export MALLOC_ARENA_MAX=1
> export MALLOC_MMAP_THRESHOLD=131072
>
> in order to decrease the used memory ..
>
> HTH,
> Adrian
>
>>
>> Here are our investigations.
>>
>> We used the following simple benchmark:
>>
>> 1. Run a python script and check the memory that
>> it requires (field "VIRT" of the "top" command).
>> This script is:
>> ----
>> import time
>> time.sleep(30)
>> print("done")
>> ----
>>
>> 2. Similarly, run and check the memory of a simple
>> bash script:
>> ----
>> #!/bin/bash
>> sleep 30
>> echo "done"
>> ----
>>
>> 3. Open a R session and check the memory used
>>
>>
>> I asked 10 of our users to run these three things on their personal
>> PCs. They are running different distributions (mainly ubuntu,
>> slackware), half of them use a 32 bits system, the other half a 64
>> one. Here is a summary of the results:
>>
>> Bash script:
>> Avg Min Max
>> 32 bits 5400 4192 9024
>> 64 bits 12900 10000 16528
>>
>> Python script:
>> Avg Min Max
>> 32 bits 8500 5004 11132
>> 64 bits 32800 30000 36336
>>
>> R:
>> Avg Min Max
>> 32 bits 26900 21000 33452
>> 64 bits 100200 93008 97496
>>
>> (as a side remark, the difference between 32 and 64 is surprisingly
>> big to me...).
>>
>> Then we ran the same things on our CentOS cluster, getting
>> surprisingly high results. I installed a machine from scratch with the
>> CentOS CD (6.2 x64) to be sure another component of the cluster was
>> not playing a role. On this freshly installed machine I get the
>> following results:
>> SH: 103MB
>> PYTHON: 114MB
>> R: 200MB
>>
>> So, compared to the highest of our users (among the 64 bits ones), we
>> have a ratio of ~7, ~3, ~2, respectively.
>>
>>
>> It is very problematic for us because many jobs now cannot run
>> properly, because they lack memory on most of our computing nodes.
>> So we really cannot stand the situation...
>>
>> Do you see any reason for this? Do you have suggestions?
>>
>> Sincerely,
>>
>> Jérémie
>> _______________________________________________
>> CentOS mailing list
>> CentOS@centos.org
>> http://lists.centos.org/mailman/listinfo/centos
>>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

09-26-2012 05:56 PM

High memory needs
 
Jérémie Dubois-Lacoste wrote:
> Dear All,
>
> We recently reinstalled our computing cluster. We were using CentOS
> 5.3 (32 bits). It is now CentOS 6.3 (64 bits), installed from the
> CentOS 6.2 x64 CD, then upgraded to 6.3.
>
> We have some issues with the memory needs of our running jobs. They
> require much more than before, it may be due to the switch from 32 to
> 64 bits, but to me this cannot explain the whole difference.

Why not? The numbers you post, unless I'm misreading them, are about twice
what the 32-bit were.
<nsip>
> 3. Open a R session and check the memory used
<snip>
> Bash script:
> Avg Min Max
> 32 bits 5400 4192 9024
> 64 bits 12900 10000 16528

5400 * 2 = 10800
4192 * 2 = 8296
9024 * 2 = 18048
>
> Python script:
> Avg Min Max
> 32 bits 8500 5004 11132
> 64 bits 32800 30000 36336

8500 * 2 = 17000
5004 * 2 = 10007
11132 * 2 = 22264

So that ranges from 2-2.5 larger.
>
> R:
> Avg Min Max
> 32 bits 26900 21000 33452
> 64 bits 100200 93008 97496

Same here, about 2 - 2.5 times larger. More larger variables.
<snip>
> Then we ran the same things on our CentOS cluster, getting
> surprisingly high results. I installed a machine from scratch with the
> CentOS CD (6.2 x64) to be sure another component of the cluster was
> not playing a role. On this freshly installed machine I get the
> following results:
> SH: 103MB
> PYTHON: 114MB
> R: 200MB
>
> So, compared to the highest of our users (among the 64 bits ones), we
> have a ratio of ~7, ~3, ~2, respectively.
>
> It is very problematic for us because many jobs now cannot run
> properly, because they lack memory on most of our computing nodes.
> So we really cannot stand the situation...
>
> Do you see any reason for this? Do you have suggestions?
>
First, what kind of compute cluster is this? Are you using something like
torque, or what? Second, how much memory do you have in each of the nodes?
And how many cores?

mark

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Michael Hennebry 09-26-2012 06:16 PM

High memory needs
 
On Wed, 26 Sep 2012, m.roth@5-cent.us wrote:


Jérémie Dubois-Lacoste wrote:



Python script:
Avg Min Max
32 bits 8500 5004 11132
64 bits 32800 30000 36336


8500 * 2 = 17000
5004 * 2 = 10007
11132 * 2 = 22264

So that ranges from 2-2.5 larger.


Huh?
3*8500=25500
< 32800
3*5004=15012
< 30000
3*11132=33396
< 36336


R:
Avg Min Max
32 bits 26900 21000 33452
64 bits 100200 93008 97496


3*26900= 80700
< 100200
4*21000=84000
< 93008
2.9*33452< 97011
< 97496

--
Michael hennebry@web.cs.ndsu.NoDak.edu
"On Monday, I'm gonna have to tell my kindergarten class,
whom I teach not to run with scissors,
that my fiance ran me through with a broadsword." -- Lily______________________________________________ _
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Jérémie Dubois-Lacoste 09-26-2012 06:21 PM

High memory needs
 
You may have misunderstood.
The detailed number I gave are obtained on distributions that
are not CentOS, and ok, it can makes sense between 32
and 64 bits.
But on CentOS 6.2, 64bits, I obtain:
SH: 103MB
PYTHON: 114MB
R: 200MB

This is from a freshly installed CentOS 6.2 machine, without
anything else. Thus the other components of our cluster are not
involved here. This machine has 16 cores.
This is MUCH more than other 64 bits distributions, as I wrote
the ratios are between 2 and 7.


2012/9/26 Michael Hennebry <hennebry@web.cs.ndsu.nodak.edu>:
> On Wed, 26 Sep 2012, m.roth@5-cent.us wrote:
>
>
>> Jérémie Dubois-Lacoste wrote:
>
>
>>> Python script:
>>> Avg Min Max
>>> 32 bits 8500 5004 11132
>>> 64 bits 32800 30000 36336
>>
>>
>> 8500 * 2 = 17000
>> 5004 * 2 = 10007
>> 11132 * 2 = 22264
>>
>> So that ranges from 2-2.5 larger.
>
>
> Huh?
> 3*8500=25500
> < 32800
> 3*5004=15012
> < 30000
> 3*11132=33396
> < 36336
>
>
>>> R:
>>> Avg Min Max
>>> 32 bits 26900 21000 33452
>>> 64 bits 100200 93008 97496
>
>
> 3*26900= 80700
> < 100200
> 4*21000=84000
> < 93008
> 2.9*33452< 97011
> < 97496
>
> --
> Michael hennebry@web.cs.ndsu.NoDak.edu
> "On Monday, I'm gonna have to tell my kindergarten class,
> whom I teach not to run with scissors,
> that my fiance ran me through with a broadsword." -- Lily
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Gordon Messmer 09-27-2012 02:18 AM

High memory needs
 
On 09/26/2012 09:14 AM, Jérémie Dubois-Lacoste wrote:
> 1. Run a python script and check the memory that
> it requires (field "VIRT" of the "top" command).

Don't use VIRT as a reference for memory used. RES is a better
indication, but even that won't tell you anything useful about shared
memory, and will lead you to believe that a process is using more memory
than it is.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Jérémie Dubois-Lacoste 09-27-2012 08:57 AM

High memory needs
 
We have a computing cluster running Sun Grid Engine, which
considers this value to check if a process exceeds the memory
limit or not. So somehow I'm bound to consider it.

I installed a machine from scratch with CentOS 6.2 x64, nothing
else, I open a terminal, I run this simple bash script and VIRT
goes beyond 100MB for it.
I understand it may not be very precise, however I still don't
understant the difference compared to other x64 ditributions,
under CentOS the value is 7 times higher!




2012/9/27 Gordon Messmer <yinyang@eburg.com>:
> On 09/26/2012 09:14 AM, Jérémie Dubois-Lacoste wrote:
>> 1. Run a python script and check the memory that
>> it requires (field "VIRT" of the "top" command).
>
> Don't use VIRT as a reference for memory used. RES is a better
> indication, but even that won't tell you anything useful about shared
> memory, and will lead you to believe that a process is using more memory
> than it is.
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Gordon Messmer 09-27-2012 03:46 PM

High memory needs
 
On 09/27/2012 01:57 AM, Jérémie Dubois-Lacoste wrote:
> We have a computing cluster running Sun Grid Engine, which
> considers this value to check if a process exceeds the memory
> limit or not. So somehow I'm bound to consider it.
>
> I installed a machine from scratch with CentOS 6.2 x64, nothing
> else, I open a terminal, I run this simple bash script and VIRT
> goes beyond 100MB for it.
> I understand it may not be very precise, however I still don't
> understant the difference compared to other x64 ditributions,
> under CentOS the value is 7 times higher!

OK, so compare the memory map of the process on the two systems. For
example, I've included the output of 'pmap $$' on two CentOS 6 systems.
One is 32bit, the other is 64. VIRT on the 32 bit system is ~5MB. On
the 64 bit system, VIRT is ~116MB.

Why? Well, most of the VIRT size is a mmapped file,
/usr/lib/locale/locale-archive. That file isn't resident in memory for
the most part, but because the process addresses the contents of that
file as if they were memory, rather than as if the content were a file,
it's accounted as part of the VIRT size.

My login shell on the 64 bit host is only using 3.5MB of RAM total
(including libraries shared with other processes), but its VIRT size is
~116MB.

If your vendor is using VIRT to make decisions about available
resources, they're doing their job profoundly wrong. RES would be
better, but still would represent far more memory use than is accurate,
especially on systems with many active processes.


64 bit system:
[gordon@vagabond:~]$ pmap $$
23966: bash
0000000000400000 872K r-x-- /usr/bin/bash
00000000006d9000 4K r---- /usr/bin/bash
00000000006da000 36K rw--- /usr/bin/bash
00000000006e3000 24K rw--- [ anon ]
00000000008e2000 36K rw--- /usr/bin/bash
000000000122d000 1836K rw--- [ anon ]
000000383c000000 128K r-x-- /usr/lib64/ld-2.15.so
000000383c21f000 4K r---- /usr/lib64/ld-2.15.so
000000383c220000 4K rw--- /usr/lib64/ld-2.15.so
000000383c221000 4K rw--- [ anon ]
000000383c400000 1712K r-x-- /usr/lib64/libc-2.15.so
000000383c5ac000 2048K ----- /usr/lib64/libc-2.15.so
000000383c7ac000 16K r---- /usr/lib64/libc-2.15.so
000000383c7b0000 8K rw--- /usr/lib64/libc-2.15.so
000000383c7b2000 20K rw--- [ anon ]
000000383cc00000 12K r-x-- /usr/lib64/libdl-2.15.so
000000383cc03000 2044K ----- /usr/lib64/libdl-2.15.so
000000383ce02000 4K r---- /usr/lib64/libdl-2.15.so
000000383ce03000 4K rw--- /usr/lib64/libdl-2.15.so
000000384d000000 148K r-x-- /usr/lib64/libtinfo.so.5.9
000000384d025000 2044K ----- /usr/lib64/libtinfo.so.5.9
000000384d224000 16K r---- /usr/lib64/libtinfo.so.5.9
000000384d228000 4K rw--- /usr/lib64/libtinfo.so.5.9
00007f8ae82aa000 48K r-x-- /usr/lib64/libnss_files-2.15.so
00007f8ae82b6000 2044K ----- /usr/lib64/libnss_files-2.15.so
00007f8ae84b5000 4K r---- /usr/lib64/libnss_files-2.15.so
00007f8ae84b6000 4K rw--- /usr/lib64/libnss_files-2.15.so
00007f8ae84b7000 102580K r---- /usr/lib/locale/locale-archive
00007f8aee8e4000 16K rw--- [ anon ]
00007f8aee8f5000 8K rw--- [ anon ]
00007f8aee8f7000 28K r--s- /usr/lib64/gconv/gconv-modules.cache
00007f8aee8fe000 4K rw--- [ anon ]
00007fff5010a000 132K rw--- [ stack ]
00007fff501ff000 4K r-x-- [ anon ]
ffffffffff600000 4K r-x-- [ anon ]
total 115904K


32 bit:
enetics@firewall:~$ pmap $$
32629: -bash
00291000 4K r-x-- [ anon ]
00952000 120K r-x-- /lib/ld-2.12.so
00970000 4K r---- /lib/ld-2.12.so
00971000 4K rw--- /lib/ld-2.12.so
00978000 1584K r-x-- /lib/libc-2.12.so
00b04000 8K r---- /lib/libc-2.12.so
00b06000 4K rw--- /lib/libc-2.12.so
00b07000 12K rw--- [ anon ]
00b0c000 12K r-x-- /lib/libdl-2.12.so
00b0f000 4K r---- /lib/libdl-2.12.so
00b10000 4K rw--- /lib/libdl-2.12.so
00b39000 48K r-x-- /lib/libnss_files-2.12.so
00b45000 4K r---- /lib/libnss_files-2.12.so
00b46000 4K rw--- /lib/libnss_files-2.12.so
00b7d000 88K r-x-- /lib/libtinfo.so.5.7
00b93000 12K rw--- /lib/libtinfo.so.5.7
08047000 836K r-x-- /bin/bash
08118000 20K rw--- /bin/bash
0811d000 20K rw--- [ anon ]
08b43000 232K rw--- [ anon ]
b7682000 28K r--s- /usr/lib/gconv/gconv-modules.cache
b7689000 2048K r---- /usr/lib/locale/locale-archive
b7889000 8K rw--- [ anon ]
b788f000 12K rw--- [ anon ]
bff3f000 84K rw--- [ stack ]
total 5204K


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Les Mikesell 09-27-2012 04:34 PM

High memory needs
 
On Thu, Sep 27, 2012 at 10:46 AM, Gordon Messmer <yinyang@eburg.com> wrote:
>
>> I understand it may not be very precise, however I still don't
>> understant the difference compared to other x64 ditributions,
>> under CentOS the value is 7 times higher!

This might explain it:
https://bugzilla.redhat.com/show_bug.cgi?id=156477
The mmapped local-archive contains all languages even though only the
ones you use are accessed from it. Other distros split them and the
installers only install what you want.

> 64 bit system:
> 00007f8ae84b7000 102580K r---- /usr/lib/locale/locale-archive
>
> 32 bit:
> b7689000 2048K r---- /usr/lib/locale/locale-archive

That's an interesting difference on its own, since the underlying
files are about 95M and 54M respectively. Does the 32 bit kernel use
some tricks to sparsely map files where the 64 bit one does it
directly with page tables?

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


All times are GMT. The time now is 09:19 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.