FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu User

 
 
LinkBack Thread Tools
 
Old 08-07-2008, 01:39 PM
"MirJafar Aki"
 
Default Can some throw bright light on Linux Memory Allocation

Hello,

I have installed Ubuntu on a machine with following
config.

RAM :* 2 GB
Machine:* Intel Core 2 Duo
OS:* Ubuntu 64 bit.
Disk:* 250 GB.


I have attached a simple program to test memory allocation on linux and something strange is happening.


1.*** If I invoke large_memory function then maximum
* * ** size I am allowed is 245GB.
*
2.*** If I use less than 245GB* ( say 244 GB) then the
* * ** second stage is also passed.* Where from OS is
* * ** allocating 244GB ?

*
3. * Strange enough, if I use chuck memory, I can
* * * goto 3817GB before I get segmentation fault.
* * * ( and buf == NULL is never reached ).

4. * Much more strange, GoogleHeapProfiler, and
* * * ValGrindHeap profile also add to my uneasiness

* * * and report that 3817GB allocation is successful.

What is happening ? Please help.

Mir
* *

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 02:15 PM
Chris Jones
 
Default Can some throw bright light on Linux Memory Allocation

MirJafar Aki wrote:
> Hello,
>
> I have installed Ubuntu on a machine with following
> config.
>
> RAM : 2 GB
> Machine: Intel Core 2 Duo
> OS: Ubuntu 64 bit.
> Disk: 250 GB.
>
>
> I have attached a simple program to test memory allocation on linux and
> something strange is happening.
>
> 1. If I invoke large_memory function then maximum
> size I am allowed is 245GB.
>
> 2. If I use less than 245GB ( say 244 GB) then the
> second stage is also passed. Where from OS is
> allocating 244GB ?

Maybe I'm missing the point here, but why on Earth would you want to try
and allocate 250G of memory on a machine with 'only' 2G ram. Is your
250G disk entirely devoted to swap space ??

Chris

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 02:32 PM
Rashkae
 
Default Can some throw bright light on Linux Memory Allocation

MirJafar Aki wrote:
> Hello,
>
> I have installed Ubuntu on a machine with following
> config.
>
> RAM : 2 GB
> Machine: Intel Core 2 Duo
> OS: Ubuntu 64 bit.
> Disk: 250 GB.
>
>
> I have attached a simple program to test memory allocation on linux and
> something strange is happening.
>
> 1. If I invoke large_memory function then maximum
> size I am allowed is 245GB.
>
> 2. If I use less than 245GB ( say 244 GB) then the
> second stage is also passed. Where from OS is
> allocating 244GB ?
>
> 3. Strange enough, if I use chuck memory, I can
> goto 3817GB before I get segmentation fault.
> ( and buf == NULL is never reached ).
>
> 4. Much more strange, GoogleHeapProfiler, and
> ValGrindHeap profile also add to my uneasiness
> and report that 3817GB allocation is successful.
>
> What is happening ? Please help.
>
> Mir
>
>

It's been a while since I read up on this and and my terminology will be
way off, (as well, some of my information will be out of date.)

The short of it, Linux doesn't allocate memory absolutely on malloc.
You can malloc more memory than you have available, but will fail when
you application tries to fill this memory and the system runs out. This
is a deliberate design decision done for performance and or other
esoteric reasons beyond my ken. I also had some sample code of an app
that would malloc large memory, then try to fill that memory and the app
itself would in theory end gracefully if the memory proved unavailable.
In practice, bad things tend to happen to the whole system once you hit
OOM.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 02:54 PM
Rashkae
 
Default Can some throw bright light on Linux Memory Allocation

Rashkae wrote:

>>
>
> It's been a while since I read up on this and and my terminology will be
> way off, (as well, some of my information will be out of date.)
>
> The short of it, Linux doesn't allocate memory absolutely on malloc.
> You can malloc more memory than you have available, but will fail when
> you application tries to fill this memory and the system runs out. This
> is a deliberate design decision done for performance and or other
> esoteric reasons beyond my ken. I also had some sample code of an app
> that would malloc large memory, then try to fill that memory and the app
> itself would in theory end gracefully if the memory proved unavailable.
> In practice, bad things tend to happen to the whole system once you hit
> OOM.
>

BTW, first google hit for Linux Memory Allocation
http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 03:15 PM
"MirJafar Aki"
 
Default Can some throw bright light on Linux Memory Allocation

On Thu, Aug 7, 2008 at 7:45 PM, Chris Jones <jonesc@hep.phy.cam.ac.uk> wrote:

MirJafar Aki wrote:

> Hello,

>

> I have installed Ubuntu on a machine with following

> config.

>

> RAM : *2 GB

> Machine: *Intel Core 2 Duo

> OS: *Ubuntu 64 bit.

> Disk: *250 GB.

>

>

> I have attached a simple program to test memory allocation on linux and

> something strange is happening.

>

> 1. * *If I invoke large_memory function then maximum

> * * * *size I am allowed is 245GB.

>

> 2. * *If I use less than 245GB *( say 244 GB) then the

> * * * *second stage is also passed. *Where from OS is

> * * * *allocating 244GB ?



Maybe I'm missing the point here, but why on Earth would you want to try

and allocate 250G of memory on a machine with 'only' 2G ram. Is your

250G disk entirely devoted to swap space ??



Chris



--

ubuntu-users mailing list

ubuntu-users@lists.ubuntu.com

Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users


Hello,

On this simple earth, I came across some problems and therefore wanted to test the memory allocation and I observed
this behavior. I thought, I must ask some expert like you.


Thanks.
Mir

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 03:22 PM
"MirJafar Aki"
 
Default Can some throw bright light on Linux Memory Allocation

Hi Rashkae,

Thanks for great explanation and the link. It was really useful
to understand the aggessive optimization done by OS.

Thanks.
Mir

On Thu, Aug 7, 2008 at 8:24 PM, Rashkae <ubuntu@tigershaunt.com> wrote:

Rashkae wrote:



>>

>

> It's been a while since I read up on this and and my terminology will be

> way off, (as well, some of my information will be out of date.)

>

> The short of it, Linux doesn't allocate memory absolutely on malloc.

> You can malloc more memory than you have available, but will fail when

> you application tries to fill this memory and the system runs out. *This

> is a deliberate design decision done for performance and or other

> esoteric reasons beyond my ken. *I also had some sample code of an app

> that would malloc large memory, then try to fill that memory and the app

> itself would in theory end gracefully if the memory proved unavailable.

> *In practice, bad things tend to happen to the whole system once you hit

> OOM.

>



BTW, first google hit for Linux Memory Allocation

http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html



--

ubuntu-users mailing list

ubuntu-users@lists.ubuntu.com

Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users



--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 06:34 PM
Sergey Skorokhodov
 
Default Can some throw bright light on Linux Memory Allocation

Hi,

> Hello,
>
> I have installed Ubuntu on a machine with following
> config.
>
> RAM : 2 GB
> Machine: Intel Core 2 Duo
> OS: Ubuntu 64 bit.
> Disk: 250 GB.
>
>
> I have attached a simple program to test memory allocation on linux
> and something strange is happening.
>
> 1. If I invoke large_memory function then maximum
> size I am allowed is 245GB.
>
> 2. If I use less than 245GB ( say 244 GB) then the
> second stage is also passed. Where from OS is
> allocating 244GB ?
>
> 3. Strange enough, if I use chuck memory, I can
> goto 3817GB before I get segmentation fault.
> ( and buf == NULL is never reached ).
>
> 4. Much more strange, GoogleHeapProfiler, and
> ValGrindHeap profile also add to my uneasiness
> and report that 3817GB allocation is successful.
>
> What is happening ? Please help.
>
I guess that the result of

size_t nb = 1024*1024*1024;
nb *= 255; // 0x3FC0000000

is much larger than the maximal value of size_t on 32-bit systems. So it
is truncated silently to 4 GB. Try

size_t nb = 1024*1024*1024*255;

and you get a warning. C/C++ consider that the programmer knows what
he/she is doing in the former case and keep silent but you can turn some
more rigorous checking on if you like.

Actually, a program is allocated with address space some less that 4 GB
on 32-bit system as some address space is used for memory mapped io and
other hardware needs.

--
Best regards,
Serge


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 07:22 PM
"MirJafar Ali"
 
Default Can some throw bright light on Linux Memory Allocation

On Fri, Aug 8, 2008 at 12:04 AM, Sergey Skorokhodov <ucayalyfish@yandex.ru> wrote:

Hi,



> Hello,

>

> I have installed Ubuntu on a machine with following

> config.

>

> RAM : *2 GB

> Machine: *Intel Core 2 Duo

> OS: *Ubuntu 64 bit.

> Disk: *250 GB.

>

>

> I have attached a simple program to test memory allocation on linux

> and something strange is happening.

>

> 1. * *If I invoke large_memory function then maximum

> * * * *size I am allowed is 245GB.

>

> 2. * *If I use less than 245GB *( say 244 GB) then the

> * * * *second stage is also passed. *Where from OS is

> * * * *allocating 244GB ?

>

> 3. * Strange enough, if I use chuck memory, I can

> * * * goto 3817GB before I get segmentation fault.

> * * * ( and buf == NULL is never reached ).

>

> 4. * Much more strange, GoogleHeapProfiler, and

> * * * ValGrindHeap profile also add to my uneasiness

> * * * and report that 3817GB allocation is successful.

>

> What is happening ? Please help.

>

I guess that the result of



size_t nb * * * = * * * 1024*1024*1024;

nb * * * * * * **= * * *255; * *// 0x3FC0000000



is much larger than the maximal value of size_t on 32-bit systems. So it

is truncated silently to 4 GB. Try



size_t nb = 1024*1024*1024*255;



and you get a warning. C/C++ consider that the programmer knows what

he/she is doing in the former case and keep silent but you can turn some

more rigorous checking on if you like.



Actually, a program is allocated with address space some less that 4 GB

on 32-bit system as some address space is used for memory mapped io and

other hardware needs.



--

Best regards,

Serge





--

ubuntu-users mailing list

ubuntu-users@lists.ubuntu.com

Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users


Hello,

In the very first mail, I had mentioned that I have installed 64 OS, so I dont think that this was the issue. But I received good
explanation of this behavior in the previous mail.*


Thanks

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-07-2008, 11:31 PM
Mark Kirkwood
 
Default Can some throw bright light on Linux Memory Allocation

You might also want to read up on controlling overcommit behaviour via
the sysctls 'vm.overcommit_memory' and 'vm.overcommit_ratio'.

These are not terribly well documented, but a quick search got me this link:

http://www.win.tue.nl/~aeb/linux/lk/lk-9.html

which seemed quite good.

regards

Mark


MirJafar Aki wrote:
> Hi Rashkae,
>
> Thanks for great explanation and the link. It was really useful
> to understand the aggessive optimization done by OS.
>
> Thanks.
> Mir
>
> On Thu, Aug 7, 2008 at 8:24 PM, Rashkae <ubuntu@tigershaunt.com
> <mailto:ubuntu@tigershaunt.com>> wrote:
>
> Rashkae wrote:
>
> >>
> >
> > It's been a while since I read up on this and and my terminology
> will be
> > way off, (as well, some of my information will be out of date.)
> >
> > The short of it, Linux doesn't allocate memory absolutely on malloc.
> > You can malloc more memory than you have available, but will
> fail when
> > you application tries to fill this memory and the system runs
> out. This
> > is a deliberate design decision done for performance and or other
> > esoteric reasons beyond my ken. I also had some sample code of
> an app
> > that would malloc large memory, then try to fill that memory and
> the app
> > itself would in theory end gracefully if the memory proved
> unavailable.
> > In practice, bad things tend to happen to the whole system once
> you hit
> > OOM.
> >
>
> BTW, first google hit for Linux Memory Allocation
> http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
>
> --
> ubuntu-users mailing list
> ubuntu-users@lists.ubuntu.com <mailto:ubuntu-users@lists.ubuntu.com>
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>
>


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 08-08-2008, 06:44 AM
Sergey Skorokhodov
 
Default Can some throw bright light on Linux Memory Allocation

Hi,

> > I have installed Ubuntu on a machine with following
> > config.
> >
> > RAM : 2 GB
> > Machine: Intel Core 2 Duo
> > OS: Ubuntu 64 bit.
> > Disk: 250 GB.
> >
> >
> > I have attached a simple program to test memory allocation
> on linux and something strange is happening.

<skip>

> In the very first mail, I had mentioned that I have installed 64 OS,
> so I dont think that this was the issue. But I received good
> explanation of this behavior in the previous mail.
>

Oh, sorry about that, I've missed this point. Anyway, you've got quite
legal 4 GB address space and the system failed to to fill it with
virtual memory later on. Quite logical, eh? Performance etc. OSes are
all like that...

--
Regards,
Serge


--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 

Thread Tools




All times are GMT. The time now is 07:40 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org