FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 08-22-2012, 09:42 PM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>>
>> This should not happen, especially on such a small archive. I've tried
>> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
>> memory allocations). They never were larger than 68 MB
>>
>> Try it yourself. The second parameter in mmap is the allocated size in byte.
>>
>>
> In the box where it works:

[snip]
> In the other box, in the gentoo chroot:
>
> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz

[snip]

> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)

It's only asking for about 65MB of RAM there. ENOMEM is nearly a
catch-all failure code for mmap().

My bet is that it's an incompatibility between the Arch kernel and the
version of glibc in the chroot.

--
:wq
 
Old 08-22-2012, 09:42 PM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 5:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>>>
>>> This should not happen, especially on such a small archive. I've tried
>>> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
>>> memory allocations). They never were larger than 68 MB
>>>
>>> Try it yourself. The second parameter in mmap is the allocated size in byte.
>>>
>>>
>> In the box where it works:
>
> [snip]
>> In the other box, in the gentoo chroot:
>>
>> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
>
> [snip]
>
>> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
>> -1, 0) = -1 ENOMEM (Cannot allocate memory)
>
> It's only asking for about 65MB of RAM there. ENOMEM is nearly a
> catch-all failure code for mmap().
>
> My bet is that it's an incompatibility between the Arch kernel and the
> version of glibc in the chroot.

(incidentally...hey! It's Gentoo-related after all. )

--
:wq
 
Old 08-22-2012, 10:19 PM
Jorge Almeida
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>
> My bet is that it's an incompatibility between the Arch kernel and the
> version of glibc in the chroot.
>
Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
is 2.14. Maybe I should update system et al. But then emerge system
asks for m4 (but not for glibc, which I find surprising).

Thanks,

J.
 
Old 08-23-2012, 12:24 AM
Jorge Almeida
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>
>> My bet is that it's an incompatibility between the Arch kernel and the
>> version of glibc in the chroot.
>>
> Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
> is 2.14. Maybe I should update system et al. But then emerge system
> asks for m4 (but not for glibc, which I find surprising).
>
Well, I found the problem: ulimit problem. Not the first time this crap bites
me, but I always forget. I just wish this was better documented, somewhere.

Sorry for the noise, and thanks to everyone.

Jorge Almeida
 
Old 08-23-2012, 12:56 AM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 8:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>>
>>> My bet is that it's an incompatibility between the Arch kernel and the
>>> version of glibc in the chroot.
>>>
>> Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
>> is 2.14. Maybe I should update system et al. But then emerge system
>> asks for m4 (but not for glibc, which I find surprising).
>>
> Well, I found the problem: ulimit problem. Not the first time this crap bites
> me, but I always forget. I just wish this was better documented, somewhere.
>
> Sorry for the noise, and thanks to everyone.

Hey, I learned something today. No complaints here.


--
:wq
 
Old 08-23-2012, 01:43 AM
Nikos Chantziaras
 
Default xz memory hungry?

On 22/08/12 21:52, Jorge Almeida wrote:

# tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
xz: (stdin): Cannot allocate memory
tar: Child returned status 1
tar: Error is not recoverable: exiting now

The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
Emerging fails on m4. tar xJvf fails both from within the chroot and from the
host system. top shows that nothing is using any amount of memory worth
mentioning. Extracting libtool-2.4.tar.xz works. I can extract
m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
related, except that I have no choice, as m4 is pulled by other packages.
What to do?

app-arch/xz-utils-5.0.3 in chroot
xz 5.0.4 in host system (Archlinux)


You should file a bug about this. Whoever put that xz there surely has
no idea that this is happening.
 
Old 08-23-2012, 03:34 AM
Paul Hartman
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> Well, I found the problem: ulimit problem. Not the first time this crap bites
> me, but I always forget. I just wish this was better documented, somewhere.

I tried to use ulimit to change stack size system-wide once, to reduce
RAM usage on 256M box, and it resulted in strange problems like this.
I changed it back to default and leave it alone since then except for
specific services because I don't fully understand the magic that
happens inside the box.
 
Old 08-23-2012, 08:09 AM
Jorge Almeida
 
Default xz memory hungry?

On Thu, Aug 23, 2012 at 2:43 AM, Nikos Chantziaras <realnc@gmail.com> wrote:
> On 22/08/12 21:52, Jorge Almeida wrote:
>>
>
>
> You should file a bug about this. Whoever put that xz there surely has no
> idea that this is happening.
>
>
Not Gentoo nor xz fault (see other messages in thread).

Thanks

J.A.
 
Old 08-23-2012, 08:22 AM
Volker Armin Hemmann
 
Default xz memory hungry?

Am Mittwoch, 22. August 2012, 22:16:19 schrieb Jorge Almeida:
> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net>
wrote:
> > Am 22.08.2012 20:52, schrieb Jorge Almeida:
> >
> > This should not happen, especially on such a small archive. I've tried
> > `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
> > memory allocations). They never were larger than 68 MB
> >
> > Try it yourself. The second parameter in mmap is the allocated size in
> > byte.
> In the box where it works:
>
> $ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7746000
> mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000
> mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb76fe000
> mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000
> mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb76e3000
> mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000
> mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000
> mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
> 3, 0) = 0xb753d000
> mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000
> mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb753c000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb753b000
> mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7745000
> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = 0xb723a000
> +++ exited with 0 +++
>
>
> In the other box, in the gentoo chroot:
>
> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779e000
> mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000
> mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb7777000
> mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000
> mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
> 3, 0) = 0xb7615000
> mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000
> mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7614000
> mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000
> mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779c000
> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 2097152, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000
> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779c000
> xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory
> +++ exited with 1 +++
>
>
> Thanks,
>
> Jorge Almeida


maybe a memory fragmentation problem? It tries to allocate 65mb continous
chunk - and can't find one?
--
#163933
 
Old 08-23-2012, 08:37 AM
Jorge Almeida
 
Default xz memory hungry?

On Thu, Aug 23, 2012 at 4:34 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> Well, I found the problem: ulimit problem. Not the first time this crap bites
>> me, but I always forget. I just wish this was better documented, somewhere.
>
> I tried to use ulimit to change stack size system-wide once, to reduce
> RAM usage on 256M box, and it resulted in strange problems like this.
> I changed it back to default and leave it alone since then except for
> specific services because I don't fully understand the magic that
> happens inside the box.
>
Last time I had a problem like this I spent a lot of time googling about
ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I
run "ulimit -v unlimited", but the question is who put the former values
there? Some hard-coded default? I couldn't find anything in init scripts nor
in bash rc files. I know that on logout the value is lost (I had to run ulimit
again on chrooting). What is the appropriate file to put "ulimit -v unlimited"
in? Perhaps ~/.bash_profile? And how can root set different hard limits for
different users? Maybe some bash guru will step in?

J.A.
 

Thread Tools




All times are GMT. The time now is 11:07 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org