FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Gentoo > Gentoo User

 
 
LinkBack Thread Tools
 
Old 08-22-2012, 06:52 PM
Jorge Almeida
 
Default xz memory hungry?

# tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
xz: (stdin): Cannot allocate memory
tar: Child returned status 1
tar: Error is not recoverable: exiting now

The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
Emerging fails on m4. tar xJvf fails both from within the chroot and from the
host system. top shows that nothing is using any amount of memory worth
mentioning. Extracting libtool-2.4.tar.xz works. I can extract
m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
related, except that I have no choice, as m4 is pulled by other packages.
What to do?

app-arch/xz-utils-5.0.3 in chroot
xz 5.0.4 in host system (Archlinux)

TIA

Jorge Almeida
 
Old 08-22-2012, 07:05 PM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)

How much do you have free? From xz's manpage:

Memory usage
The memory usage of xz varies from a few hundred kilobytes
to several gigabytes
depending on the compression settings. The settings used
when compressing a file
determine the memory requirements of the decompressor.
Typically the decompressor
needs 5 % to 20 % of the amount of memory that the
compressor needed when creating
the file. For example, decompressing a file created with xz
-9 currently requires
65 MiB of memory. Still, it is possible to have .xz files that
require several giga‐
bytes of memory to decompress.


Three things come to mind:

1) You may not have enough memory free
2) There may be a bug (either compile/link-induced or code-induced) in
the copy of xz you're using
3) Upstream used some insane settings, causing a massive increase in
the amount of RAM required to decompress that stream.


You could download the .tar.xz file, decompress it on a different box,
and then recompress it with lighter settings.

unxz filename.tar.xz
xz -1 filename.tar

--
:wq
 
Old 08-22-2012, 08:12 PM
Neil Bothwick
 
Default xz memory hungry?

On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:

> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
>
> unxz filename.tar.xz
> xz -1 filename.tar

You'd have to rebuild the ebuild's manifest after doing that, or portage
will download the original again.


--
Neil Bothwick

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9
 
Old 08-22-2012, 08:24 PM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 4:12 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:
>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>
> You'd have to rebuild the ebuild's manifest after doing that, or portage
> will download the original again.

Fair point.


--
:wq
 
Old 08-22-2012, 08:32 PM
Jorge Almeida
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>> xz: (stdin): Cannot allocate memory
>>
>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>
> How much do you have free? From xz's manpage:

Almost all of it! It's a one-user workstation, which was essentialy idle.
>
>
I read the man page of xz, but it suggested nothing to me.
>
> Three things come to mind:
>
> 1) You may not have enough memory free
> 2) There may be a bug (either compile/link-induced or code-induced) in
> the copy of xz you're using
> 3) Upstream used some insane settings, causing a massive increase in
> the amount of RAM required to decompress that stream.
>
>
> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
>
> unxz filename.tar.xz
> xz -1 filename.tar
>
Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
really the appropriate word. Of course, the hash digests are now wrong, so
emerge still fails. Any idea how to find which amount of memory is needed? I
would setup appropriate swap, if possible. The LFS site
http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html
shows that there exists a tar.bz2 tarball. I think the ebuild should pull
that... Can't believe there are no gentooers out there with boxes with less ram.

Thanks a lot

Jorge Almeida
 
Old 08-22-2012, 08:39 PM
Florian Philipp
 
Default xz memory hungry?

Am 22.08.2012 20:52, schrieb Jorge Almeida:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)
>
> TIA
>
> Jorge Almeida
>

This should not happen, especially on such a small archive. I've tried
`strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
memory allocations). They never were larger than 68 MB

Try it yourself. The second parameter in mmap is the allocated size in byte.

Regards,
Florian Philipp
 
Old 08-22-2012, 08:43 PM
Florian Philipp
 
Default xz memory hungry?

Am 22.08.2012 22:32, schrieb Jorge Almeida:
> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>> xz: (stdin): Cannot allocate memory
>>>
>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>
>> How much do you have free? From xz's manpage:
>
> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>
>>
> I read the man page of xz, but it suggested nothing to me.
>>
>> Three things come to mind:
>>
>> 1) You may not have enough memory free
>> 2) There may be a bug (either compile/link-induced or code-induced) in
>> the copy of xz you're using
>> 3) Upstream used some insane settings, causing a massive increase in
>> the amount of RAM required to decompress that stream.
>>
>>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>>
> Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
> really the appropriate word. Of course, the hash digests are now wrong, so
> emerge still fails. Any idea how to find which amount of memory is needed? I
> would setup appropriate swap, if possible. [...]

There is a table in `man xz` showing the memory requirements. Even with
the highest setting, you only need 65 MB memory for decompression (674
MB for compression, though).

Regards,
Florian Philipp
 
Old 08-22-2012, 09:10 PM
Neil Bothwick
 
Default xz memory hungry?

On Wed, 22 Aug 2012 21:32:56 +0100, Jorge Almeida wrote:

> Done that. It extracts now, so 3) is the correct hypothesis, and
> "insane" is really the appropriate word. Of course, the hash digests
> are now wrong, so emerge still fails.

sudo ebuild $PORTDIR/cat/pkg/pkg-ver.ebuild manifest


--
Neil Bothwick

The computer revolution is over. The computers won.
 
Old 08-22-2012, 09:12 PM
Michael Mol
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 4:43 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 22:32, schrieb Jorge Almeida:
>> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>>> xz: (stdin): Cannot allocate memory
>>>>
>>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>>
>>> How much do you have free? From xz's manpage:
>>
>> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>>
>>>
>> I read the man page of xz, but it suggested nothing to me.
>>>
>>> Three things come to mind:
>>>
>>> 1) You may not have enough memory free
>>> 2) There may be a bug (either compile/link-induced or code-induced) in
>>> the copy of xz you're using
>>> 3) Upstream used some insane settings, causing a massive increase in
>>> the amount of RAM required to decompress that stream.
>>>
>>>
>>> You could download the .tar.xz file, decompress it on a different box,
>>> and then recompress it with lighter settings.
>>>
>>> unxz filename.tar.xz
>>> xz -1 filename.tar
>>>
>> Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
>> really the appropriate word. Of course, the hash digests are now wrong, so
>> emerge still fails. Any idea how to find which amount of memory is needed? I
>> would setup appropriate swap, if possible. [...]
>
> There is a table in `man xz` showing the memory requirements. Even with
> the highest setting, you only need 65 MB memory for decompression (674
> MB for compression, though).

Still not sure about this portion: "Still, it is possible to have .xz files that
require several giga‐bytes of memory to decompress."

But, yeah, this seems very wonky.

--
:wq
 
Old 08-22-2012, 09:16 PM
Jorge Almeida
 
Default xz memory hungry?

On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>
> This should not happen, especially on such a small archive. I've tried
> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
> memory allocations). They never were larger than 68 MB
>
> Try it yourself. The second parameter in mmap is the allocated size in byte.
>
>
In the box where it works:

$ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7746000
mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000
mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb76fe000
mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000
mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb76e3000
mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000
mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000
mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xb753d000
mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000
mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb753c000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb753b000
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7745000
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb723a000
+++ exited with 0 +++


In the other box, in the gentoo chroot:

# strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779e000
mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000
mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb7777000
mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000
mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xb7615000
mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000
mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7614000
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000
mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779c000
mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000
mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779c000
xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory
+++ exited with 1 +++


Thanks,

Jorge Almeida
 

Thread Tools




All times are GMT. The time now is 09:58 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org