# tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
xz: (stdin): Cannot allocate memory
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
Emerging fails on m4. tar xJvf fails both from within the chroot and from the
host system. top shows that nothing is using any amount of memory worth
mentioning. Extracting libtool-2.4.tar.xz works. I can extract
m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
related, except that I have no choice, as m4 is pulled by other packages.
What to do?
app-arch/xz-utils-5.0.3 in chroot
xz 5.0.4 in host system (Archlinux)
TIA
Jorge Almeida
08-22-2012, 07:05 PM
Michael Mol
xz memory hungry?
On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)
How much do you have free? From xz's manpage:
Memory usage
The memory usage of xz varies from a few hundred kilobytes
to several gigabytes
depending on the compression settings. The settings used
when compressing a file
determine the memory requirements of the decompressor.
Typically the decompressor
needs 5 % to 20 % of the amount of memory that the
compressor needed when creating
the file. For example, decompressing a file created with xz
-9 currently requires
65 MiB of memory. Still, it is possible to have .xz files that
require several giga‐
bytes of memory to decompress.
Three things come to mind:
1) You may not have enough memory free
2) There may be a bug (either compile/link-induced or code-induced) in
the copy of xz you're using
3) Upstream used some insane settings, causing a massive increase in
the amount of RAM required to decompress that stream.
You could download the .tar.xz file, decompress it on a different box,
and then recompress it with lighter settings.
unxz filename.tar.xz
xz -1 filename.tar
--
:wq
08-22-2012, 08:12 PM
Neil Bothwick
xz memory hungry?
On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:
> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
>
> unxz filename.tar.xz
> xz -1 filename.tar
You'd have to rebuild the ebuild's manifest after doing that, or portage
will download the original again.
--
Neil Bothwick
"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9
08-22-2012, 08:24 PM
Michael Mol
xz memory hungry?
On Wed, Aug 22, 2012 at 4:12 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:
>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>
> You'd have to rebuild the ebuild's manifest after doing that, or portage
> will download the original again.
Fair point.
--
:wq
08-22-2012, 08:32 PM
Jorge Almeida
xz memory hungry?
On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>> xz: (stdin): Cannot allocate memory
>>
>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>
> How much do you have free? From xz's manpage:
Almost all of it! It's a one-user workstation, which was essentialy idle.
>
>
I read the man page of xz, but it suggested nothing to me.
>
> Three things come to mind:
>
> 1) You may not have enough memory free
> 2) There may be a bug (either compile/link-induced or code-induced) in
> the copy of xz you're using
> 3) Upstream used some insane settings, causing a massive increase in
> the amount of RAM required to decompress that stream.
>
>
> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
>
> unxz filename.tar.xz
> xz -1 filename.tar
>
Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
really the appropriate word. Of course, the hash digests are now wrong, so
emerge still fails. Any idea how to find which amount of memory is needed? I
would setup appropriate swap, if possible. The LFS site
http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html
shows that there exists a tar.bz2 tarball. I think the ebuild should pull
that... Can't believe there are no gentooers out there with boxes with less ram.
Thanks a lot
Jorge Almeida
08-22-2012, 08:39 PM
Florian Philipp
xz memory hungry?
Am 22.08.2012 20:52, schrieb Jorge Almeida:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)
>
> TIA
>
> Jorge Almeida
>
This should not happen, especially on such a small archive. I've tried
`strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
memory allocations). They never were larger than 68 MB
Try it yourself. The second parameter in mmap is the allocated size in byte.
Regards,
Florian Philipp
08-22-2012, 08:43 PM
Florian Philipp
xz memory hungry?
Am 22.08.2012 22:32, schrieb Jorge Almeida:
> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>> xz: (stdin): Cannot allocate memory
>>>
>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>
>> How much do you have free? From xz's manpage:
>
> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>
>>
> I read the man page of xz, but it suggested nothing to me.
>>
>> Three things come to mind:
>>
>> 1) You may not have enough memory free
>> 2) There may be a bug (either compile/link-induced or code-induced) in
>> the copy of xz you're using
>> 3) Upstream used some insane settings, causing a massive increase in
>> the amount of RAM required to decompress that stream.
>>
>>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>>
> Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
> really the appropriate word. Of course, the hash digests are now wrong, so
> emerge still fails. Any idea how to find which amount of memory is needed? I
> would setup appropriate swap, if possible. [...]
There is a table in `man xz` showing the memory requirements. Even with
the highest setting, you only need 65 MB memory for decompression (674
MB for compression, though).
Regards,
Florian Philipp
08-22-2012, 09:10 PM
Neil Bothwick
xz memory hungry?
On Wed, 22 Aug 2012 21:32:56 +0100, Jorge Almeida wrote:
> Done that. It extracts now, so 3) is the correct hypothesis, and
> "insane" is really the appropriate word. Of course, the hash digests
> are now wrong, so emerge still fails.
The computer revolution is over. The computers won.
08-22-2012, 09:12 PM
Michael Mol
xz memory hungry?
On Wed, Aug 22, 2012 at 4:43 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 22:32, schrieb Jorge Almeida:
>> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>>> xz: (stdin): Cannot allocate memory
>>>>
>>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>>
>>> How much do you have free? From xz's manpage:
>>
>> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>>
>>>
>> I read the man page of xz, but it suggested nothing to me.
>>>
>>> Three things come to mind:
>>>
>>> 1) You may not have enough memory free
>>> 2) There may be a bug (either compile/link-induced or code-induced) in
>>> the copy of xz you're using
>>> 3) Upstream used some insane settings, causing a massive increase in
>>> the amount of RAM required to decompress that stream.
>>>
>>>
>>> You could download the .tar.xz file, decompress it on a different box,
>>> and then recompress it with lighter settings.
>>>
>>> unxz filename.tar.xz
>>> xz -1 filename.tar
>>>
>> Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is
>> really the appropriate word. Of course, the hash digests are now wrong, so
>> emerge still fails. Any idea how to find which amount of memory is needed? I
>> would setup appropriate swap, if possible. [...]
>
> There is a table in `man xz` showing the memory requirements. Even with
> the highest setting, you only need 65 MB memory for decompression (674
> MB for compression, though).
Still not sure about this portion: "Still, it is possible to have .xz files that
require several giga‐bytes of memory to decompress."
But, yeah, this seems very wonky.
--
:wq
08-22-2012, 09:16 PM
Jorge Almeida
xz memory hungry?
On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>
> This should not happen, especially on such a small archive. I've tried
> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
> memory allocations). They never were larger than 68 MB
>
> Try it yourself. The second parameter in mmap is the allocated size in byte.
>
>
In the box where it works:
xz memory hungry?
