On Mon, 2008-09-22 at 15:29 +1200, Martin Langhoff wrote:
> On Mon, Sep 22, 2008 at 12:51 PM, Bruno Wolff III <bruno@wolff.to> wrote:
> > Doing atomic yum transactions is going to be hard.
>
> No doubt about that. RPM/Yum atomicity without FS snapshot support is
> impossible to achieve. But I am sure a lot of things can be improved.
>
> > Probably your effort would
> > be better spent keeping the system from crashing during an update then
> > trying to implement atomic updates.
>
> I cannot speak for Alexandre - but in my case that is just not
> possible. I will soon be dealing with thousands of servers running in
> rural schools in many countries. Very unreliable power, spotty
> internet connectivity. Once yum has started an operation, I hope it
> can work hard at completing it, even if that's across several reboots.
>

Rpm updates should behave as expected. When updating packages rpm
installs all the new packages first, then cleans up the old packages.
So, at no time is the system without at least 1 copy of the package. The
most common problem we've run into with yum is getting halfway through a
transaction and power or network or something fails such that the second
part, cleaning up the older packages, cannot occur.

this is where yum-complete-transaction comes in. It will finish up the
remaining steps since it has enumerated them.

Hope this helps, explain.

-sv


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, Sep 22, 2008 at 12:07:45AM -0400, Seth Vidal wrote:

So, at no time is the system without at least 1 copy of the package. The
most common problem we've run into with yum is getting halfway through a
transaction and power or network or something fails such that the second
part, cleaning up the older packages, cannot occur.

this is where yum-complete-transaction comes in. It will finish up the
remaining steps since it has enumerated them.


Seth,

Due to memory pressure and constraints on the XO, we regularly see some
RPM %pre- and %post-scripts fail due to OOM. Is there anything which
could reasonably be done to mitigate this problem for our yum-users?

Thanks,

Michael

(Background: we believe that yum is used on our systems primarily by
hackers and 'technically minded' G1G1 donors for customizing OLPC
software releases; to date, general software updates have been made
available through olpc-update, which is atomic, bandwidth-efficient, and
which permits 'emergency rollback' to the previous (known-good) OS
tree.)

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, 2008-09-22 at 00:31 -0400, Michael Stone wrote:
> On Mon, Sep 22, 2008 at 12:07:45AM -0400, Seth Vidal wrote:
> >So, at no time is the system without at least 1 copy of the package. The
> >most common problem we've run into with yum is getting halfway through a
> >transaction and power or network or something fails such that the second
> >part, cleaning up the older packages, cannot occur.
> >
> >this is where yum-complete-transaction comes in. It will finish up the
> >remaining steps since it has enumerated them.
>
> Seth,
>
> Due to memory pressure and constraints on the XO, we regularly see some
> RPM %pre- and %post-scripts fail due to OOM. Is there anything which
> could reasonably be done to mitigate this problem for our yum-users?

Not much - if it's OOM'ing the only real option is:
1. get more memory
2. have it use less memory

if you want to pursue 2 that's fine - but to be very clear - everything
after:
Running RPM Transaction Test

is in rpm-land as to trimming out memory/cpu usage. There's very little
yum can do there.

-sv


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal <skvidal@fedoraproject.org> wrote:
> Not much - if it's OOM'ing the only real option is:
> 1. get more memory
> 2. have it use less memory
>
> if you want to pursue 2 that's fine - but to be very clear - everything
> after:
> Running RPM Transaction Test
>
> is in rpm-land as to trimming out memory/cpu usage. There's very little
> yum can do there.

I *think* that what Michael is talking about is that when you get to
that point yum is still running and can be a substantial part of the
memory pressure.

Perhaps yum's memory footprint could be trimmed? If by then it's
loaded hard-to-shed things (like python libs) perhaps it can exec a
minimalistic "yum stage 2" that drives rpm?

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Martin,

I'm sure there are easier things to prune than yum itself. I was simply
asking whether yum's transacation-completion machinery was applicable
here.

In the meantime, I suspect that we will recommend the use of a temporary
off-NAND swap partition or swap-file by the people who need it.

Michael

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, Sep 22, 2008 at 5:00 PM, Michael Stone <michael@laptop.org> wrote:
> I'm sure there are easier things to prune than yum itself.

Perhaps you are right. I was probably expressing my own worries about
yum memory usage on the XO :-)

In a 30s unscientific test yum does seem to consume a fair bit of RAM
while RPM runs. The F9 machine I use for dev/build purposes had 9
updates to apply.

I just did `yum -yt update` while running repeatedly `ps_mem.py |
grep yum ` (in hindsight, I should have grepped for rpm as well). It
quickly ramped up to 44.9 MiB private + 1.5 MiB shared during the
Resolving Dependencies stage, and stayed there for the duration. While
rpm was running, yum was _still_ sitting on thos 46MiB.

It looks like it has the whole package metadata in a memory structure.
Perhaps it's needed for the plugins, though I hope the internal APIs
don't require that all that stuff is in memory.

> whether yum's transacation-completion machinery was applicable
> here.

>From what I read in this thread the transaction-completion machinery
doesn't try to parse the repo data into memory. It just reads the plan
from the transaction log and tries to execute it. If that's true, then
it's got a good chance as a workaround.

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, 2008-09-22 at 17:19 +1200, Martin Langhoff wrote:
> On Mon, Sep 22, 2008 at 5:00 PM, Michael Stone <michael@laptop.org> wrote:
> > I'm sure there are easier things to prune than yum itself.
>
> Perhaps you are right. I was probably expressing my own worries about
> yum memory usage on the XO :-)
>
> In a 30s unscientific test yum does seem to consume a fair bit of RAM
> while RPM runs. The F9 machine I use for dev/build purposes had 9
> updates to apply.
>
> I just did `yum -yt update` while running repeatedly `ps_mem.py |
> grep yum ` (in hindsight, I should have grepped for rpm as well). It
> quickly ramped up to 44.9 MiB private + 1.5 MiB shared during the
> Resolving Dependencies stage, and stayed there for the duration. While
> rpm was running, yum was _still_ sitting on thos 46MiB.

What arch? How many repos. did you have enabled?
In my tests the latest yum does a "large" update in about 40MB RSS, a
few packages is much more likely to be in the 25-30MB range (VSZ is
higher, but is just unpaged stuff, not actual swap).

> It looks like it has the whole package metadata in a memory structure.
> Perhaps it's needed for the plugins, though I hope the internal APIs
> don't require that all that stuff is in memory.

We drop all the cached data from within yum when the depsolving is
finished, however we have little control of how much of that is then
usable by rpm to do the transaction (that'd be mostly upto python and
libc).
We also have plans to split transactions, but I'm not sure you should
rely on that to significantly drop the memory usage requirements
(although it will make the failure case much nicer).

--
James Antill <james.antill@redhat.com>
Red Hat
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

2008/9/22 James Antill <james.antill@redhat.com>:
> What arch? How many repos. did you have enabled?

i386 - and the very minimum repos

$ yum repolist
Loaded plugins: refresh-packagekit
repo id repo name status
fedora Fedora 9 - i386 enabled : 9,897
updates Fedora 9 - i386 - Updates enabled : 10
updates-newkey Fedora 9 - i386 - Updates Newkey enabled : 3,951
repolist: 13,858

> In my tests the latest yum does a "large" update in about 40MB RSS, a
> few packages is much more likely to be in the 25-30MB range (VSZ is
> higher, but is just unpaged stuff, not actual swap).

I'm looking at the output of ps_mem.py which uses the kernel smap data
-- which I understand is regarded as much more reliable than what top
reports these days - see
http://www.pixelbeat.org/scripts/ps_mem.py

> We drop all the cached data from within yum when the depsolving is
> finished, however we have little control of how much of that is then

There was no observable drop in memory footprint from depsolving to
rpm stages, even if we were just updating 9 packages.

If my rough test is reasonably accurate then it's easy to see why
people would get OOM with yum on the XO -- just reading the release
repo, yum is sitting on a good chunk of our available memory, and we
have _no_ swap available.

One workaround would be to say --disablerepo=release but that means
any install that requires packages from the main repo will fail.

> We also have plans to split transactions, but I'm not sure you should
> rely on that to significantly drop the memory usage requirements
> (although it will make the failure case much nicer).

I'm *all* for making the failure case better as the XS installations
will fail... plenty :-)

cheers,



m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff

--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

On Mon, 2008-09-22 at 16:53 +1200, Martin Langhoff wrote:
> On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal <skvidal@fedoraproject.org> wrote:
> > Not much - if it's OOM'ing the only real option is:
> > 1. get more memory
> > 2. have it use less memory
> >
> > if you want to pursue 2 that's fine - but to be very clear - everything
> > after:
> > Running RPM Transaction Test
> >
> > is in rpm-land as to trimming out memory/cpu usage. There's very little
> > yum can do there.
>
> I *think* that what Michael is talking about is that when you get to
> that point yum is still running and can be a substantial part of the
> memory pressure.
>
> Perhaps yum's memory footprint could be trimmed?
How much memory do you have?

I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.

In the past, yum occasionally had OOM'ed, but this hasn't happened for
quite a while. The key to me had been to slim down the kernel memory
requirements (Switching off SELinux worked wonders! This particular
machine OOMs when booting with SELinux enabled)

Ralf




--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list

Quoth Ralf Corsepius:
> On Mon, 2008-09-22 at 16:53 +1200, Martin Langhoff wrote:
> > On Mon, Sep 22, 2008 at 4:42 PM, Seth Vidal <skvidal@fedoraproject.org>
wrote:
> > > Not much - if it's OOM'ing the only real option is:
> > > 1. get more memory
> > > 2. have it use less memory
> > >
> > > if you want to pursue 2 that's fine - but to be very clear - everything
> > > after:
> > > Running RPM Transaction Test
> > >
> > > is in rpm-land as to trimming out memory/cpu usage. There's very little
> > > yum can do there.
> >
> > I *think* that what Michael is talking about is that when you get to
> > that point yum is still running and can be a substantial part of the
> > memory pressure.
> >
> > Perhaps yum's memory footprint could be trimmed?
> How much memory do you have?
>
> I am able to run yum (FC9) on an i586 w/ 64MB RAM and 256MB swap.
>
> In the past, yum occasionally had OOM'ed, but this hasn't happened for
> quite a while. The key to me had been to slim down the kernel memory
> requirements (Switching off SELinux worked wonders! This particular
> machine OOMs when booting with SELinux enabled)
>
> Ralf

Right, I run yum on a similar (i586, 128M of ram, plenty of swap) machine and
it works great nowadays.

Regards,
--
Conrad Meyer <konrad@tylerc.org>


--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list