aufs based upgrade tests
 

Hi,

during the last UDS we talked informally about using the "aufs"
overlay filesystem layer for release upgrade testing. I build a
prototype implementation of this now that should be ok for public
testing.

The idea discussed with Evan Dandrea (and others) was to create a
writable overlay into /tmp on top the systemdirs in "/" and then run
the release upgrade. This way we can test easily if the system would
upgrade cleanly (if no dpkg errors/maintainer script failures
happen). All writes go into /tmp so after the upgrade and on the next
reboot the system is back to its pre-upgraded state again (modulo
/home, that is not overlayed). It also means the next boot takes a
*long* time to clean /tmp - when I did test it on one of my production
machines that wait made me *really* nervous :) But its ok, it just takes
long (up to ~20 minutes or so).

Feedback is welcome, here is how to enable it:
$ sudo -s
# mkdir -p /etc/update-manager/release-upgrades.d/
# echo "[Aufs]" > /etc/update-manager/release-upgrades.d/aufs.cfg
# echo "EnableFullOverlay=yes" >> /etc/update-manager/release-upgrades.d/aufs.cfg
# exit
$ update-manager -d
(or do-release-upgrade -d)

then follow the upgrade procedure noramlly. You will see a message
dialog saying "Sandbox mode". If you don't see that, something is
wrong and you need to cancel the upgrade (no worries, just hit
"cancel" in the final confirmation screen). When the final
confirmation screen appears, check with "mount" on a terminal if there
are really "aufs" mounts. Remember to remove the file (aufs.cfg) after
the next reboot if you want to perform a regular upgrade (otherwise
the next run will be a sandbox upgrade again :)

Caveats:
- it got *very little testing*, so be careful and have backups
- it does not work with the intrepid server kernel, it lacks the aufs
module
- the free space calculation is wrong (too small) when aufs is
enabled, better have plenty of free space in /tmp (~4gb for
ubuntu-desktop)
- the system feels slow during the upgrade (more so than normal
due to aufs it seems, may also need more RAM, but I'm not sure about
this yet)
- the next reboot will take *ages* (because /tmp will be cleared from
all the files written during the upgrade)
- there is a problem with some packages (libthunar-vfs-1-2 is one)
where dpkg return "Invalid cross-device link" error. it seems
like its a combination of aufs/dpkg problem and currently under
investigation


Cheers,
Michael



--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

On Sat, Mar 14, 2009 at 3:19 AM, Michael Vogt <mvo@ubuntu.com> wrote:
> Feedback is welcome, here is how to enable it:

This worked as well as I would expect given that my mirror (iinet) is
missing many Jaunty files.

Perhaps it would be better not to overlay the /var/apt/cache
directory, so as to save the user from having to download files twice?
(I am not aware of any danger except that it could potentially cause
the /var partition to run out of free space if it didn't start with
enough free space.)

> dialog saying "Sandbox mode". If you don't see that, something is

FYI, I didn't see that dialog until I had clicked both
Update Manager->Upgrade
Release notes->Upgrade
downloaded the upgrade tool and entered my password. By that stage I
was a little worried that I had not seen the "Sandbox mode" dialog.
Perhaps this dialog could be displayed earlier? (before having to
enter the password at least?)

--
John C. McCabe-Dansted
PhD Student
University of Western Australia

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

Hey Michael,

Michael Vogt [2009-03-13 19:19 +0100]:
> The idea discussed with Evan Dandrea (and others) was to create a
> writable overlay into /tmp on top the systemdirs in "/" and then run
> the release upgrade.

That's a great idea!

Some questions:

* Is it possible to use some different directory than /tmp/ ? My root
partition doesn't have enough space; could that be set in the
configuration file?

* Once the upgrade is complete, why throw away /tmp? If I use a
different directory, couldn't we have some bits in initramfs which
check for those and re-mount the aufs? Then after reboot, if
everything seems to work, could another tool copy back the aufs
contents to the real file system, to "commit" the upgrade? Or
doesn't that work for some reason?

Thanks!

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

ma, 2009-03-16 kello 09:14 +0100, Martin Pitt kirjoitti:
> Hey Michael,
>
> Michael Vogt [2009-03-13 19:19 +0100]:
> > The idea discussed with Evan Dandrea (and others) was to create a
> > writable overlay into /tmp on top the systemdirs in "/" and then run
> > the release upgrade.
>
> That's a great idea!
>
> Some questions:
>
> * Is it possible to use some different directory than /tmp/ ? My root
> partition doesn't have enough space; could that be set in the
> configuration file?

Better yet, have it obey $TMPDIR.

> * Once the upgrade is complete, why throw away /tmp? If I use a
> different directory, couldn't we have some bits in initramfs which
> check for those and re-mount the aufs? Then after reboot, if
> everything seems to work, could another tool copy back the aufs
> contents to the real file system, to "commit" the upgrade? Or
> doesn't that work for some reason?

I thought /tmp was getting cleared up by the normal bootup routines.
Thus, if you set TMPDIR to something else than /tmp, you get to keep the
upgraded system, yes?


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

On Sun, Mar 15, 2009 at 01:40:22AM +0900, John McCabe-Dansted wrote:
> On Sat, Mar 14, 2009 at 3:19 AM, Michael Vogt <mvo@ubuntu.com> wrote:
> > Feedback is welcome, here is how to enable it:
>
> This worked as well as I would expect given that my mirror (iinet) is
> missing many Jaunty files.

Thanks for giving it a try!

> Perhaps it would be better not to overlay the /var/apt/cache
> directory, so as to save the user from having to download files twice?
> (I am not aware of any danger except that it could potentially cause
> the /var partition to run out of free space if it didn't start with
> enough free space.)

Thanks, that is a good suggestion, I added it in bzr and it will be
part of the next upload.

> > dialog saying "Sandbox mode". If you don't see that, something is
>
> FYI, I didn't see that dialog until I had clicked both
> Update Manager->Upgrade
> Release notes->Upgrade
> downloaded the upgrade tool and entered my password. By that stage I
> was a little worried that I had not seen the "Sandbox mode" dialog.
> Perhaps this dialog could be displayed earlier? (before having to
> enter the password at least?)

This is a bit tricky, I see what I can do. The update-manager in
intrepid does not know about this new mode, its relatively dumb and
just downloads the right code. That knows about the sandbox mode. I
should make this more obvious in my description at least.

Cheers,
Michael

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

On Mon, Mar 16, 2009 at 09:14:18AM +0100, Martin Pitt wrote:
> Hey Michael,
>
> Michael Vogt [2009-03-13 19:19 +0100]:
> > The idea discussed with Evan Dandrea (and others) was to create a
> > writable overlay into /tmp on top the systemdirs in "/" and then run
> > the release upgrade.
>
> That's a great idea!
>
> Some questions:
>
> * Is it possible to use some different directory than /tmp/ ? My root
> partition doesn't have enough space; could that be set in the
> configuration file?

Yes, if you use:
[Aufs]
RWDir=/some/dir

It will use that directory for the write operations. I will put
together a wiki page that describes the various options.

> * Once the upgrade is complete, why throw away /tmp? If I use a
> different directory, couldn't we have some bits in initramfs which
> check for those and re-mount the aufs? Then after reboot, if
> everything seems to work, could another tool copy back the aufs
> contents to the real file system, to "commit" the upgrade? Or
> doesn't that work for some reason?

That is the longer term goal indeed. This first step will help making
testing the upgrade process easier.

There is already some code in my branch that supports performing the
upgrade in a aufs overlay and writing the changes back to the real
system (via rsync) at the end of the upgrade (if no dpkg errors
occured). This seems to work already, but I have not done a really
good test to compare if the upgrade done with this process is bit-wise
identical to a normal one.

The next step would be integration with the initramfs tools so that
the user can test the changes and apply.

There is currently a known issue with the aufs/dpkg interaction:
https://bugs.edge.launchpad.net/ubuntu/+source/update-manager/+bug/342451

Aufs does not support rename() on direcotries under certain
conditions. This makes the libthunar-vfs-1-2 upgrade fail (a direcotry
with files in it is replaced with a single file of the same
name). This needs some dpkg love to make it work around this
limitation.

Cheers,
Michael

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

aufs based upgrade tests
 

On Mon, Mar 16, 2009 at 11:39:05AM +0200, Lars Wirzenius wrote:
> ma, 2009-03-16 kello 09:14 +0100, Martin Pitt kirjoitti:
> > Hey Michael,
> >
> > Michael Vogt [2009-03-13 19:19 +0100]:
> > > The idea discussed with Evan Dandrea (and others) was to create a
> > > writable overlay into /tmp on top the systemdirs in "/" and then run
> > > the release upgrade.
> >
> > That's a great idea!
> >
> > Some questions:
> >
> > * Is it possible to use some different directory than /tmp/ ? My root
> > partition doesn't have enough space; could that be set in the
> > configuration file?
>
> Better yet, have it obey $TMPDIR.

Thanks, this is fixed in bzr now.

> > * Once the upgrade is complete, why throw away /tmp? If I use a
> > different directory, couldn't we have some bits in initramfs which
> > check for those and re-mount the aufs? Then after reboot, if
> > everything seems to work, could another tool copy back the aufs
> > contents to the real file system, to "commit" the upgrade? Or
> > doesn't that work for some reason?
>
> I thought /tmp was getting cleared up by the normal bootup routines.
> Thus, if you set TMPDIR to something else than /tmp, you get to keep the
> upgraded system, yes?

If you set it to something else (that is not in /tmp), you keep the
overlay. The next reboot will still be intrepid but when you mount the
overlay (there is currently no automatic way for this), then you can
play with the jaunty system again.

Cheers,
Michael

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel