FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu Server Development

 
 
LinkBack Thread Tools
 
Old 03-25-2010, 07:09 PM
Jamie Strandboge
 
Default libvirt 0.7.7 and Lucid

Hi,

For some work[1] I am doing, I did a merge of 0.7.7-4 from Debian
unstable for Lucid and made it available in my PPA[2] (though it hasn't
built yet-- people can grab the source and build it locally if desired).
This package may be worth considering for Lucid, since there are many
bug fixes and few new features in 0.7.6 and 0.7.7[3] (Lucid currently
has 0.7.5).

My testing with QRT[4], manually and the internal test suite (enabled by
default in the build) shows that 0.7.7 is quite solid, though it does a
couple of things differently:

1. the setmem, setmaxmem and setvcpus virsh commands are for hotplugging
memory and vcpus only. This was always the intent of these commands, but
libvirt did not enforce it. Now it does and qemu-kvm doesn't seem to
support hotplugging of memory and vcpus (at least with how libvirt
interfaces with it). In practice, this probably won't affect many users
as the recommended method has always been to use 'virsh define' and
restart the guest. virt-manager appears to use 'virsh define' for
setmaxmem, I don't know how eucalyptus deals with this.

2. upstream decided to make setmaxmem go away entirely, and 0.7.7's
implementation of setmaxmem is in the middle of this transition and
doesn't seem to work at all. 'virsh define' still does though. I'm also
not sure if this is a bug or new design, but while you can define a VM
with different <memory> and <currentMemory>, when you start the VM,
<memory> is always allocated for the VM and <currentMemory> is ignored.

3. libvirt seems to try to guard against hypervisor issues regarding
detach-device and detach-disk. Specifically, these can be hot-plugged,
but not hot-unplugged (libvirt claims the qemu-kvm hypervisor doesn't
support detach of these):
<disk type='block' device='disk'>
<driver name='phy'/>
<source dev='...path.../device_disk.img'/>
<target dev='sdb' bus='scsi'/>
<alias name='scsi0-0-1'/>
<address type='drive' controller='0' bus='0' unit='1'/>
</disk>

<disk type='file' device='disk'>
<driver name='file'/>
<source file='...path.../device_disk.img'/>
<target dev='sdc' bus='scsi'/>
<alias name='scsi0-0-2'/>
<address type='drive' controller='0' bus='0' unit='2'/>
</disk>

In 0.7.5, libvirt would try to detach these devices via the hypervisor,
but in 0.7.7 it gives only a clear error message to the user. These QRT
tests were built around upstream eucalyptus functionality, so this needs
careful testing. Attaching and detaching virtio disks works fine in my
testing.

4. libvirt chown's the disk files to root:root for people using
qemu:///system. I don't know why it is doing this, but it is likely
related to upstream changes (and assumptions) made for the DAC security
driver. This seems like someone will need to at least investigate if not
patch.


Beyond the above, preliminary testing indicates that 0.7.7 is quite
solid. I would like to see this go into Lucid, since my apparmor work
will be based on this and I'd rather not have to backport my work to
0.7.5. That said, I don't have the time to perform all the testing
required, so if a commitment to testing resources can't be made for
0.7.7 by the server team and QA, then I recommend sticking with 0.7.5.

[1]https://blueprints.launchpad.net/ubuntu/+spec/security-lucid-libvirt-apparmor-devel
[2]https://launchpad.net/~jdstrand/+archive/ppa?field.series_filter=lucid
[3]http://libvirt.org/news.html
[4]https://code.launchpad.net/~ubuntu-bugcontrol/qa-regression-testing/master

--
Jamie Strandboge | http://www.canonical.com
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 03-31-2010, 03:14 PM
Dustin Kirkland
 
Default libvirt 0.7.7 and Lucid

Sorry for my delayed response. I have had IRC conversations with
Jamie, Thierry, and others about this, but I felt that I should roll
that up and respond inline below for the sake of posterity and
accuracy of the Wayback Machine years from now ;-)

On Thu, Mar 25, 2010 at 1:09 PM, Jamie Strandboge <jamie@canonical.com> wrote:
> Hi,
>
> For some work[1] I am doing, I did a merge of 0.7.7-4 from Debian
> unstable for Lucid and made it available in my PPA[2] (though it hasn't
> built yet-- people can grab the source and build it locally if desired).
> This package may be worth considering for Lucid, since there are many
> bug fixes and few new features in 0.7.6 and 0.7.7[3] (Lucid currently
> has 0.7.5).

Thanks, Jamie, for your sustained work on libvirt. Your help is much
appreciated by the Ubuntu Server Team ;-)

> My testing with QRT[4], manually and the internal test suite (enabled by
> default in the build) shows that 0.7.7 is quite solid, though it does a
> couple of things differently:

Gotta love knowing that there's an upstream test suite ;-)

> 1. the setmem, setmaxmem and setvcpus virsh commands are for hotplugging
> memory and vcpus only. This was always the intent of these commands, but
> libvirt did not enforce it. Now it does and qemu-kvm doesn't seem to
> support hotplugging of memory and vcpus (at least with how libvirt
> interfaces with it). In practice, this probably won't affect many users
> as the recommended method has always been to use 'virsh define' and
> restart the guest. virt-manager appears to use 'virsh define' for
> setmaxmem, I don't know how eucalyptus deals with this.

I'm pretty sure that Eucalyptus does not use memory or CPU hotplug. I
have CC'd Dan Nurmi on this note to definitely confirm. Dan- every
Eucalyptus instance is started anew from a fresh libvirt xml file,
with mem/cpu/disk specified by the user's selection of machine-type,
correct? And there's no hotplug of memory or cpu to running
Eucalyptus instances, right?

> 2. upstream decided to make setmaxmem go away entirely, and 0.7.7's
> implementation of setmaxmem is in the middle of this transition and
> doesn't seem to work at all. 'virsh define' still does though. I'm also
> not sure if this is a bug or new design, but while you can define a VM
> with different <memory> and <currentMemory>, when you start the VM,
> <memory> is always allocated for the VM and <currentMemory> is ignored.

Interesting. I can't exactly say what this will affect. Does the new
Virt-Manager handle this by disabling the prompt for both memory and
currentMemory? This has been a source of confusion for Virt-Manager
users for some time now, so I think it would be good to simplify this,
if that's what they've done in Libvirt upstream.

> 3. libvirt seems to try to guard against hypervisor issues regarding
> detach-device and detach-disk. Specifically, these can be hot-plugged,
> but not hot-unplugged (libvirt claims the qemu-kvm hypervisor doesn't
> support detach of these):
> * *<disk type='block' device='disk'>
> * * *<driver name='phy'/>
> * * *<source dev='...path.../device_disk.img'/>
> * * *<target dev='sdb' bus='scsi'/>
> * * *<alias name='scsi0-0-1'/>
> * * *<address type='drive' controller='0' bus='0' unit='1'/>
> * *</disk>
>
> * *<disk type='file' device='disk'>
> * * *<driver name='file'/>
> * * *<source file='...path.../device_disk.img'/>
> * * *<target dev='sdc' bus='scsi'/>
> * * *<alias name='scsi0-0-2'/>
> * * *<address type='drive' controller='0' bus='0' unit='2'/>
> * *</disk>
>
> In 0.7.5, libvirt would try to detach these devices via the hypervisor,
> but in 0.7.7 it gives only a clear error message to the user. These QRT
> tests were built around upstream eucalyptus functionality, so this needs
> careful testing. Attaching and detaching virtio disks works fine in my
> testing.

Uh, oh... Okay, this might be the big-nasty. Eucalyptus does use
scsi hotplug for attaching EBS volumes. Not sure about detach...
Dan, again can you advise us here? Does Eucalyptus need to hot unplug
scsi disks from instances?

As for virtio, we're planning to file a blueprint for Lucid+1 and try
to get UEC using virtio for networking and disks.

> 4. libvirt chown's the disk files to root:root for people using
> qemu:///system. I don't know why it is doing this, but it is likely
> related to upstream changes (and assumptions) made for the DAC security
> driver. This seems like someone will need to at least investigate if not
> patch.

Hmm, okay, I think this is okay. Looking at
/var/lib/eucalyptus/instances/admin/*/disk, these are owned by
root:root right now with libvirt 0.7.5-5ubuntu15 and eucalyptus
1.6.2-0ubuntu26, which is working.

> Beyond the above, preliminary testing indicates that 0.7.7 is quite
> solid. I would like to see this go into Lucid, since my apparmor work
> will be based on this and I'd rather not have to backport my work to
> 0.7.5. That said, I don't have the time to perform all the testing
> required, so if a commitment to testing resources can't be made for
> 0.7.7 by the server team and QA, then I recommend sticking with 0.7.5.

Right, so Jamie and I are going to spend most of Thursday this week,
dedicating time to testing libvirt 0.7.7, and specifically with UEC.

If we can quickly fix any regressions that we find, then I think it's
fine to upload libvirt 0.7.7 to Lucid.

However, if we encounter regressions with UEC and libvirt 0.7.7 which
are not quickly stomped, then I would probably recommend against the
new version for Lucid, as we've come so far with UEC and I'm quite
happy with its stability for 10.04 LTS at this point.

Is that reasonable?

:-Dustin

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-01-2010, 06:01 PM
Scott Moser
 
Default libvirt 0.7.7 and Lucid

On Wed, 31 Mar 2010, Dustin Kirkland wrote:
> On Thu, Mar 25, 2010 at 1:09 PM, Jamie Strandboge <jamie@canonical.com> wrote:
> > 3. libvirt seems to try to guard against hypervisor issues regarding
> > detach-device and detach-disk. Specifically, these can be hot-plugged,
> > but not hot-unplugged (libvirt claims the qemu-kvm hypervisor doesn't
> > support detach of these):
> > * *<disk type='block' device='disk'>
> > * * *<driver name='phy'/>
> > * * *<source dev='...path.../device_disk.img'/>
> > * * *<target dev='sdb' bus='scsi'/>
> > * * *<alias name='scsi0-0-1'/>
> > * * *<address type='drive' controller='0' bus='0' unit='1'/>
> > * *</disk>

hotplug attach and detach of scsi driver disks is an absolute requirement
for UEC with Eucalyptus 1.6.2. It is the basis of the "EBS" (Elastic
Block Store) functionality (euca-attach-volume, euca-detach-volume).

Our kernel team recently fixed bug 458201 so the user no longer sees scary
messages in dmesg when they *do* detach. In my [limited]
experience/discussion with kvm folks it is generally accepted that scsi
device emulation is to be avoided. However, we're stuck with it for now.

> As for virtio, we're planning to file a blueprint for Lucid+1 and try
> to get UEC using virtio for networking and disks.

Heres to hoping we can do something here.

> > 4. libvirt chown's the disk files to root:root for people using
> > qemu:///system. I don't know why it is doing this, but it is likely
> > related to upstream changes (and assumptions) made for the DAC security
> > driver. This seems like someone will need to at least investigate if not
> > patch.
>
> Hmm, okay, I think this is okay. Looking at
> /var/lib/eucalyptus/instances/admin/*/disk, these are owned by
> root:root right now with libvirt 0.7.5-5ubuntu15 and eucalyptus
> 1.6.2-0ubuntu26, which is working.

Could this be related to apparmor ? As I found in
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/544435 (but I guess
I ddin't comment there). If you chown root:root the qemu source and qemu
backing device of a qcow image it will work. If either is user-owned, it
will not.
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-02-2010, 03:29 AM
Dustin Kirkland
 
Default libvirt 0.7.7 and Lucid

On Thu, Apr 1, 2010 at 12:01 PM, Scott Moser <smoser@ubuntu.com> wrote:
> On Wed, 31 Mar 2010, Dustin Kirkland wrote:
>> On Thu, Mar 25, 2010 at 1:09 PM, Jamie Strandboge <jamie@canonical.com> wrote:
>> > 3. libvirt seems to try to guard against hypervisor issues regarding
>> > detach-device and detach-disk. Specifically, these can be hot-plugged,
>> > but not hot-unplugged (libvirt claims the qemu-kvm hypervisor doesn't
>> > support detach of these):
>> > * *<disk type='block' device='disk'>
>> > * * *<driver name='phy'/>
>> > * * *<source dev='...path.../device_disk.img'/>
>> > * * *<target dev='sdb' bus='scsi'/>
>> > * * *<alias name='scsi0-0-1'/>
>> > * * *<address type='drive' controller='0' bus='0' unit='1'/>
>> > * *</disk>
>
> hotplug attach and detach of scsi driver disks is an absolute requirement
> for UEC with Eucalyptus 1.6.2. *It is the basis of the "EBS" (Elastic
> Block Store) functionality (euca-attach-volume, euca-detach-volume).

Jamie and I spent a good bit of today testing libvirt 0.7.7, in
particular against Lucid UEC.

The basic tests from Mathias' lp:~mathiaz/%2Bjunk/uec-testing-scripts
all ran just fine, as well as my basic tests of virsh, and
virt-manager.

The package build also runs the upstream test suite, all test of which pass:
* http://launchpadlibrarian.net/42047543/buildlog_ubuntu-lucid-amd64.libvirt_0.7.7-4ubuntu1~jdstrand2_FULLYBUILT.txt.gz

However, hot add of scsi disks (as used by Eucalyptus's EBS
implementation) did not work, unfortunately. Note that we used these
instructions from Scott to test:
* http://paste.ubuntu.com/407717/

After a conversation with upstream in #virt in IRC, Jamie filed the
upstream bug:
* https://bugzilla.redhat.com/show_bug.cgi?id=578975

We think that we've identified the problematic commit (but this
doesn't unapply cleanly). See:
264e98d6a85bb467f364006c1dac3993688691ce [PATCH] Make hotplug use new
device_add where possible

Libvirt needs to go back to using pci_add for scsi hot attach of disks.

Jamie and I are in agreement that this regression is a blocker for
uploading 0.7.7.

We also defined the functional acceptance criteria for 0.7.7 upload to
be (assuming we can get this fixed):
* the upstream QA tests all pass
* the security regression tests all pass
* Mathias' UEC tests all pass
* and Dan's S3/EBS tests all pass

:-Dustin

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-02-2010, 04:08 PM
Jamie Strandboge
 
Default libvirt 0.7.7 and Lucid

On Thu, 2010-04-01 at 14:01 -0400, Scott Moser wrote:
> > > 4. libvirt chown's the disk files to root:root for people using
> > > qemu:///system. I don't know why it is doing this, but it is likely
> > > related to upstream changes (and assumptions) made for the DAC security
> > > driver. This seems like someone will need to at least investigate if not
> > > patch.
> >
> > Hmm, okay, I think this is okay. Looking at
> > /var/lib/eucalyptus/instances/admin/*/disk, these are owned by
> > root:root right now with libvirt 0.7.5-5ubuntu15 and eucalyptus
> > 1.6.2-0ubuntu26, which is working.
>
> Could this be related to apparmor ? As I found in
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/544435 (but I guess
> I ddin't comment there). If you chown root:root the qemu source and qemu
> backing device of a qcow image it will work. If either is user-owned, it
> will not.

I looked at this a bit more and this is not related to apparmor. 0.7.7
uses a stacked security driver implementation. The primary driver is
AppArmor, the secondary the DAC security driver (not to be confused with
standard DAC permissions, which are also checked by the kernel on exec
of qemu-kvm). If AppArmor allows it, then the DAC security driver is
consulted. The DAC security driver is what is doing the chowning. The
DAC security driver is always in use, so if you disable AppArmor or
SELinux, then DAC becomes the primary driver and there is no secondary
(and you'll see the same chowning). This is the new way from upstream.

--
Jamie Strandboge | http://www.canonical.com
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-02-2010, 04:22 PM
Dustin Kirkland
 
Default libvirt 0.7.7 and Lucid

Per conversation in IRC [1], I'm recommending that we stick with
libvirt 0.7.5 for Lucid, due to the incompatibility it introduces with
hotplug of scsi disks (and the resulting regression it introduces in
UEC/Eucalyptus for EBS volumes).

[1] http://irclogs.ubuntu.com/2010/04/02/%23ubuntu-server.html#t15:33

Maverick will certainly introduce >= libvirt 0.7.7, which will force
us to move to virtio disks for UEC/Eucalyptus (which will introduce a
nice performance bump, and possibly a stability enhancement, but most
importantly align us with upstream's recommended way of adding disks),
and we'll have an entire release cycle to work our way through any
issues that arise (rather than a mere 2 hectic weeks).

Tremendous thanks to you, Jamie, for all the work you've done on
Libvirt, helping the Ubuntu Server team out immensely in the Karmic
and Lucid cycles!!!

Cheers,
:-Dustin

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-02-2010, 04:58 PM
Jamie Strandboge
 
Default libvirt 0.7.7 and Lucid

On Fri, 2010-04-02 at 10:22 -0600, Dustin Kirkland wrote:
> Per conversation in IRC [1], I'm recommending that we stick with
> libvirt 0.7.5 for Lucid, due to the incompatibility it introduces with
> hotplug of scsi disks (and the resulting regression it introduces in
> UEC/Eucalyptus for EBS volumes).

I second this. If there were no regressions or something easy to
cherrypick, that would be one thing, but this is too big of a change.

> Maverick will certainly introduce >= libvirt 0.7.7, which will force
> us to move to virtio disks for UEC/Eucalyptus (which will introduce a
> nice performance bump, and possibly a stability enhancement, but most
> importantly align us with upstream's recommended way of adding disks),
> and we'll have an entire release cycle to work our way through any
> issues that arise (rather than a mere 2 hectic weeks).

If there are no objections, I plan to merge this in maverick very soon
after it opens.

> Tremendous thanks to you, Jamie, for all the work you've done on
> Libvirt, helping the Ubuntu Server team out immensely in the Karmic
> and Lucid cycles!!!

Thanks, and thanks to you (and the server and Eucalyptus teams) for your
time and a well considered decision.

--
Jamie Strandboge | http://www.canonical.com
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 
Old 04-02-2010, 05:33 PM
Jamie Strandboge
 
Default libvirt 0.7.7 and Lucid

On Fri, 2010-04-02 at 11:58 -0500, Jamie Strandboge wrote:
> On Fri, 2010-04-02 at 10:22 -0600, Dustin Kirkland wrote:
> > Per conversation in IRC [1], I'm recommending that we stick with
> > libvirt 0.7.5 for Lucid, due to the incompatibility it introduces with
> > hotplug of scsi disks (and the resulting regression it introduces in
> > UEC/Eucalyptus for EBS volumes).
>
> I second this. If there were no regressions or something easy to
> cherrypick, that would be one thing, but this is too big of a change.

Btw, this is also being tracked in:
https://launchpad.net/bugs/553737


--
Jamie Strandboge | http://www.canonical.com
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
 

Thread Tools




All times are GMT. The time now is 10:08 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org