Default use of caching on raw volumes
 

Looks like i'm in a writing mood today, so just bare with me :)
A few weeks ago i had problems on one of my kvm hosts with cpu loads. It wasn't very obvious problem, since none of the vm's were having a high cpu usage. Things started to get weird when i upgraded all of the vm's to lucid ( some reinstalled clean, some upgraded ). At times loadavg did peak to 32 on a 16 core server. I started disabling services on the vm's until i found the reason - the zabbix monitoring i use ( for total of 10 or so hosts ) *was writing ( apparently ) too much data to the disk in it's pgsql database. The zabbix server and the pgsql server are on separate vm's. I went on and disabled zabbix for a while. I am using raw, allocated images for performance and it was still hanging up the system during extensive read/writes. Then i stumbled upon this :*
https://www.redhat.com/archives/virt-tools-list/2010-August/msg00027.html
I went ahead and put the cache='none' on the drives of the pgsql vm and there you have it - loadavg went down to 1-3 from 30+ with zabbix running.I did some testing with force destroying the vm around 20 times and i didn't encounter any data loss whatsoever.Now the question is - did the ubuntu team do some testing with this ? If yes, did you find anything that breaks up so bad, that it's not ready for an LTS or any other sort of release.And if testing shows things are ok, would there be any merit in changing either libvirt or virt-manager to do this by default just for ubuntu ( not really sure what upstream will do about it ) ?

--

Nikolai K. Bochev
System Administrator



--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

On 2010-09-01 21:19, Nikolai K. Bochev wrote:

> And if testing shows things are ok, would there be any merit in changing either libvirt or virt-manager to do this by default just for ubuntu ( not really sure what upstream will do about it ) ?

FWIW, I would welcome such a change myself. Adding the cache='none'
directive to a VM's .xml file is the very first thing I do after its
installation, for performance reasons.

I have been running several VMs (all Lucid) for about four months now
that way, without ever encountering any issues¹.

The host is also Lucid, with unattended-upgrades set to automatically
reboot the machine when necessary, so the VMs sometimes just get killed.


Regards,
Oliver


1) well, besides that disk I/O from the guests is still too slow for my
taste. I get near native speed (about 80 MB/s) using direct i/o during
tests with dd, but regular i/o during normal operation is only about 35
MB/s. If anyone knows how to force direct i/o on all processes (or
something with similar results), please let me know.


--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

Quoting Nikolai K. Bochev (n.bochev@grandstarco.com):
> Now the question is - did the ubuntu team do some testing with this ?

I don't know about Dustin. I myself haven't really played with the
caching options.

From what I see online it definately seems to make a huge difference.
For Lucid and Maverick I think the best we can do at this point is to
tell the user about it. For Natty, well the best option is to have you
work with upstream to change the default there. Sounds like you've got
a posse forming to back you up :) In either case, some precise
benchmarking (involving say 30 runs of each test and showing mean +/-
95% confidence interval) might be enough to convince me that it was
worth writing and carrying a patch to change the default in Ubuntu,
which I would then try to push upstream.

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

+ Serge E. Hallyn <serge.hallyn@canonical.com> wrote:

> From what I see online it definately seems to make a huge difference.
> For Lucid and Maverick I think the best we can do at this point is to
> tell the user about it. *For Natty, well the best option is to have you
> work with upstream to change the default there. *Sounds like you've got
> a posse forming to back you up *:)

Just want to point out that there's a bug about this on LP:
https://bugs.launchpad.net/ubuntu/+source/virt-manager/+bug/568445

--
Alex Muntada <alexm@alexm.org>
http://alexm.org/

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

Quoting Alex Muntada (alexm@alexm.org):
> + Serge E. Hallyn <serge.hallyn@canonical.com> wrote:
>
> > From what I see online it definately seems to make a huge difference.
> > For Lucid and Maverick I think the best we can do at this point is to
> > tell the user about it. *For Natty, well the best option is to have you
> > work with upstream to change the default there. *Sounds like you've got
> > a posse forming to back you up *:)
>
> Just want to point out that there's a bug about this on LP:
> https://bugs.launchpad.net/ubuntu/+source/virt-manager/+bug/568445

Hm, I see, and Anthony's comment #6 makes a lot of sense to me.
Like he says, I only use virt-manager for casual use. For real
tests, well I tend to use kvm cmdline, but on remote servers
lately I use libvirt directly so that I can do things like create
base.xml, then
for i in `seq 1 20`; do
cp base.xml vm$i.xml
sed -i 'whatever' vm$i.xml
virsh define vm$i.xml
done

So I guess before we consider carrying a patch just in ubuntu,
we should answer the question - do we expect users who are trying
out kvm just once to use virt-manager or testdrive?

-serge

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

On Thu, 2010-09-02 at 08:15 -0500, Serge E. Hallyn wrote:
> Quoting Alex Muntada (alexm@alexm.org):
> > + Serge E. Hallyn <serge.hallyn@canonical.com> wrote:
> >
> > > From what I see online it definately seems to make a huge difference.
> > > For Lucid and Maverick I think the best we can do at this point is to
> > > tell the user about it. For Natty, well the best option is to have you
> > > work with upstream to change the default there. Sounds like you've got
> > > a posse forming to back you up :)
> >
> > Just want to point out that there's a bug about this on LP:
> > https://bugs.launchpad.net/ubuntu/+source/virt-manager/+bug/568445
>
> Hm, I see, and Anthony's comment #6 makes a lot of sense to me.
> Like he says, I only use virt-manager for casual use. For real
> tests, well I tend to use kvm cmdline, but on remote servers
> lately I use libvirt directly so that I can do things like create
> base.xml, then
> for i in `seq 1 20`; do
> cp base.xml vm$i.xml
> sed -i 'whatever' vm$i.xml
> virsh define vm$i.xml
> done
>
> So I guess before we consider carrying a patch just in ubuntu,
> we should answer the question - do we expect users who are trying
> out kvm just once to use virt-manager or testdrive?

Anthony's observation about virt-manager may be valid, but I sure know a
lot of people who use it every day as a part of their daily workflow (me
not included -- I use virsh/libvirt exclusively (surprise)).

I like Serge's idea of testing and if there is a positive improvement,
make the change in libvirt itself so all consumers (like eucalyptus,
openstack, straight virsh users, etc) can benefit. If that doesn't fit
virt-manager use cases, update it to use a different sensible default.

--
Jamie Strandboge | http://www.canonical.com
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam

Default use of caching on raw volumes
 

Serge E. Hallyn <serge.hallyn@canonical.com> wrote:
> Quoting Nikolai K. Bochev (n.bochev@grandstarco.com):
>> Now the question is - did the ubuntu team do some testing with this ?
>
> I don't know about Dustin. *I myself haven't really played with the
> caching options.

I have, only a bit though. Mostly to make an informed default in the
TestDrive wrapper for KVM.

I tested and verified this part of the kvm(1) manpage:
* http://manpages.ubuntu.com/kvm

Some block drivers perform badly with cache=writethrough, most
notably, qcow2. If performance is more important than correctness,
cache=writeback should be used with qcow2.

For TestDrive, "performance" is more important than "correctness"
since most TestDrive VMs are just throwaway VMs, and all use qcow2.

I have not conducted any deeper analysis, though. Sorry.

:-Dustin

--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam