FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Ubuntu > Ubuntu User

 
 
LinkBack Thread Tools
 
Old 07-19-2012, 10:10 AM
Wipe_Out
 
Default NFS performance (can't use "async")

Hi,
Recently I have been testing various options to create a NFS server for storing virtual machine disk images.. For data*integrity*sake running as "async" is not an option and this appears to be the most commonly recommended "improvement"..

The issue I have is not so much with raw throughput, I have been able to saturate a 1GbE connections easily.. The issue is more to do with write operations.. On FreeBSD+ZFS I can get about 800 FSYNCS/s.. On Ubuntu 12.04 (tested with ext3/ext4/XFS/BtrFS) I can only get 15-20.. This is on the exact same hardware..

I don't think its*file system*related because of all the file systems ZFS and BtrFS should have the biggest overhead but switching from ext4 to BtrFS didn't change the performance results..

I haven't yet tried other distro's to see if the slow NFS is Ubuntu related but I can't see why is would be..
Does anyone have any ideas why there is such a huge difference between the performance of the FreeBSD and Ubuntu NFS servers?
--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 07-19-2012, 10:25 AM
Steve Flynn
 
Default NFS performance (can't use "async")

On 19 July 2012 11:10, Wipe_Out <wipe_out@users.sourceforge.net> wrote:

> I don't think its file system related because of all the file systems ZFS
> and BtrFS should have the biggest overhead but switching from ext4 to BtrFS
> didn't change the performance results..

Which would leave the network and the interface config. Have you
ensured that the NFS mount is being made with the exact same options
under both operating systems and that the network interface which the
data is flowing over is configured in the same manner?

Is this v3 or v4 NFS?

I only use NFS under AIX (and even then, we only use v3) so I can't
comment on any particular aspect of NFS throughput under Ubuntu or
FreeBSD but that's where I'd start comparing configs and set-up. It's
a hell of a difference you're seeing there so I suspect it's something
fundamental rather than a piss-poor Ubuntu implementation.

--
Steve

When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 07-19-2012, 11:04 AM
Wipe_Out
 
Default NFS performance (can't use "async")

On 19 July 2012 11:25, Steve Flynn <anothermindbomb@gmail.com> wrote:



Which would leave the network and the interface config. Have you

ensured that the NFS mount is being made with the exact same options

under both operating systems and that the network interface which the

data is flowing over is configured in the same manner?



Is this v3 or v4 NFS?



I only use NFS under AIX (and even then, we only use v3) so I can't

comment on any particular aspect of NFS throughput under Ubuntu or

FreeBSD but that's where I'd start comparing configs and set-up. It's

a hell of a difference you're seeing there so I suspect it's something

fundamental rather than a piss-poor Ubuntu implementation.



--

Hi Steve,
Both are exported and mounted NFSv3 and looking at the /etc/exports files of the two systems I couldn't see anything significantly different between them..

Using the exact same hardware so the NIC's, cables and switch is the same..*The network aspects of the kernels may be tuned differently but not sure how to compare these..

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 07-19-2012, 11:27 AM
Steve Flynn
 
Default NFS performance (can't use "async")

On 19 July 2012 12:04, Wipe_Out <wipe_out@users.sourceforge.net> wrote:

> Both are exported and mounted NFSv3 and looking at the /etc/exports files of
> the two systems I couldn't see anything significantly different between
> them..

No surprise there. For reference, we export NFS mounts like so:

/some_filesystem -public,sec=sys,rw,sec=dh:krb5:krb5i:krb5p,rw

and we mount with

/some_filesystem:
dev = "/client_data"
vfs = nfs
nodename = one_of_our_aix_servers
mount = true
options = bg,hard,intr,sec=sys:dh:krb5:krb5i:krb5p
account = false


> Using the exact same hardware so the NIC's, cables and switch is the same..
> The network aspects of the kernels may be tuned differently but not sure how
> to compare these..

(I'm no expert so take this with the usual grain of salt)

I'd start with an eyeball of the output from "ifconfig -a" on both
systems. Make sure they both look sensible and that one of them isn't
configured to route all traffic through some flaky old F5 firewall in
the basement, by accident.

Providing both installations look the same, have the same MTU,
sensible error levels for Tx and Rx after a test session and so forth
it's time to start poking around in /proc to compare tuning values.
This is where I can offer little, as I have no Linux or FreeBSD
systems to hand - just AIX which isn't going to help you much.

This /might/ help: http://nfs.sourceforge.net/nfs-howto/ar01s05.html


Out of interest, what are you using to generate the load on the NFS
filesystem and what are you using to measure the number of FSYNCS? Are
you sure that the two are equivalent under both installations (e.g.
bonnie on Linux is running with a 64K chunk of data to write to the
NFS filesystem whereas whatever you're using on FreeBSD is running
with a 2K chunk)... I'm struggling to come up with a plausible reason
for such a massive difference in sync rates between two installations
on the same kit, talking to the same filesystem.

--
Steve

When one person suffers from a delusion it is insanity. When many
people suffer from a delusion it is called religion.

--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 07-19-2012, 05:42 PM
Wipe_Out
 
Default NFS performance (can't use "async")

On 19 July 2012 12:27, Steve Flynn <anothermindbomb@gmail.com> wrote:



No surprise there. For reference, we export NFS mounts like so:



/some_filesystem -public,sec=sys,rw,sec=dh:krb5:krb5i:krb5p,rw



and we mount with



/some_filesystem:

* * * * dev * * * * * * = "/client_data"

* * * * vfs * * * * * * = nfs

* * * * nodename * * * *= one_of_our_aix_servers

* * * * mount * * * * * = true

* * * * options * * * * = bg,hard,intr,sec=sys:dh:krb5:krb5i:krb5p

* * * * account * * * * = false





(I'm no expert so take this with the usual grain of salt)



I'd start with an eyeball of the output from "ifconfig -a" on both

systems. Make sure they both look sensible and that one of them isn't

configured to route all traffic through some flaky old F5 firewall in

the basement, by accident.



Providing both installations look the same, have the same MTU,

sensible error levels for Tx and Rx after a test session and so forth

it's time to start poking around in /proc to compare tuning values.

This is where I can offer little, as I have no Linux or FreeBSD

systems to hand - just AIX which isn't going to help you much.



This /might/ help: http://nfs.sourceforge.net/nfs-howto/ar01s05.html





Out of interest, what are you using to generate the load on the NFS

filesystem and what are you using to measure the number of FSYNCS? Are

you sure that the two are equivalent under both installations (e.g.

bonnie on Linux is running with a 64K chunk of data to write to the

NFS filesystem whereas whatever you're using on FreeBSD is running

with a 2K chunk)... I'm struggling to come up with a plausible reason

for such a massive difference in sync rates between two installations

on the same kit, talking to the same filesystem.




Thanks for your thoughts and input Steve.. Might just stick with FreeBSD for the storage server and Ubuntu/Debian for the KVM hosts..
Still have to investigate the option of using iSCSI+LVM for VM drives instead of file based virtual disks over NFS so don't want to loose too much more time on the Linus NFS issue..

I am testing the storage from a Proxmox VE server which includes a utility called pveperf that tests the FSYNC's per second.. Below is the relevant source code for the test..

sub test_fsync {* * my $basedir = shift;* * drop_cache ();* * my $dir = "$basedir/ptest.$$";* * eval { mkdir $dir;
my $data = "" x 4000) . "
"; my $starttime = [gettimeofday];
my $count; my $elapsed = 0; for ($count=1;;$count++) {
* *my $m = $count % 300; * *my $filename = "$dir/tf_$m.dat"; * *open (TMP, ">$filename") || die "open failed";
* *print TMP $data; * *File::Sync::fsync (*TMP); * *close (TMP);
* *$elapsed = tv_interval ($starttime); * *last if $elapsed > 3; }
my $sps = $count /$elapsed; # fsync per second printf "FSYNCS/SECOND: * * %.2f
", $sps;
* * };* * my $err = $@;* * system ("rm -rf $dir");* * die $err if $err;}
--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 
Old 07-19-2012, 07:41 PM
NoOp
 
Default NFS performance (can't use "async")

On 07/19/2012 03:10 AM, Wipe_Out wrote:
> Hi,
>
> Recently I have been testing various options to create a NFS server for
> storing virtual machine disk images.. For data integrity sake running as
> "async" is not an option and this appears to be the most commonly
> recommended "improvement"..
>
> The issue I have is not so much with raw throughput, I have been able to
> saturate a 1GbE connections easily.. The issue is more to do with write
> operations.. On FreeBSD+ZFS I can get about 800 FSYNCS/s.. On Ubuntu 12.04
> (tested with ext3/ext4/XFS/BtrFS) I can only get 15-20.. This is on the
> exact same hardware..
>
> I don't think its file system related because of all the file systems ZFS
> and BtrFS should have the biggest overhead but switching from ext4 to BtrFS
> didn't change the performance results..
>
> I haven't yet tried other distro's to see if the slow NFS is Ubuntu related
> but I can't see why is would be..
>
> Does anyone have any ideas why there is such a huge difference between the
> performance of the FreeBSD and Ubuntu NFS servers?

<https://launchpad.net/+search?field.text=nfs+performance&x=0&y=0>

Perhaps you are running into these bugs?
<https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1006446>
[nfs4 causes enormous load in ubuntu-server making it unusable]
<quote>
SLOWNESS:
That is the thing. The ubuntu nfs server is actually NOT slow at all.
During my dd tests, it wrote half a gigabyte per second to its iscsi
backends. It's _throughput_ is better than that of the redhat server.
As far as I can tell, it falls down because it makes client side
processes that want to do IO wait on other writes. A simple 'ls' has to
wait until a write has been completed. And both server and client used
async nfs. People's firefoxes freeze all the time because firefox need
to read and write a lot to its cache and other files in the .mozilla
directory.
The dovecot imap server almost grinds to a halt trying to write all
those little files in people's /home/$USER/Maildir's.
The problems go on and on. Basically, a complete network of workstations
is almost unusable because of this.
Upfront tests were done of course, but showed only the excellent
throughput but not the appalling `waiting´ behaviour.
With redhat 6 there is no such problem.
</quote>

That bug was been marked as a duplicate of:
<https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/879334>
[nfsd from nfs-kernel-server very slow and system load from 25%-100%
from nfsd]




--
ubuntu-users mailing list
ubuntu-users@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
 

Thread Tools




All times are GMT. The time now is 08:19 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org