Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Red Hat Linux (http://www.linux-archive.org/red-hat-linux/)
-   -   Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7 (http://www.linux-archive.org/red-hat-linux/682014-bug-800181-nfsv4-rhel-6-2-over-six-times-slower-than-5-7-a.html)

07-10-2012 01:54 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Sorry, it's 800181 - I missed the final "1" when I copies it.

m.roth@5-cent.us wrote:
> For any redhatters on the list, I'm going to be reopening this bug today.
>
> I am also VERY unhappy with Redhat. I filed the bug months ago, and it was
> *never* assigned - no one apparently even looked at it. It's a
> show-stopper for us, since it hits us on our home directory servers.
>
> A week or so ago, I updated our test system to 6.3, and *nothing* has
> changed. Unpack a large file locally, and it's seconds. Unpack from an
> NFS-mounted directory to a local disk takes about 1.5min. NFS mount either
> an ext3 or ext4 fs, cd to that directory, and I run a job to unpack a
> large file to the NFS-mounted directory, and it's between 6.5 and 7.5
> *MINUTES*. We cannot move our home directory servers to 6.x with this
> unacknowledged ->BUG<-.
>
> Large file is defined as a 28M .gz file, unpacked to 92M.
>
> This is 100% repeatable.
>
> I tried sending an email to our support weeks ago, and got no response.
> Maybe it takes shaming in a public forum to get anyone to acknowledge this
> exists....
>
> mark
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

07-11-2012 02:38 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
For those with threaded mail tools, after it d/l pop-3 at home, I wound up
forwarding this to myself at work, then back to this account, which is
subscribed to the redhat list.

> Subject: Re: Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
> Date: Tue, 10 Jul 2012 21:47:29 -0600
> From: Corey Kovacs <corey.kovacs@gmail.com>
>
> TAM = Technical Account Manager.
>
> I asked about auditing and you asked about selinux auditing. Since when is
> selinux auditing? I mean the auditd daemon. It can tax the system severely
> if not set up correctly.

/var/log/audit/audit.log
And I *said* I was at home, and couldn't look, but yes, auditd is running.
>
> I asked about your exports file, you give me the format for a generic
> exports file. If you didn't notice, i am an RHCA. I think I know what the
> general format is.

Yeah, and I'm completely unimpressed with your RHCA. You've come on as
*sure* that it's my fault for misconfiguration, not that we might have
found a bug.
>
> I asked about kerberos, you said you didn't know.. how can you NOT know
> if you are using kerberos?
>
> I asked you to give us something to work with. You said "read the damn
> bug". I did, it's so fricking vague it's ridiculous.

Vague? Really?
>
> You seem to have very little information/knowledge of your system which
> isn't too surprising at this point.
<snip>
> On Tue, Jul 10, 2012 at 8:47 PM, mark <m.roth@5-cent.us> wrote:
<snip>
>>
>> I'll say it one more time: we found the problem on CentOS. We went to
>> our test RHEL system. Updated it. Exported a directory *from* the RHEL box
>> to itself, to /mnt/foo, and ran the test, and got the same results.
>>
>> In fact, I ran it twice today, updating the kernel in between, and with
>> 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
>> getting with 6.2
>> <snip>
>>
>> Now, all that said and done, here are some questions for you which
>> might help us figure what would help.
>>>
>>> 1. What options are present on the mount? (cat /proc/mounts, thinks
>>> like sync can be a problem)

/scratch/foo <same_server>(rw,sync,no_wdelay)

<snip>
>> 2. What does your /etc/exports config look like on your server node
>> (cat /etc/exports)

As if I don't know what/how to tell you that? See above.
<snip>
>> 3. You are using NFSv4, are you using Kerberos with it?

No.
>>
>> I don't believe we have kerborous set with NFS. We do use it for other
>> things.
>>
>>
>> 3.a. If so, what mode are you using for your gss/krb flag? (krb5,
>>> krb5i, krb5p)
>>> 4. What's your network speed? Are you sure? (ethtool ethX to make sure)
>>>
>> Gigabit.
<snip>
>> Do you mean selinux auditing? As I said, doing it on the local drive
>> takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.
Therefore,
>> there's nothing that could affect it on the one server.
>>
>> 7. How many clients are hitting your server and how many nfsd threads
>> are you running on it?
>>
>> No other clients. This is a test system.
>>
>>
>>> This is by no means an exhaustive list of things to look at.
>>>
>>> Anyway, in order to get any real help, you cannot just shout out, "My
>>> stuff is broke, it's Red Hat's fault, no one will listen to me!"
>>>
>>> Give us something to work with.

Maybe you should actually *read* everything I wrote. Screw your I'm an
RHCE, here's a real world test:

>From what I wrote:
1. Is this a) a test system, b) a system in use (prod or dev)?
2. Did it export anything before this test?
3. Is it exporting anything to any other server, or only to the same machine?
4. Did I try unpacking to a local drive?
5. Did I find the problem unpacking to an NFS mounted directory that was
mounted FROM THE SAME MACHINE?

<snip>
mark

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Allen Chen 07-26-2012 05:41 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
On 07/11/2012 10:38 AM, m.roth@5-cent.us wrote:

For those with threaded mail tools, after it d/l pop-3 at home, I wound up
forwarding this to myself at work, then back to this account, which is
subscribed to the redhat list.


Subject: Re: Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
Date: Tue, 10 Jul 2012 21:47:29 -0600
From: Corey Kovacs<corey.kovacs@gmail.com>

TAM = Technical Account Manager.

I asked about auditing and you asked about selinux auditing. Since when is
selinux auditing? I mean the auditd daemon. It can tax the system severely
if not set up correctly.

/var/log/audit/audit.log
And I *said* I was at home, and couldn't look, but yes, auditd is running.

I asked about your exports file, you give me the format for a generic
exports file. If you didn't notice, i am an RHCA. I think I know what the
general format is.

Yeah, and I'm completely unimpressed with your RHCA. You've come on as
*sure* that it's my fault for misconfiguration, not that we might have
found a bug.

I asked about kerberos, you said you didn't know.. how can you NOT know
if you are using kerberos?

I asked you to give us something to work with. You said "read the damn
bug". I did, it's so fricking vague it's ridiculous.

Vague? Really?

You seem to have very little information/knowledge of your system which
isn't too surprising at this point.

<snip>

On Tue, Jul 10, 2012 at 8:47 PM, mark<m.roth@5-cent.us> wrote:

<snip>

I'll say it one more time: we found the problem on CentOS. We went to
our test RHEL system. Updated it. Exported a directory *from* the RHEL box
to itself, to /mnt/foo, and ran the test, and got the same results.

In fact, I ran it twice today, updating the kernel in between, and with
6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
getting with 6.2
<snip>

Now, all that said and done, here are some questions for you which
might help us figure what would help.

1. What options are present on the mount? (cat /proc/mounts, thinks
like sync can be a problem)

/scratch/foo<same_server>(rw,sync,no_wdelay)

<snip>

2. What does your /etc/exports config look like on your server node
(cat /etc/exports)

As if I don't know what/how to tell you that? See above.
<snip>

3. You are using NFSv4, are you using Kerberos with it?

No.

I don't believe we have kerborous set with NFS. We do use it for other
things.


3.a. If so, what mode are you using for your gss/krb flag? (krb5,

krb5i, krb5p)
4. What's your network speed? Are you sure? (ethtool ethX to make sure)


Gigabit.

<snip>

Do you mean selinux auditing? As I said, doing it on the local drive
takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.

Therefore,

there's nothing that could affect it on the one server.

7. How many clients are hitting your server and how many nfsd threads
are you running on it?

No other clients. This is a test system.



This is by no means an exhaustive list of things to look at.

Anyway, in order to get any real help, you cannot just shout out, "My
stuff is broke, it's Red Hat's fault, no one will listen to me!"

Give us something to work with.

Maybe you should actually *read* everything I wrote. Screw your I'm an
RHCE, here's a real world test:

> From what I wrote:
1. Is this a) a test system, b) a system in use (prod or dev)?
2. Did it export anything before this test?
3. Is it exporting anything to any other server, or only to the same machine?
4. Did I try unpacking to a local drive?
5. Did I find the problem unpacking to an NFS mounted directory that was
mounted FROM THE SAME MACHINE?

<snip>
mark


I did a quick test on my CentOS 6.2, and I don't see any slow untar.
Here are the steps I did:

On server:
# uname -a
Linux backup62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux


on my desktop:
# uname -a
Linux centos62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux

# mount -t nfs server-ip:/images /mnt
# time tar xvfz /mnt/hs21.tgz
...
real 0m5.496s
user 0m0.438s
sys 0m0.176s

# cd /mnt
time tar xvfz hs21.tgz
...
real 0m20.634s
user 0m0.414s
sys 0m0.135s

# ll hs21.tgz
-rw-r--r-- 1 root root 43725908 Apr 2 2008 hs21.tgz

Is there anything you can do with the DNS settings on the server side?

Allen

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

07-26-2012 07:19 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Allen Chen wrote:
> On 07/11/2012 10:38 AM, m.roth@5-cent.us wrote:
>>> From: Corey Kovacs<corey.kovacs@gmail.com>
<snip>
>>> On Tue, Jul 10, 2012 at 8:47 PM, mark<m.roth@5-cent.us> wrote:
>> <snip>
>>>> I'll say it one more time: we found the problem on CentOS. We went to
>>>> our test RHEL system. Updated it. Exported a directory *from* the RHEL
>>>> box to itself, to /mnt/foo, and ran the test, and got the same results.
>>>>
>>>> In fact, I ran it twice today, updating the kernel in between, and
>>>> with 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
>>>> getting with 6.2
>>>> <snip>
>>>>
>>>> Now, all that said and done, here are some questions for you which
>>>> might help us figure what would help.
>>>>> 1. What options are present on the mount? (cat /proc/mounts, thinks
>>>>> like sync can be a problem)
>> /scratch/foo<same_server>(rw,sync,no_wdelay)
>>
<snip>
>>>> 2. What does your /etc/exports config look like on your server node
>>>> (cat /etc/exports)
<snip>
>>>> Do you mean selinux auditing? As I said, doing it on the local drive
>>>> takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.
>> Therefore,
>>>> there's nothing that could affect it on the one server.
>>
> I did a quick test on my CentOS 6.2, and I don't see any slow untar.
> Here are the steps I did:
>
> On server:
> # uname -a
> Linux backup62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
> i686 i686 i386 GNU/Linux
>
> on my desktop:
> # uname -a
> Linux centos62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
> i686 i686 i386 GNU/Linux
> # mount -t nfs server-ip:/images /mnt
> # time tar xvfz /mnt/hs21.tgz
> ...
> real 0m5.496s
> user 0m0.438s
> sys 0m0.176s
>
> # cd /mnt
> time tar xvfz hs21.tgz
> ...
> real 0m20.634s
> user 0m0.414s
> sys 0m0.135s
>
> # ll hs21.tgz
> -rw-r--r-- 1 root root 43725908 Apr 2 2008 hs21.tgz

Allen,

what does /etc/exports read? On my system, if I have it as

/scratch/foo <fwdn.hostname>(rw,sync,no_wdelay)

I get the delay. Following someone's post a week or two ago, I tried
/scratch/foo <fwdn.hostname>(rw,async,no_wdelay)

and it goes at a reasonable speed. That (a)sync seems to override
everything else. However, I'm trying to get together with my manager to
decide if we want to use async - that's above my pay grade, and we have to
consider that we have a fair number of users that run jobs that run for
literally days, sometimes over a week, and loss of any data at all might
mean false results, or having to rerun it.
>
> Is there anything you can do with the DNS settings on the server side?

DNS has nothing to do with the test case.

mark

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Allen Chen 07-26-2012 08:25 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
On 07/26/2012 03:19 PM, m.roth@5-cent.us wrote:

Allen Chen wrote:

On 07/11/2012 10:38 AM, m.roth@5-cent.us wrote:

From: Corey Kovacs<corey.kovacs@gmail.com>

<snip>

On Tue, Jul 10, 2012 at 8:47 PM, mark<m.roth@5-cent.us> wrote:

<snip>

I'll say it one more time: we found the problem on CentOS. We went to
our test RHEL system. Updated it. Exported a directory *from* the RHEL
box to itself, to /mnt/foo, and ran the test, and got the same results.

In fact, I ran it twice today, updating the kernel in between, and
with 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
getting with 6.2
<snip>

Now, all that said and done, here are some questions for you which
might help us figure what would help.

1. What options are present on the mount? (cat /proc/mounts, thinks
like sync can be a problem)

/scratch/foo<same_server>(rw,sync,no_wdelay)


<snip>

2. What does your /etc/exports config look like on your server node
(cat /etc/exports)

<snip>

Do you mean selinux auditing? As I said, doing it on the local drive
takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.

Therefore,

there's nothing that could affect it on the one server.

I did a quick test on my CentOS 6.2, and I don't see any slow untar.
Here are the steps I did:

On server:
# uname -a
Linux backup62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux

on my desktop:
# uname -a
Linux centos62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux
# mount -t nfs server-ip:/images /mnt
# time tar xvfz /mnt/hs21.tgz
...
real 0m5.496s
user 0m0.438s
sys 0m0.176s

# cd /mnt
time tar xvfz hs21.tgz
...
real 0m20.634s
user 0m0.414s
sys 0m0.135s

# ll hs21.tgz
-rw-r--r-- 1 root root 43725908 Apr 2 2008 hs21.tgz

Allen,

what does /etc/exports read? On my system, if I have it as

/scratch/foo<fwdn.hostname>(rw,sync,no_wdelay)

I get the delay. Following someone's post a week or two ago, I tried
/scratch/foo<fwdn.hostname>(rw,async,no_wdelay)

and it goes at a reasonable speed. That (a)sync seems to override
everything else. However, I'm trying to get together with my manager to
decide if we want to use async - that's above my pay grade, and we have to
consider that we have a fair number of users that run jobs that run for
literally days, sometimes over a week, and loss of any data at all might
mean false results, or having to rerun it.

Is there anything you can do with the DNS settings on the server side?

DNS has nothing to do with the test case.

mark


# cat /etc/exports
/images 192.168.1.0/24(rw) 10.235.4.2/32(rw)

Allen

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

mark 07-26-2012 11:29 PM

Bug 800181: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
On 07/26/12 16:25, Allen Chen wrote:

On 07/26/2012 03:19 PM, m.roth@5-cent.us wrote:

Allen Chen wrote:

On 07/11/2012 10:38 AM, m.roth@5-cent.us wrote:

From: Corey Kovacs<corey.kovacs@gmail.com>

<snip>

On Tue, Jul 10, 2012 at 8:47 PM, mark<m.roth@5-cent.us> wrote:

<snip>

I'll say it one more time: we found the problem on CentOS. We went to
our test RHEL system. Updated it. Exported a directory *from* the
RHEL
box to itself, to /mnt/foo, and ran the test, and got the same
results.

In fact, I ran it twice today, updating the kernel in between, and
with 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we
were
getting with 6.2
<snip>

Now, all that said and done, here are some questions for you which
might help us figure what would help.

1. What options are present on the mount? (cat /proc/mounts, thinks
like sync can be a problem)

/scratch/foo<same_server>(rw,sync,no_wdelay)


<snip>

2. What does your /etc/exports config look like on your server node
(cat /etc/exports)

<snip>

Do you mean selinux auditing? As I said, doing it on the local drive
takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.

Therefore,

there's nothing that could affect it on the one server.

I did a quick test on my CentOS 6.2, and I don't see any slow untar.
Here are the steps I did:

On server:
# uname -a
Linux backup62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux

on my desktop:
# uname -a
Linux centos62 2.6.32-220.el6.i686 #1 SMP Tue Dec 6 16:15:40 GMT 2011
i686 i686 i386 GNU/Linux
# mount -t nfs server-ip:/images /mnt
# time tar xvfz /mnt/hs21.tgz
...
real 0m5.496s
user 0m0.438s
sys 0m0.176s

# cd /mnt
time tar xvfz hs21.tgz
...
real 0m20.634s
user 0m0.414s
sys 0m0.135s

# ll hs21.tgz
-rw-r--r-- 1 root root 43725908 Apr 2 2008 hs21.tgz

Allen,

what does /etc/exports read? On my system, if I have it as

/scratch/foo<fwdn.hostname>(rw,sync,no_wdelay)

I get the delay. Following someone's post a week or two ago, I tried
/scratch/foo<fwdn.hostname>(rw,async,no_wdelay)

and it goes at a reasonable speed. That (a)sync seems to override
everything else. However, I'm trying to get together with my manager to
decide if we want to use async - that's above my pay grade, and we
have to
consider that we have a fair number of users that run jobs that run for
literally days, sometimes over a week, and loss of any data at all might
mean false results, or having to rerun it.

Is there anything you can do with the DNS settings on the server side?

DNS has nothing to do with the test case.


# cat /etc/exports
/images 192.168.1.0/24(rw) 10.235.4.2/32(rw)


Since you're not specifying explicitly, I suspect it's doing async.

mark


--
Why the Libertarian idea of a privatized courts won't work,
from a slogan from Kenya: why bother hiring a lawyer, when
you can buy a judge?

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list


All times are GMT. The time now is 06:07 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.