Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   Red Hat Linux (http://www.linux-archive.org/red-hat-linux/)
-   -   Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7 (http://www.linux-archive.org/red-hat-linux/682012-bug-80018-nfsv4-rhel-6-2-over-six-times-slower-than-5-7-a.html)

07-10-2012 01:46 PM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
For any redhatters on the list, I'm going to be reopening this bug today.

I am also VERY unhappy with Redhat. I filed the bug months ago, and it was
*never* assigned - no one apparently even looked at it. It's a
show-stopper for us, since it hits us on our home directory servers.

A week or so ago, I updated our test system to 6.3, and *nothing* has
changed. Unpack a large file locally, and it's seconds. Unpack from an
NFS-mounted directory to a local disk takes about 1.5min. NFS mount either
an ext3 or ext4 fs, cd to that directory, and I run a job to unpack a
large file to the NFS-mounted directory, and it's between 6.5 and 7.5
*MINUTES*. We cannot move our home directory servers to 6.x with this
unacknowledged ->BUG<-.

Large file is defined as a 28M .gz file, unpacked to 92M.

This is 100% repeatable.

I tried sending an email to our support weeks ago, and got no response.
Maybe it takes shaming in a public forum to get anyone to acknowledge this
exists....

mark

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Corey Kovacs 07-10-2012 02:57 PM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Maybe it takes posting your config to the list so we can see if you made a
mistake. Are you a paying customer actually using RHEL? If this a real bug
I doubt it would be so ignored. Anyone else having a similar problem?

Lets start by being sensible and not antagonistic.

C
On Jul 10, 2012 7:52 AM, <m.roth@5-cent.us> wrote:

> For any redhatters on the list, I'm going to be reopening this bug today.
>
> I am also VERY unhappy with Redhat. I filed the bug months ago, and it was
> *never* assigned - no one apparently even looked at it. It's a
> show-stopper for us, since it hits us on our home directory servers.
>
> A week or so ago, I updated our test system to 6.3, and *nothing* has
> changed. Unpack a large file locally, and it's seconds. Unpack from an
> NFS-mounted directory to a local disk takes about 1.5min. NFS mount either
> an ext3 or ext4 fs, cd to that directory, and I run a job to unpack a
> large file to the NFS-mounted directory, and it's between 6.5 and 7.5
> *MINUTES*. We cannot move our home directory servers to 6.x with this
> unacknowledged ->BUG<-.
>
> Large file is defined as a 28M .gz file, unpacked to 92M.
>
> This is 100% repeatable.
>
> I tried sending an email to our support weeks ago, and got no response.
> Maybe it takes shaming in a public forum to get anyone to acknowledge this
> exists....
>
> mark
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

07-10-2012 03:47 PM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Corey Kovacs wrote:
> Maybe it takes posting your config to the list so we can see if you made a

The config's the same for 5.8 and 6.3, I think. I do know my manager
worked on it some, and had done some work with NFS 4 configuration.

I can check with my manager, but I believe that we'd be willing to send
that to RHEL; I don't know that he'd want it posted publicly, for security
reasons (but I could be wrong; I'll ask him).

> mistake. Are you a paying customer actually using RHEL? If this a real bug
> I doubt it would be so ignored. Anyone else having a similar problem?

This is 100% reproducible. A user found it on CentOS 6.2; we tried it on
several servers, then went to an RHEL one. All identical responses.
>
> Lets start by being sensible and not antagonistic.

At what point would you decide to "allow" me to be antagonistic?

Let me restate my case: I opened a bugzilla case under my own account.
After weeks, I used the email address for support: yes, we *do* have a
contract with RHEL, and we are a US federal gov't agency. It's been
something like two weeks, and I've not seen a response.

The *only* thing that happened on the bug was they waited about a month,
then there was a followup, if you read it, that they were on 6.3, and so
they put it to dontfix.

*That's* why I'm posting all over. This is the *first* responses I've
gotten. I guess it *does* take public humiliation.

mark
>
> C
> On Jul 10, 2012 7:52 AM, <m.roth@5-cent.us> wrote:
>
>> For any redhatters on the list, I'm going to be reopening this bug
>> today.
>>
>> I am also VERY unhappy with Redhat. I filed the bug months ago, and it
>> was
>> *never* assigned - no one apparently even looked at it. It's a
>> show-stopper for us, since it hits us on our home directory servers.
>>
>> A week or so ago, I updated our test system to 6.3, and *nothing* has
>> changed. Unpack a large file locally, and it's seconds. Unpack from an
>> NFS-mounted directory to a local disk takes about 1.5min. NFS mount
>> either
>> an ext3 or ext4 fs, cd to that directory, and I run a job to unpack a
>> large file to the NFS-mounted directory, and it's between 6.5 and 7.5
>> *MINUTES*. We cannot move our home directory servers to 6.x with this
>> unacknowledged ->BUG<-.
>>
>> Large file is defined as a 28M .gz file, unpacked to 92M.
>>
>> This is 100% repeatable.
>>
>> I tried sending an email to our support weeks ago, and got no response.
>> Maybe it takes shaming in a public forum to get anyone to acknowledge
>> this
>> exists....
>>
>> mark
>>
>> --
>> redhat-list mailing list
>> unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
>> https://www.redhat.com/mailman/listinfo/redhat-list
>>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

07-10-2012 03:48 PM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Corey Kovacs wrote:
> Maybe it takes posting your config to the list so we can see if you made a

The config's the same for 5.8 and 6.3, I think. I do know my manager
worked on it some, and had done some work with NFS 4 configuration.

I can check with my manager, but I believe that we'd be willing to send
that to RHEL; I don't know that he'd want it posted publicly, for security
reasons (but I could be wrong; I'll ask him).

> mistake. Are you a paying customer actually using RHEL? If this a real bug
> I doubt it would be so ignored. Anyone else having a similar problem?

This is 100% reproducible. A user found it on CentOS 6.2; we tried it on
several servers, then went to an RHEL one. All identical responses.
>
> Lets start by being sensible and not antagonistic.

At what point would you decide to "allow" me to be antagonistic?

Let me restate my case: I opened a bugzilla case under my own account.
After weeks, I used the email address for support: yes, we *do* have a
contract with RHEL, and we are a US federal gov't agency. It's been
something like two weeks, and I've not seen a response.

The *only* thing that happened on the bug was they waited about a month,
then there was a followup, if you read it, that they were on 6.3, and so
they put it to dontfix.

*That's* why I'm posting all over. This is the *first* responses I've
gotten. I guess it *does* take public humiliation.

mark


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Corey Kovacs 07-11-2012 12:41 AM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Well, looking at the Bugzilla, it looks like they are asking you to contact
your support rep. If you are working in the government, there will be a TAM
assigned to handle these problems. If you are a contractor working on
systems to be delivered to the government, then you need to be a paying
customer to expect support from Red Hat. Especially when your original
problem manifested on CentOS. Granted, it's the same code base, but that
doesn't make your problem on CentOS, a problem for Red Hat.

As far as when you get to be antagonistic goes, never if you want to be
professional about it....

Now, all that said and done, here are some questions for you which might
help us figure what would help.

1. What options are present on the mount? (cat /proc/mounts, thinks like
sync can be a problem)
2. What does your /etc/exports config look like on your server node (cat
/etc/exports)
3. You are using NFSv4, are you using Kerberos with it?
3.a. If so, what mode are you using for your gss/krb flag? (krb5,
krb5i, krb5p)
4. What's your network speed? Are you sure? (ethtool ethX to make sure)
5. Selinux?
6. Auditing?
7. How many clients are hitting your server and how many nfsd threads are
you running on it?

This is by no means an exhaustive list of things to look at.

Anyway, in order to get any real help, you cannot just shout out, "My stuff
is broke, it's Red Hat's fault, no one will listen to me!"

Give us something to work with.

By the way, in looking at the responses to your bugzilla, it doesn't look
to me like they were shamed into responding. It looks like are telling you
to go through proper channels if they exist. full stop.

Ideally, you'll get this resolved using your assigned TAM etc, however I am
sure others as well as I are happy to help if you are willing to be
professional about it.


Corey
RHCA #*110-541-489*



On Tue, Jul 10, 2012 at 9:48 AM, <m.roth@5-cent.us> wrote:

> Corey Kovacs wrote:
> > Maybe it takes posting your config to the list so we can see if you made
> a
>
> The config's the same for 5.8 and 6.3, I think. I do know my manager
> worked on it some, and had done some work with NFS 4 configuration.
>
> I can check with my manager, but I believe that we'd be willing to send
> that to RHEL; I don't know that he'd want it posted publicly, for security
> reasons (but I could be wrong; I'll ask him).
>
> > mistake. Are you a paying customer actually using RHEL? If this a real
> bug
> > I doubt it would be so ignored. Anyone else having a similar problem?
>
> This is 100% reproducible. A user found it on CentOS 6.2; we tried it on
> several servers, then went to an RHEL one. All identical responses.
> >
> > Lets start by being sensible and not antagonistic.
>
> At what point would you decide to "allow" me to be antagonistic?
>
> Let me restate my case: I opened a bugzilla case under my own account.
> After weeks, I used the email address for support: yes, we *do* have a
> contract with RHEL, and we are a US federal gov't agency. It's been
> something like two weeks, and I've not seen a response.
>
> The *only* thing that happened on the bug was they waited about a month,
> then there was a followup, if you read it, that they were on 6.3, and so
> they put it to dontfix.
>
> *That's* why I'm posting all over. This is the *first* responses I've
> gotten. I guess it *does* take public humiliation.
>
> mark
>
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

mark 07-11-2012 02:47 AM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
On 07/10/12 20:41, Corey Kovacs wrote:

Well, looking at the Bugzilla, it looks like they are asking you to contact
your support rep. If you are working in the government, there will be a TAM


I'm waiting for my manager to tell me how to file. I have no idea what a
TAM is - we do all this ourselves.



assigned to handle these problems. If you are a contractor working on
systems to be delivered to the government, then you need to be a paying
customer to expect support from Red Hat. Especially when your original
problem manifested on CentOS. Granted, it's the same code base, but that
doesn't make your problem on CentOS, a problem for Red Hat.


I'll say it one more time: we found the problem on CentOS. We went to
our test RHEL system. Updated it. Exported a directory *from* the RHEL
box to itself, to /mnt/foo, and ran the test, and got the same results.


In fact, I ran it twice today, updating the kernel in between, and with
6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were
getting with 6.2

<snip>

Now, all that said and done, here are some questions for you which might
help us figure what would help.

1. What options are present on the mount? (cat /proc/mounts, thinks like
sync can be a problem)


I"m not at work. I'll have to answer that in the morning. I will tell
you that when we were first trying to figure it out, two months ago, I
did try no sync.



2. What does your /etc/exports config look like on your server node (cat
/etc/exports)


/scratch/foo <servername>: options


3. You are using NFSv4, are you using Kerberos with it?


I don't believe we have kerborous set with NFS. We do use it for other
things.



3.a. If so, what mode are you using for your gss/krb flag? (krb5,
krb5i, krb5p)
4. What's your network speed? Are you sure? (ethtool ethX to make sure)


Gigabit.


5. Selinux?

Permissive.


6. Auditing?


Do you mean selinux auditing? As I said, doing it on the local drive
takes seconds. Doing it from a 5.x NFS server takes about 1.5 min.
Therefore, there's nothing that could affect it on the one server.



7. How many clients are hitting your server and how many nfsd threads are
you running on it?


No other clients. This is a test system.


This is by no means an exhaustive list of things to look at.

Anyway, in order to get any real help, you cannot just shout out, "My stuff
is broke, it's Red Hat's fault, no one will listen to me!"

Give us something to work with.


Try reading the damn bug.



By the way, in looking at the responses to your bugzilla, it doesn't look
to me like they were shamed into responding. It looks like are telling you
to go through proper channels if they exist. full stop.


No, they gave *ZERO* responses until today. The one and only response
before today was, "oh, we're up to 6.3, we'll not even look at it".


And my manager, who's a fed, and I, a contractor, along with the other
admin under him, who is also a contractor, handle the licenses, etc, so
there's no one else to wait for.


mark

--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Corey Kovacs 07-11-2012 03:47 AM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
TAM = Technical Account Manager.

I asked about auditing and you asked about selinux auditing. Since when is
selinux auditing? I mean the auditd daemon. It can tax the system severely
if not set up correctly.

I asked about your exports file, you give me the format for a generic
exports file. If you didn't notice, i am an RHCA. I think I know what the
general format is.

I asked about kerberos, you said you didn't know.. how can you NOT know if
you are using kerberos?

I asked you to give us something to work with. You said "read the damn
bug". I did, it's so fricking vague it's ridiculous.

You seem to have very little information/knowledge of your system which
isn't too surprising at this point.

So, you seem to have decided to be very unproessional about this, so I say
good luck. I will not respond to you again. I can only hope others don't
either.


Have fun....


On Tue, Jul 10, 2012 at 8:47 PM, mark <m.roth@5-cent.us> wrote:

> On 07/10/12 20:41, Corey Kovacs wrote:
>
>> Well, looking at the Bugzilla, it looks like they are asking you to
>> contact
>> your support rep. If you are working in the government, there will be a
>> TAM
>>
>
> I'm waiting for my manager to tell me how to file. I have no idea what a
> TAM is - we do all this ourselves.
>
>
> assigned to handle these problems. If you are a contractor working on
>> systems to be delivered to the government, then you need to be a paying
>> customer to expect support from Red Hat. Especially when your original
>> problem manifested on CentOS. Granted, it's the same code base, but that
>> doesn't make your problem on CentOS, a problem for Red Hat.
>>
>
> I'll say it one more time: we found the problem on CentOS. We went to our
> test RHEL system. Updated it. Exported a directory *from* the RHEL box to
> itself, to /mnt/foo, and ran the test, and got the same results.
>
> In fact, I ran it twice today, updating the kernel in between, and with
> 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were getting
> with 6.2
> <snip>
>
> Now, all that said and done, here are some questions for you which might
>> help us figure what would help.
>>
>> 1. What options are present on the mount? (cat /proc/mounts, thinks like
>> sync can be a problem)
>>
>
> I"m not at work. I'll have to answer that in the morning. I will tell you
> that when we were first trying to figure it out, two months ago, I did try
> no sync.
>
>
> 2. What does your /etc/exports config look like on your server node (cat
>> /etc/exports)
>>
>
> /scratch/foo <servername>: options
>
>
> 3. You are using NFSv4, are you using Kerberos with it?
>>
>
> I don't believe we have kerborous set with NFS. We do use it for other
> things.
>
>
> 3.a. If so, what mode are you using for your gss/krb flag? (krb5,
>> krb5i, krb5p)
>> 4. What's your network speed? Are you sure? (ethtool ethX to make sure)
>>
>
> Gigabit.
>
> 5. Selinux?
>>
> Permissive.
>
> 6. Auditing?
>>
>
> Do you mean selinux auditing? As I said, doing it on the local drive takes
> seconds. Doing it from a 5.x NFS server takes about 1.5 min. Therefore,
> there's nothing that could affect it on the one server.
>
>
> 7. How many clients are hitting your server and how many nfsd threads are
>> you running on it?
>>
>
> No other clients. This is a test system.
>
>
>> This is by no means an exhaustive list of things to look at.
>>
>> Anyway, in order to get any real help, you cannot just shout out, "My
>> stuff
>> is broke, it's Red Hat's fault, no one will listen to me!"
>>
>> Give us something to work with.
>>
>
> Try reading the damn bug.
>
>
>
>> By the way, in looking at the responses to your bugzilla, it doesn't look
>> to me like they were shamed into responding. It looks like are telling you
>> to go through proper channels if they exist. full stop.
>>
>
> No, they gave *ZERO* responses until today. The one and only response
> before today was, "oh, we're up to 6.3, we'll not even look at it".
>
> And my manager, who's a fed, and I, a contractor, along with the other
> admin under him, who is also a contractor, handle the licenses, etc, so
> there's no one else to wait for.
>
>
> mark
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@**redhat.com<redhat-list-request@redhat.com>
> ?subject=unsubscribe
> https://www.redhat.com/**mailman/listinfo/redhat-list<https://www.redhat.com/mailman/listinfo/redhat-list>
>
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Tom Curl 07-11-2012 10:49 AM

Bug 80018: NFSv4 on RHEL 6.2 over six times slower than 5.7
 
Good answer Corey!

On Tue, 2012-07-10 at 21:47 -0600, Corey Kovacs wrote:
> TAM = Technical Account Manager.
>
> I asked about auditing and you asked about selinux auditing. Since when is
> selinux auditing? I mean the auditd daemon. It can tax the system severely
> if not set up correctly.
>
> I asked about your exports file, you give me the format for a generic
> exports file. If you didn't notice, i am an RHCA. I think I know what the
> general format is.
>
> I asked about kerberos, you said you didn't know.. how can you NOT know if
> you are using kerberos?
>
> I asked you to give us something to work with. You said "read the damn
> bug". I did, it's so fricking vague it's ridiculous.
>
> You seem to have very little information/knowledge of your system which
> isn't too surprising at this point.
>
> So, you seem to have decided to be very unproessional about this, so I say
> good luck. I will not respond to you again. I can only hope others don't
> either.
>
>
> Have fun....
>
>
> On Tue, Jul 10, 2012 at 8:47 PM, mark <m.roth@5-cent.us> wrote:
>
> > On 07/10/12 20:41, Corey Kovacs wrote:
> >
> >> Well, looking at the Bugzilla, it looks like they are asking you to
> >> contact
> >> your support rep. If you are working in the government, there will be a
> >> TAM
> >>
> >
> > I'm waiting for my manager to tell me how to file. I have no idea what a
> > TAM is - we do all this ourselves.
> >
> >
> > assigned to handle these problems. If you are a contractor working on
> >> systems to be delivered to the government, then you need to be a paying
> >> customer to expect support from Red Hat. Especially when your original
> >> problem manifested on CentOS. Granted, it's the same code base, but that
> >> doesn't make your problem on CentOS, a problem for Red Hat.
> >>
> >
> > I'll say it one more time: we found the problem on CentOS. We went to our
> > test RHEL system. Updated it. Exported a directory *from* the RHEL box to
> > itself, to /mnt/foo, and ran the test, and got the same results.
> >
> > In fact, I ran it twice today, updating the kernel in between, and with
> > 6.3, it's taking a consistent 7.5 min, instead of the 6.5 we were getting
> > with 6.2
> > <snip>
> >
> > Now, all that said and done, here are some questions for you which might
> >> help us figure what would help.
> >>
> >> 1. What options are present on the mount? (cat /proc/mounts, thinks like
> >> sync can be a problem)
> >>
> >
> > I"m not at work. I'll have to answer that in the morning. I will tell you
> > that when we were first trying to figure it out, two months ago, I did try
> > no sync.
> >
> >
> > 2. What does your /etc/exports config look like on your server node (cat
> >> /etc/exports)
> >>
> >
> > /scratch/foo <servername>: options
> >
> >
> > 3. You are using NFSv4, are you using Kerberos with it?
> >>
> >
> > I don't believe we have kerborous set with NFS. We do use it for other
> > things.
> >
> >
> > 3.a. If so, what mode are you using for your gss/krb flag? (krb5,
> >> krb5i, krb5p)
> >> 4. What's your network speed? Are you sure? (ethtool ethX to make sure)
> >>
> >
> > Gigabit.
> >
> > 5. Selinux?
> >>
> > Permissive.
> >
> > 6. Auditing?
> >>
> >
> > Do you mean selinux auditing? As I said, doing it on the local drive takes
> > seconds. Doing it from a 5.x NFS server takes about 1.5 min. Therefore,
> > there's nothing that could affect it on the one server.
> >
> >
> > 7. How many clients are hitting your server and how many nfsd threads are
> >> you running on it?
> >>
> >
> > No other clients. This is a test system.
> >
> >
> >> This is by no means an exhaustive list of things to look at.
> >>
> >> Anyway, in order to get any real help, you cannot just shout out, "My
> >> stuff
> >> is broke, it's Red Hat's fault, no one will listen to me!"
> >>
> >> Give us something to work with.
> >>
> >
> > Try reading the damn bug.
> >
> >
> >
> >> By the way, in looking at the responses to your bugzilla, it doesn't look
> >> to me like they were shamed into responding. It looks like are telling you
> >> to go through proper channels if they exist. full stop.
> >>
> >
> > No, they gave *ZERO* responses until today. The one and only response
> > before today was, "oh, we're up to 6.3, we'll not even look at it".
> >
> > And my manager, who's a fed, and I, a contractor, along with the other
> > admin under him, who is also a contractor, handle the licenses, etc, so
> > there's no one else to wait for.
> >
> >
> > mark
> >
> > --
> > redhat-list mailing list
> > unsubscribe mailto:redhat-list-request@**redhat.com<redhat-list-request@redhat.com>
> > ?subject=unsubscribe
> > https://www.redhat.com/**mailman/listinfo/redhat-list<https://www.redhat.com/mailman/listinfo/redhat-list>
> >


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list


All times are GMT. The time now is 03:18 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.