FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora User

 
 
LinkBack Thread Tools
 
Old 09-28-2010, 05:39 PM
JD
 
Default NFS Buffering

On 09/28/2010 07:07 AM, Simon Andrews wrote:
> I have a fedora 13 box on which I have a remote mounted nfs share over a
> fairly slow (10Mb/s) link. I'm then transferring data onto this share
> from a different machine using scp.
>
> The problem is that after scp reports that it's 100% complete the
> program will hang for ~20 mins before it will move on to another file.
> At this point it can't be killed.
>
> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still being written).
>
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
>
> Thanks
>
> Simon.
>
I understand you mean your desktop nfs-mounts a directory
exported to your machine by a server over a 10 mb/s link.
I understaand you mean that you have write permissions to
write to this nfs mounted directory on your machine.
I understand that you scp from a third machine (again at 10 mb/s ???? -
you did not specify this part)
to the nfs mounted directory on YOUR machine.

Just to get the data from the third machine to yours before it
is even sent to the nfs server):

On a 10megabit/s link, only 80% of which is payload data,
transferring 2GB (I assume you mean binary GB):
.80 * 10000000 = 8000000 bits/s payload data
8000000 / 8 = 100000 bytes/s payload data
2147483648 / 1000000 = 2147.48 number of seconds it takes to download 2 GB.
2147.48 /60.00 = 35.79 minutes to download 2GB.
So, that's close enough to your 20 minutes download time.
I am being a bit pessimistic here as to how much of the ether bandwirdth
is used
for payload data.
At 90%, the transfer time comes down to 31 minutes.

So, 20 minutes is absolutely miraculous!!!
Be HAPPY!!







--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-28-2010, 06:58 PM
Kwan Lowe
 
Default NFS Buffering

On Tue, Sep 28, 2010 at 1:39 PM, JD <jd1008@gmail.com> wrote:

> At 90%, the transfer time comes down to 31 minutes.
>
> So, 20 minutes is absolutely miraculous!!!
> Be HAPPY!!

Could be compression on the SCP link...
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-28-2010, 08:31 PM
JD
 
Default NFS Buffering

On 09/28/2010 11:58 AM, Kwan Lowe wrote:
> On Tue, Sep 28, 2010 at 1:39 PM, JD<jd1008@gmail.com> wrote:
>
>> At 90%, the transfer time comes down to 31 minutes.
>>
>> So, 20 minutes is absolutely miraculous!!!
>> Be HAPPY!!
> Could be compression on the SCP link..
Hmm... doubtful!
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 01:26 AM
Samuel Kidman
 
Default NFS Buffering

On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews <simon.andrews@bbsrc.ac.uk> wrote:

I have a fedora 13 box on which I have a remote mounted nfs share over a

fairly slow (10Mb/s) link. *I'm then transferring data onto this share

from a different machine using scp.



The problem is that after scp reports that it's 100% complete the

program will hang for ~20 mins before it will move on to another file.

At this point it can't be killed.



It looks like the nfs daemon is caching write data (around 2GB of it)

which lets scp think its finished when actually there's loads of data

sitting in a write buffer. *The hanging is presumably the time it takes

to flush the buffer (there is a process called nfsiod which is active

during this time and df shows data is still being written).



Does anyone know how to either make this buffer smaller, or get rid of

it all together so the scp can accruately report on its progress?



Thanks



Simon.



--

users mailing list

users@lists.fedoraproject.org

To unsubscribe or change subscription options:

https://admin.fedoraproject.org/mailman/listinfo/users

Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines


This Article contains information on how to adjust the buffer size of NFS and optimise file transfers. Also scp has a -C option to enable compression.

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 02:19 AM
JD
 
Default NFS Buffering

On 09/28/2010 06:26 PM, Samuel Kidman wrote:
>
>
> On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews
> <simon.andrews@bbsrc.ac.uk <mailto:simon.andrews@bbsrc.ac.uk>> wrote:
>
> I have a fedora 13 box on which I have a remote mounted nfs share
> over a
> fairly slow (10Mb/s) link. I'm then transferring data onto this share
> from a different machine using scp.
>
> The problem is that after scp reports that it's 100% complete the
> program will hang for ~20 mins before it will move on to another file.
> At this point it can't be killed.
>
> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it
> takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still being written).
>
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
>
> Thanks
>
> Simon.
>
Hey! Simon,
Listen: buffering is done by the filesystem internals
in collaboration with the block io layer. Once the filesystem
commits the write to block io layer, the write call returns to
the calling program, and there is not an iota you can do about
it! In the case of nfs, buffering is done by the nfsiod.
Buffering will be done at both the server AND the client.
This is especially noticeable when the nfs client writes onto
and nfs mounted filesystem.
nfsiod is the "helper" kernel thead. There will be as many of these
as the admin configures the system for. Ditto with the main dispatcher,
the nfsd process.
The nfsiod is what buffers writes on the client side.

If you want the scp to function more synchronously, you
need to rewrite scp, so that it calls fsync after each write!
This will force scp process to wait for the data to be flushed
before the write call returns.

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 02:21 AM
JD
 
Default NFS Buffering

On 09/28/2010 06:26 PM, Samuel Kidman wrote:
>
>
> On Tue, Sep 28, 2010 at 10:07 PM, Simon Andrews
> <simon.andrews@bbsrc.ac.uk <mailto:simon.andrews@bbsrc.ac.uk>> wrote:
>
> I have a fedora 13 box on which I have a remote mounted nfs share
> over a
> fairly slow (10Mb/s) link. I'm then transferring data onto this share
> from a different machine using scp.
>
> The problem is that after scp reports that it's 100% complete the
> program will hang for ~20 mins before it will move on to another file.
> At this point it can't be killed.
>
> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it
> takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still being written).
>
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
>
> Thanks
>
> Simon.
>
Hey! Simon,

you could also re-write scp to open the file descriptor with
O_SYNC to force all writes to be synchronous, and you will
obviate the need to call fsync().
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 08:55 AM
JB
 
Default NFS Buffering

Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:

>
> I have a fedora 13 box on which I have a remote mounted nfs share over a
> fairly slow (10Mb/s) link. I'm then transferring data onto this share
> from a different machine using scp.
>
> The problem is that after scp reports that it's 100% complete the
> program will hang for ~20 mins before it will move on to another file.
> At this point it can't be killed.
>
> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still being written).
>
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?
>
> Thanks
>
> Simon.
>

Hi Simon,

you know, we are Linux users from Missouri USA here ...
Can you tell us what system your nfs server is installed on, how its exported
nfs shares are configured ?
Then we can gain some valuable clues regarding performance of all factors
involved.

JB


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 09:03 AM
Simon Andrews
 
Default NFS Buffering

On 29/09/2010 03:19, JD wrote:
>
>
>> <simon.andrews@bbsrc.ac.uk<mailto:simon.andrews@bb src.ac.uk>> wrote:
>> The problem is that after scp reports that it's 100% complete the
>> program will hang for ~20 mins before it will move on to another file.
>> At this point it can't be killed.

> Hey! Simon,
> Listen: buffering is done by the filesystem internals
> in collaboration with the block io layer. Once the filesystem
> commits the write to block io layer, the write call returns to
> the calling program, and there is not an iota you can do about
> it!

I think I get the general process by which the caching happens, and I'm
not necessarily aiming to get rid of it, just set the cache size to
something which is appropriate for the speed of the link I'm operating over.


> In the case of nfs, buffering is done by the nfsiod.
> Buffering will be done at both the server AND the client.
> This is especially noticeable when the nfs client writes onto
> and nfs mounted filesystem.
> nfsiod is the "helper" kernel thead. There will be as many of these
> as the admin configures the system for.

OK, so I'm the admin on the client system. How to I configure nfsiod?


> Ditto with the main dispatcher,
> the nfsd process.
> The nfsiod is what buffers writes on the client side.

Which means that that is the part which is causing my problems. The
main problem is the disparity between the size of the cache on the
client (somewhere around 2GB) and the speed of transfer onto the NFS
mounted share (around 2MB/s). This means that every time the cache
needs to be flushed there is a ~20min wait during which the client is
completely blocked (so no chance to kill it or interact with it in any
way). Give me a 5MB cache and I'm a happy man!


> If you want the scp to function more synchronously, you
> need to rewrite scp, so that it calls fsync after each write!
> This will force scp process to wait for the data to be flushed
> before the write call returns.

I don't need to be that draconian, just configure a sensible cache size.
I just can't see where I can set that.

Thanks for any advice

Simon.




--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 09:50 AM
Simon Andrews
 
Default NFS Buffering

On 29/09/2010 09:55, JB wrote:
> Simon Andrews<simon.andrews<at> bbsrc.ac.uk> writes:
> Hi Simon,
>
> you know, we are Linux users from Missouri USA here ...
> Can you tell us what system your nfs server is installed on, how its exported
> nfs shares are configured ?
> Then we can gain some valuable clues regarding performance of all factors
> involved.

The server is an exported NFS share from a NetApp filer. Since the
server isn't maintained by us I can't provide specifics about how it's
configured.

The limiting factor seems to be the speed of the link between our client
and the server. We're actually operating pretty close to the available
wire speed so I don't think there's a problem with the setup in that
respect.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-29-2010, 11:19 AM
JB
 
Default NFS Buffering

Simon Andrews <simon.andrews <at> bbsrc.ac.uk> writes:

> ...

Hi,
thanks.

> It looks like the nfs daemon is caching write data (around 2GB of it)
> which lets scp think its finished when actually there's loads of data
> sitting in a write buffer. The hanging is presumably the time it takes
> to flush the buffer (there is a process called nfsiod which is active
> during this time and df shows data is still be

Your nfs client is a Fedora 13.
Can you tell me what nfsiod is and where is came from ?
Can you see it ?
$ ps aux | grep -i nfsiod


JB


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 

Thread Tools




All times are GMT. The time now is 10:36 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org