FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Fedora User

 
 
LinkBack Thread Tools
 
Old 09-30-2010, 12:21 AM
"Wolfgang S. Rupprecht"
 
Default NFS Buffering

Simon Andrews <simon.andrews@bbsrc.ac.uk> writes:
> Does anyone know how to either make this buffer smaller, or get rid of
> it all together so the scp can accruately report on its progress?

Why not cut to the chase and run scp to the remote server. TCP will do
a wonderful job of filling your pipe, but not too much.

-wolfgang
--
Wolfgang S. Rupprecht http://www.wsrcc.com/wolfgang/ (IPv6-only)
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 01:10 AM
Tom Horsley
 
Default NFS Buffering

On Wed, 29 Sep 2010 17:21:25 -0700
Wolfgang S. Rupprecht wrote:

> Why not cut to the chase and run scp to the remote server. TCP will do
> a wonderful job of filling your pipe, but not too much.

I see long delays with scp between two different systems talking
to ext3 filesystems on both ends. The scp command will say 100%,
but not actually exit, and the network monitor applet will show
lots of network traffic for quite a while. When the network traffic
dies down, the scp command finally exits.

I always figured it must be caching lots of packets in the 8 gig
of memory I have.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 02:25 AM
JB
 
Default NFS Buffering

JB <jb.1234abcd <at> gmail.com> writes:

> ...

There are other things to consider.

$ man 5 exports
...
/etc/exports
...
General Options
...
async ...
sync ...

$ man 5 nfs
...
/etc/fstab
...
Valid options for either the nfs or nfs4 file system type
...
wsize=n The maximum number of bytes per network WRITE request
that the NFS client can send when writing data to a file
on an NFS server. ...
...
ac / noac Selects whether the client may cache file attributes.
...
Editor's Note:
The noac option is a mixture of a generic option, sync, and
an NFS-specific option actimeo=0. So it causes a significant performance
penalty.
But do not confuse file attributes caching with data caching.
...
The sync mount option
The NFS client treats the sync mount option differently than some other
file systems (refer to mount(8) for a description of the generic sync
and async mount options). If neither sync nor async is specified (or
if the async option is specified), the NFS client delays sending appli-
cation writes to the server until any of these events occur:

Memory pressure forces reclamation of system memory resources.

An application flushes file data explicitly with sync(2),
msync(2), or fsync(3).

An application closes a file with close(2).

The file is locked/unlocked via fcntl(2).

In other words, under normal circumstances, data written by an applica-
tion may not immediately appear on the server that hosts the file.

If the sync option is specified on a mount point, any system call that
writes data to files on that mount point causes that data to be flushed
to the server before the system call returns control to user space.
This provides greater data cache coherence among clients, but at a sig-
nificant performance cost.

Applications can use the O_SYNC open flag to force application writes
to individual files to go to the server immediately without the use of
the sync mount option.
...
NFS version 4 caching features
...
A file delegation ...
... Once a file has been delegated to a client, the client can cache
that file’s data and metadata aggressively without contacting the
server.
...

JB


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 06:11 AM
JB
 
Default NFS Buffering

JB <jb.1234abcd <at> gmail.com> writes:

> ...

How nfsiod's number of read-ahead/write-behind worker threads is determined
and configured ?

$ cat linux-2.6.34.i686/fs/nfs/internal.h
/*
* NFS internal definitions
*/
...
/* Maximum number of readahead requests
* FIXME: this should really be a sysctl so that users may tune it to suit
* their needs. People that do NFS over a slow network, might for
* instance want to reduce it to something closer to 1 for improved
* interactive response.
*/
#define NFS_MAX_READAHEAD (RPC_DEF_SLOT_TABLE - 1)
...

$ cat linux-2.6.34.i686/include/linux/sunrpc/xprt.h
/*
...
* Declarations for the RPC transport interface.
...
*/
...
#define RPC_DEF_SLOT_TABLE (16U)
...

So, the max # of workers is 16 - 1 = 15

See: linux-2.6.34.i686/Documentation/kernel-parameters.txt
...
sunrpc.tcp_slot_table_entries=
sunrpc.udp_slot_table_entries=
[NFS,SUNRPC]
Sets the upper limit on the number of simultaneous
RPC calls that can be sent from the client to a
server. Increasing these values may allow you to
improve throughput, but will also increase the
amount of memory reserved for use by the client.
...

$ lsmod |grep -i sunrpc
sunrpc 163601 1
$ $ modinfo sunrpc
filename: /lib/modules/2.6.34.7-56.fc13.i686/kernel/net/sunrpc/sunrpc.ko
license: GPL
srcversion: 274F8EC6B56054A06EDF2A4
depends:
vermagic: 2.6.34.7-56.fc13.i686 SMP mod_unload 686
parm: min_resvportortnr
parm: max_resvportortnr
parm: tcp_slot_table_entries:slot_table_size
parm: udp_slot_table_entries:slot_table_size

$ ls /sys/module/sunrpc/parameters/
max_resvport pool_mode udp_slot_table_entries
min_resvport tcp_slot_table_entries
$ cat /sys/module/sunrpc/parameters/udp_slot_table_entries
16
$ cat /sys/module/sunrpc/parameters/tcp_slot_table_entries
16

Managing NFS and NIS, 2nd Edition.
By Hal Stern, Mike Eisler and Ricardo Labiaga
18.5. NFS async thread tuning.
...
If you are running eight NFS async threads on an NFS client, then the client
will generate eight NFS write requests at once when it is performing
a sequential write to a large file. The eight requests are handled by the NFS
async threads. ... when a Solaris process issues a new write requests while
all the NFS async threads are blocked waiting for a reply from the server,
the write request is queued in the kernel and the requesting process returns
successfully without blocking. The requesting process does not issue an RPC to
the NFS server itself, only the NFS async threads do. When an NFS async thread
RPC call completes, it proceeds to grab the next request from the queue and
sends a new RPC to the server. It may be necessary to reduce the number of NFS
requests if a server cannot keep pace with the incoming NFS write requests.
...

And for those who are not into NFS ...

Lesley Gore - It's My Party (1965)
http://www.youtube.com/watch?v=XsYJyVEUaC4

JB


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 08:22 AM
Simon Andrews
 
Default NFS Buffering

On 29/09/2010 16:27, Ian Chapman wrote:
> On 29/09/10 22:51, Simon Andrews wrote:
>
>> That sounds like it might be an answer. It's a shame there's no way to
>> specify this per-process, but this machine does have quite a bit of RAM
>> in it, so having this specified as a percentage of RAM might make it
>> larger than we'd want.
>>
>> I'll have a play and see what effect this has.
>
> No problem, there's also a good page here which explains a bit more how
> all this works, as well as some explanation on other tunables that might
> suit you better.
>
> http://www.westnet.com/~gsmith/content/linux-pdflush.htm

Thanks, that's really helpful.

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 08:33 AM
Simon Andrews
 
Default NFS Buffering

On 29/09/2010 20:09, JB wrote:
>> Thanks.
>> Could you please give us (on Fedora 13) an unedited output of:
> # cat /etc/mtab
> # cat /proc/mounts

remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,relatime,vers=3,rsize=65536,wsize=65536,namlen= 255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=149.155.100.6,mountvers=3,mountport=4046,mo untproto=tcp,addr=XXX.XXX.XXX.XXX
0 0

remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
rw,tcp,addr=XXX.XXX.XXX.XXX 0 0


--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 08:35 AM
Simon Andrews
 
Default NFS Buffering

On 30/09/2010 01:21, Wolfgang S. Rupprecht wrote:
>
> Simon Andrews<simon.andrews@bbsrc.ac.uk> writes:
>> Does anyone know how to either make this buffer smaller, or get rid of
>> it all together so the scp can accruately report on its progress?
>
> Why not cut to the chase and run scp to the remote server. TCP will do
> a wonderful job of filling your pipe, but not too much.

I'd love to, but I don't control the remote end of the connection and
the only access I have is via an NFS mount.

--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 09-30-2010, 11:31 AM
JB
 
Default NFS Buffering

JB <jb.1234abcd <at> gmail.com> writes:

> ...

Firewall impact.

See 'man 5 nfs' - Mounting through a firewall.

Because you do not have access to your nfs server, the nfs client remains to
be checked for any messages coming from nfs server that may be blocked or
perhaps erroneously forwarded by your Fedora 13 client.

Your nfs server may be sending unsolicited NEW messages (as opposite to
ESTABLISHED, RELATED) that may be rejected by your (default) firewall settings.

You are also communicating with the source-of-the-big-file machine.

Iptables is your default netfilter/firewall on Fedora 13.
# iptables -L -n -v

If needed you should add extra log rules (man iptables; see LOG target; see
presumably /var/log/messages).

Test the firewall on your nfs client machine from an external machine.

JB



--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 10-02-2010, 06:13 AM
NiftyFedora Mitch
 
Default NFS Buffering

On Wed, Sep 29, 2010 at 8:27 AM, Ian Chapman
<packages@amiga-hardware.com> wrote:
> On 29/09/10 22:51, Simon Andrews wrote:
>
>> That sounds like it might be an answer. *It's a shame there's no way to
>> specify this per-process, but this machine does have quite a bit of RAM
>> in it, so having this specified as a percentage of RAM might make it
>> larger than we'd want.
>>
>> I'll have a play and see what effect this has.
>
> No problem, there's also a good page here which explains a bit more how
> all this works, as well as some explanation on other tunables that might
> suit you better.
>
> http://www.westnet.com/~gsmith/content/linux-pdflush.htm
>
> --

If you have a lot of ram you can boot with a kernel flag
that specifies a smaller memory resource.

OR

You can write a program that gobbles a lot of memory
(compare malloc(); calloc()). Running a memory hog
can trigger IO of dirty pages out to disk or trigger
a the high/low water mark used to trigger IO. This works
because at the OS level all of free memory can be used
by the OS as a buffer of one type or another.

The nice thing about a littlepiggie program is that
you can run it from user space and also adjust
how much memory it grabs and when so you can tune
things to your liking. When you kill it you quickly
get all the memory back.




--
* * * * NiftyFedora
* * * * T o m** M i t c h e l l
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 
Old 10-02-2010, 06:19 AM
NiftyFedora Mitch
 
Default NFS Buffering

On Thu, Sep 30, 2010 at 1:33 AM, Simon Andrews
<simon.andrews@bbsrc.ac.uk> wrote:
> On 29/09/2010 20:09, JB wrote:
>>> Thanks.
>>> Could you please give us (on Fedora 13) an unedited output of:
>> # cat /etc/mtab
>> # cat /proc/mounts
>
> remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
> rw,relatime,vers=3,rsize=65536,wsize=65536,namlen= 255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mou ntaddr=NNN.MMM.100.6,mountvers=3,mountport=4046,mo untproto=tcp,addr=XXX.XXX.XXX.XXX
> 0 0
>
> remote.server.name:/vol/ftp/ftp-1 /mnt/remote nfs
> rw,tcp,addr=XXX.XXX.XXX.XXX 0 0

Do check with traceroute how packets are routed. It looks like the
server has multiple IP addresses. If you pick the wrong one
a straight shot route can be lost in preference to some other
out past the neighbors barn path.


--
* * * * NiftyFedora
* * * * T o m** M i t c h e l l
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
 

Thread Tools




All times are GMT. The time now is 12:05 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org