Linux Archive - No buffer space available - loses network connectivity

Linux Archive (http://www.linux-archive.org/)

- CentOS (http://www.linux-archive.org/centos/)

- - No buffer space available - loses network connectivity (http://www.linux-archive.org/centos/570770-no-buffer-space-available-loses-network-connectivity.html)

No buffer space available - loses network connectivity

Hi,

I have a centos 5.6 xen vps which loses network connectivity once in a
while with following error.

=========================================
-bash-3.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
=========================================

All my investigation so far led me to believe that it is because
skbuff cache getting full.

================================================== =======================

PROC-SLABINFO
skbuff_fclone_cache 227 308 512 7 1 : tunables 54
27 8 : slabdata 44 44 0
skbuff_head_cache 1574 1650 256 15 1 : tunables 120 60
8 : slabdata 110 110 0

SLAB-TOP
Active / Total Objects (% used) : 2140910 / 2200115 (97.3%)
Active / Total Slabs (% used) : 139160 / 139182 (100.0%)
Active / Total Caches (% used) : 88 / 136 (64.7%)
Active / Total Size (% used) : 512788.94K / 520252.14K (98.6%)
Minimum / Average / Maximum Object : 0.02K / 0.24K / 128.00K

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
664000 620290 93% 0.09K 16600 40 66400K buffer_head
409950 408396 99% 0.21K 22775 18 91100K dentry_cache
343056 340307 99% 0.08K 7147 48 28588K selinux_inode_security
338590 336756 99% 0.74K 67718 5 270872K ext3_inode_cache
143665 143363 99% 0.06K 2435 59 9740K size-64
99540 99407 99% 0.25K 6636 15 26544K size-256
96450 96447 99% 0.12K 3215 30 12860K size-128
60858 60858 100% 0.52K 8694 7 34776K radix_tree_node
12420 11088 89% 0.16K 540 23 2160K vm_area_struct
5895 4185 70% 0.25K 393 15 1572K filp
4816 3355 69% 0.03K 43 112 172K size-32
2904 2810 96% 0.09K 66 44 264K sysfs_dir_cache
2058 1937 94% 0.58K 343 6 1372K proc_inode_cache
1728 1215 70% 0.02K 12 144 48K anon_vma
1650 1590 96% 0.25K 110 15 440K skbuff_head_cache
1498 1493 99% 2.00K 749 2 2996K size-2048
1050 1032 98% 0.55K 150 7 600K inode_cache
792 767 96% 1.00K 198 4 792K size-1024
649 298 45% 0.06K 11 59 44K pid
600 227 37% 0.09K 15 40 60K journal_head
590 298 50% 0.06K 10 59 40K delayacct_cache
496 424 85% 0.50K 62 8 248K size-512
413 156 37% 0.06K 7 59 28K fs_cache
404 44 10% 0.02K 2 202 8K biovec-1
390 293 75% 0.12K 13 30 52K bio
327 327 100% 4.00K 327 1 1308K size-4096
320 190 59% 0.38K 32 10 128K ip_dst_cache
308 227 73% 0.50K 44 7 176K skbuff_fclone_cache
258 247 95% 0.62K 43 6 172K sock_inode_cache
254 254 100% 1.84K 127 2 508K task_struct
252 225 89% 0.81K 28 9 224K signal_cache
240 203 84% 0.73K 48 5 192K shmem_inode_cache
204 204 100% 2.06K 68 3 544K sighand_cache
202 4 1% 0.02K 1 202 4K revoke_table
195 194 99% 0.75K 39 5 156K UDP
159 77 48% 0.07K 3 53 12K eventpoll_pwq
145 139 95% 0.75K 29 5 116K files_cache
144 41 28% 0.02K 1 144 4K journal_handle
140 140 100% 0.88K 35 4 140K mm_struct
140 77 55% 0.19K 7 20 28K eventpoll_epi
135 135 100% 2.12K 135 1 540K kmem_cache
121 45 37% 0.69K 11 11 88K UNIX
119 114 95% 0.52K 17 7 68K idr_layer_cache
118 41 34% 0.06K 2 59 8K blkdev_ioc
112 32 28% 0.03K 1 112 4K tcp_bind_bucket
110 56 50% 0.17K 5 22 20K file_lock_cache
106 35 33% 0.07K 2 53 8K avc_node
105 98 93% 1.50K 21 5 168K TCP
105 100 95% 1.04K 15 7 120K bio_map_info
92 1 1% 0.04K 1 92 4K dnotify_cache
80 18 22% 0.19K 4 20 16K tw_sock_TCP
70 44 62% 0.27K 5 14 20K blkdev_requests
59 23 38% 0.06K 1 59 4K biovec-4
59 13 22% 0.06K 1 59 4K fib6_nodes
59 11 18% 0.06K 1 59 4K ip_fib_hash
59 11 18% 0.06K 1 59 4K ip_fib_alias
53 53 100% 0.07K 1 53 4K taskstats_cache
53 1 1% 0.07K 1 53 4K inotify_watch_cache
48 48 100% 0.16K 2 24 8K sigqueue
48 8 16% 0.08K 1 48 4K crq_pool
45 27 60% 0.25K 3 15 12K mnt_cache
45 29 64% 0.25K 3 15 12K dquot
45 32 71% 0.25K 3 15 12K sgpool-8
40 19 47% 0.19K 2 20 8K key_jar
32 32 100% 0.50K 4 8 16K sgpool-16
32 32 100% 1.00K 8 4 32K sgpool-32
32 32 100% 2.00K 16 2 64K sgpool-64
32 32 100% 4.00K 32 1 128K sgpool-128
31 31 100% 8.00K 31 1 248K size-8192
30 27 90% 1.54K 6 5 48K blkdev_queue
30 14 46% 0.12K 1 30 4K request_sock_TCP
30 3 10% 0.12K 1 30 4K inet_peer_cache

FREE-M
total used free shared buffers cached
Mem: 8192 4821 3370 0 500 2793
-/+ buffers/cache: 1527 6664
Swap: 8191 0 8191

PROC-MEMINFO
MemTotal: 8388608 kB
MemFree: 3451384 kB
Buffers: 512352 kB
Cached: 2860580 kB
SwapCached: 0 kB
Active: 2971812 kB
Inactive: 1178908 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 8388608 kB
LowFree: 3451384 kB
SwapTotal: 8388600 kB
SwapFree: 8388600 kB
Dirty: 644 kB
Writeback: 0 kB
AnonPages: 777524 kB
Mapped: 24816 kB
Slab: 557716 kB
PageTables: 16300 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 12582904 kB
Committed_AS: 1624128 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 4556 kB
VmallocChunk: 34359733771 kB
================================================== =========

Can any one provide any insights into a possible solution of this issue.

--
Kind Regards,
Sherin
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

No buffer space available - loses network connectivity

I haven't worked with xen in a few months, but I'd highly suggest looking at the xen host server itself instead of the vps. *Setup some sort of monitoring on the VPS, coordinate the time it looses internet connection to the host server logs, maybe it'll provide some insight. *Unless, its this VPS. *Was this built off of XEN's templates or a P2V system?

On Thu, Sep 1, 2011 at 12:36 AM, Sherin George <list@sheringeorge.co.cc> wrote:

Hi,

I have a centos 5.6 xen vps which loses network connectivity once in a

while with following error.

=========================================

-bash-3.2# ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

ping: sendmsg: No buffer space available

ping: sendmsg: No buffer space available

ping: sendmsg: No buffer space available

ping: sendmsg: No buffer space available

=========================================

All my investigation so far led me to believe that it is because

skbuff cache getting full.

================================================== =======================

PROC-SLABINFO

skbuff_fclone_cache * *227 * *308 * *512 * *7 * *1 : tunables * 54

27 * *8 : slabdata * * 44 * * 44 * * *0

skbuff_head_cache * 1574 * 1650 * *256 * 15 * *1 : tunables *120 * 60

*8 : slabdata * *110 * *110 * * *0

SLAB-TOP

*Active / Total Objects (% used) * *: 2140910 / 2200115 (97.3%)

*Active / Total Slabs (% used) * * *: 139160 / 139182 (100.0%)

*Active / Total Caches (% used) * * : 88 / 136 (64.7%)

*Active / Total Size (% used) * * * : 512788.94K / 520252.14K (98.6%)

*Minimum / Average / Maximum Object : 0.02K / 0.24K / 128.00K

*OBJS ACTIVE *USE OBJ SIZE *SLABS OBJ/SLAB CACHE SIZE NAME

664000 620290 *93% * *0.09K *16600 * * * 40 * * 66400K buffer_head

409950 408396 *99% * *0.21K *22775 * * * 18 * * 91100K dentry_cache

343056 340307 *99% * *0.08K * 7147 * * * 48 * * 28588K selinux_inode_security

338590 336756 *99% * *0.74K *67718 * * * *5 * *270872K ext3_inode_cache

143665 143363 *99% * *0.06K * 2435 * * * 59 * * *9740K size-64

*99540 *99407 *99% * *0.25K * 6636 * * * 15 * * 26544K size-256

*96450 *96447 *99% * *0.12K * 3215 * * * 30 * * 12860K size-128

*60858 *60858 100% * *0.52K * 8694 * * * *7 * * 34776K radix_tree_node

*12420 *11088 *89% * *0.16K * *540 * * * 23 * * *2160K vm_area_struct

*5895 * 4185 *70% * *0.25K * *393 * * * 15 * * *1572K filp

*4816 * 3355 *69% * *0.03K * * 43 * * *112 * * * 172K size-32

*2904 * 2810 *96% * *0.09K * * 66 * * * 44 * * * 264K sysfs_dir_cache

*2058 * 1937 *94% * *0.58K * *343 * * * *6 * * *1372K proc_inode_cache

*1728 * 1215 *70% * *0.02K * * 12 * * *144 * * * *48K anon_vma

*1650 * 1590 *96% * *0.25K * *110 * * * 15 * * * 440K skbuff_head_cache

*1498 * 1493 *99% * *2.00K * *749 * * * *2 * * *2996K size-2048

*1050 * 1032 *98% * *0.55K * *150 * * * *7 * * * 600K inode_cache

* 792 * *767 *96% * *1.00K * *198 * * * *4 * * * 792K size-1024

* 649 * *298 *45% * *0.06K * * 11 * * * 59 * * * *44K pid

* 600 * *227 *37% * *0.09K * * 15 * * * 40 * * * *60K journal_head

* 590 * *298 *50% * *0.06K * * 10 * * * 59 * * * *40K delayacct_cache

* 496 * *424 *85% * *0.50K * * 62 * * * *8 * * * 248K size-512

* 413 * *156 *37% * *0.06K * * *7 * * * 59 * * * *28K fs_cache

* 404 * * 44 *10% * *0.02K * * *2 * * *202 * * * * 8K biovec-1

* 390 * *293 *75% * *0.12K * * 13 * * * 30 * * * *52K bio

* 327 * *327 100% * *4.00K * *327 * * * *1 * * *1308K size-4096

* 320 * *190 *59% * *0.38K * * 32 * * * 10 * * * 128K ip_dst_cache

* 308 * *227 *73% * *0.50K * * 44 * * * *7 * * * 176K skbuff_fclone_cache

* 258 * *247 *95% * *0.62K * * 43 * * * *6 * * * 172K sock_inode_cache

* 254 * *254 100% * *1.84K * *127 * * * *2 * * * 508K task_struct

* 252 * *225 *89% * *0.81K * * 28 * * * *9 * * * 224K signal_cache

* 240 * *203 *84% * *0.73K * * 48 * * * *5 * * * 192K shmem_inode_cache

* 204 * *204 100% * *2.06K * * 68 * * * *3 * * * 544K sighand_cache

* 202 * * *4 * 1% * *0.02K * * *1 * * *202 * * * * 4K revoke_table

* 195 * *194 *99% * *0.75K * * 39 * * * *5 * * * 156K UDP

* 159 * * 77 *48% * *0.07K * * *3 * * * 53 * * * *12K eventpoll_pwq

* 145 * *139 *95% * *0.75K * * 29 * * * *5 * * * 116K files_cache

* 144 * * 41 *28% * *0.02K * * *1 * * *144 * * * * 4K journal_handle

* 140 * *140 100% * *0.88K * * 35 * * * *4 * * * 140K mm_struct

* 140 * * 77 *55% * *0.19K * * *7 * * * 20 * * * *28K eventpoll_epi

* 135 * *135 100% * *2.12K * *135 * * * *1 * * * 540K kmem_cache

* 121 * * 45 *37% * *0.69K * * 11 * * * 11 * * * *88K UNIX

* 119 * *114 *95% * *0.52K * * 17 * * * *7 * * * *68K idr_layer_cache

* 118 * * 41 *34% * *0.06K * * *2 * * * 59 * * * * 8K blkdev_ioc

* 112 * * 32 *28% * *0.03K * * *1 * * *112 * * * * 4K tcp_bind_bucket

* 110 * * 56 *50% * *0.17K * * *5 * * * 22 * * * *20K file_lock_cache

* 106 * * 35 *33% * *0.07K * * *2 * * * 53 * * * * 8K avc_node

* 105 * * 98 *93% * *1.50K * * 21 * * * *5 * * * 168K TCP

* 105 * *100 *95% * *1.04K * * 15 * * * *7 * * * 120K bio_map_info

* *92 * * *1 * 1% * *0.04K * * *1 * * * 92 * * * * 4K dnotify_cache

* *80 * * 18 *22% * *0.19K * * *4 * * * 20 * * * *16K tw_sock_TCP

* *70 * * 44 *62% * *0.27K * * *5 * * * 14 * * * *20K blkdev_requests

* *59 * * 23 *38% * *0.06K * * *1 * * * 59 * * * * 4K biovec-4

* *59 * * 13 *22% * *0.06K * * *1 * * * 59 * * * * 4K fib6_nodes

* *59 * * 11 *18% * *0.06K * * *1 * * * 59 * * * * 4K ip_fib_hash

* *59 * * 11 *18% * *0.06K * * *1 * * * 59 * * * * 4K ip_fib_alias

* *53 * * 53 100% * *0.07K * * *1 * * * 53 * * * * 4K taskstats_cache

* *53 * * *1 * 1% * *0.07K * * *1 * * * 53 * * * * 4K inotify_watch_cache

* *48 * * 48 100% * *0.16K * * *2 * * * 24 * * * * 8K sigqueue

* *48 * * *8 *16% * *0.08K * * *1 * * * 48 * * * * 4K crq_pool

* *45 * * 27 *60% * *0.25K * * *3 * * * 15 * * * *12K mnt_cache

* *45 * * 29 *64% * *0.25K * * *3 * * * 15 * * * *12K dquot

* *45 * * 32 *71% * *0.25K * * *3 * * * 15 * * * *12K sgpool-8

* *40 * * 19 *47% * *0.19K * * *2 * * * 20 * * * * 8K key_jar

* *32 * * 32 100% * *0.50K * * *4 * * * *8 * * * *16K sgpool-16

* *32 * * 32 100% * *1.00K * * *8 * * * *4 * * * *32K sgpool-32

* *32 * * 32 100% * *2.00K * * 16 * * * *2 * * * *64K sgpool-64

* *32 * * 32 100% * *4.00K * * 32 * * * *1 * * * 128K sgpool-128

* *31 * * 31 100% * *8.00K * * 31 * * * *1 * * * 248K size-8192

* *30 * * 27 *90% * *1.54K * * *6 * * * *5 * * * *48K blkdev_queue

* *30 * * 14 *46% * *0.12K * * *1 * * * 30 * * * * 4K request_sock_TCP

* *30 * * *3 *10% * *0.12K * * *1 * * * 30 * * * * 4K inet_peer_cache

FREE-M

* * * * * * total * * * used * * * free * * shared * *buffers * * cached

Mem: * * * * *8192 * * * 4821 * * * 3370 * * * * *0 * * * *500 * * * 2793

-/+ buffers/cache: * * * 1527 * * * 6664

Swap: * * * * 8191 * * * * *0 * * * 8191

PROC-MEMINFO

MemTotal: * * *8388608 kB

MemFree: * * * 3451384 kB

Buffers: * * * *512352 kB

Cached: * * * *2860580 kB

SwapCached: * * * * *0 kB

Active: * * * *2971812 kB

Inactive: * * *1178908 kB

HighTotal: * * * * * 0 kB

HighFree: * * * * * *0 kB

LowTotal: * * *8388608 kB

LowFree: * * * 3451384 kB

SwapTotal: * * 8388600 kB

SwapFree: * * *8388600 kB

Dirty: * * * * * * 644 kB

Writeback: * * * * * 0 kB

AnonPages: * * *777524 kB

Mapped: * * * * *24816 kB

Slab: * * * * * 557716 kB

PageTables: * * *16300 kB

NFS_Unstable: * * * *0 kB

Bounce: * * * * * * *0 kB

CommitLimit: *12582904 kB

Committed_AS: *1624128 kB

VmallocTotal: 34359738367 kB

VmallocUsed: * * *4556 kB

VmallocChunk: 34359733771 kB

================================================== =========

Can any one provide any insights into a possible solution of this issue.

--

Kind Regards,

Sherin

_______________________________________________

CentOS mailing list

CentOS@centos.org

http://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos