FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 05-07-2011, 07:35 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

Hi All

I have xeon server with 16 Gb Ram and no Swap memory.I am running
cassandra server on two node in cluster.When there is high load on
server kswapd0 kicks inn and take 100% cpu and make machine very slow
and we need to restart out cassandra server.I have latest kernel
2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
hurting us badly this our production server any quick help will be
appreciated.

--
S.Ali Ahsan



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 01:01 PM
Peter Kjellstr÷m
 
Default kswapd taking 100% cpu with no swap on system

On Saturday, May 07, 2011 09:35:48 PM Ali Ahsan wrote:
> Hi All
>
> I have xeon server with 16 Gb Ram and no Swap memory.I am running
> cassandra server on two node in cluster.When there is high load on
> server kswapd0 kicks inn and take 100% cpu and make machine very slow
> and we need to restart out cassandra server.I have latest kernel
> 2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
> hurting us badly this our production server any quick help will be
> appreciated.

There is more than one bug that causes this behaviour. A few related memory
managent situations (possibly responsible) may actually be avoided if you add
some swap (even if it's not used). My suggestion would be to add some swap,
set swappiness to 0 and see what happens.

/Peter
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 01:01 PM
Peter Kjellstr÷m
 
Default kswapd taking 100% cpu with no swap on system

On Saturday, May 07, 2011 09:35:48 PM Ali Ahsan wrote:
> Hi All
>
> I have xeon server with 16 Gb Ram and no Swap memory.I am running
> cassandra server on two node in cluster.When there is high load on
> server kswapd0 kicks inn and take 100% cpu and make machine very slow
> and we need to restart out cassandra server.I have latest kernel
> 2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
> hurting us badly this our production server any quick help will be
> appreciated.

There is more than one bug that causes this behaviour. A few related memory
managent situations (possibly responsible) may actually be avoided if you add
some swap (even if it's not used). My suggestion would be to add some swap,
set swappiness to 0 and see what happens.

/Peter
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 01:07 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

Thanks for reply,I dont know it has some thing to do with linux
scheduler* but some on suggested about linux scheduler cat
/sys/block/sda/queue/scheduler there are quite few of them noop
[anticipatory] deadline cfq.I have changed from cfq default to
anticipatory,Now problem is less for me kswapd0* take 10-50% of cpu
in very short bust less then milliseconds.Also i have changed
/proc/sys/vm/pagecache from 100 to 10.*
Any more suggestion will help me make server better,As for i have
read the best scheduler for databases are deadline like for mysql it
worked great.



On 05/09/2011 06:01 PM, Peter Kjellstr÷m wrote:

On Saturday, May 07, 2011 09:35:48 PM Ali Ahsan wrote:


Hi All

I have xeon server with 16 Gb Ram and no Swap memory.I am running
cassandra server on two node in cluster.When there is high load on
server kswapd0 kicks inn and take 100% cpu and make machine very slow
and we need to restart out cassandra server.I have latest kernel
2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
hurting us badly this our production server any quick help will be
appreciated.



There is more than one bug that causes this behaviour. A few related memory
managent situations (possibly responsible) may actually be avoided if you add
some swap (even if it's not used). My suggestion would be to add some swap,
set swappiness to 0 and see what happens.

/Peter



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos






--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ahsan@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 01:07 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

Thanks for reply,I dont know it has some thing to do with linux
scheduler* but some on suggested about linux scheduler cat
/sys/block/sda/queue/scheduler there are quite few of them noop
[anticipatory] deadline cfq.I have changed from cfq default to
anticipatory,Now problem is less for me kswapd0* take 10-50% of cpu
in very short bust less then milliseconds.Also i have changed
/proc/sys/vm/pagecache from 100 to 10.*
Any more suggestion will help me make server better,As for i have
read the best scheduler for databases are deadline like for mysql it
worked great.



On 05/09/2011 06:01 PM, Peter Kjellstr÷m wrote:

On Saturday, May 07, 2011 09:35:48 PM Ali Ahsan wrote:


Hi All

I have xeon server with 16 Gb Ram and no Swap memory.I am running
cassandra server on two node in cluster.When there is high load on
server kswapd0 kicks inn and take 100% cpu and make machine very slow
and we need to restart out cassandra server.I have latest kernel
2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
hurting us badly this our production server any quick help will be
appreciated.



There is more than one bug that causes this behaviour. A few related memory
managent situations (possibly responsible) may actually be avoided if you add
some swap (even if it's not used). My suggestion would be to add some swap,
set swappiness to 0 and see what happens.

/Peter



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos






--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ahsan@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 03:09 PM
Peter Kjellstr÷m
 
Default kswapd taking 100% cpu with no swap on system

Please don't top-post it's hard to follow.

On Monday, May 09, 2011 03:07:57 PM Ali Ahsan wrote:
> Thanks for reply,I dont know it has some thing to do with linux
> scheduler but some on suggested about linux scheduler cat
> /sys/block/sda/queue/scheduler there are quite few of them noop
> [anticipatory] deadline cfq.I have changed from cfq default to
> anticipatory,Now problem is less for me kswapd0 take 10-50% of cpu in
> very short bust less then milliseconds.Also i have changed
> /proc/sys/vm/pagecache from 100 to 10.

This means that you're limiting the amount of RAM that can be used as
pagecache (basically caching I/O) to 10%. This may not be an issue for you but
may also severely limit your performance (all depending on work load).

An alternative may be to simple set swappiness to 0.

> Any more suggestion will help me
> make server better,As for i have read the best scheduler for databases
> are deadline like for mysql it worked great.

You'll really have to try them to know what works for you. The two main
variables that'll affect the choice is your I/O load and the device under
(type of raid for example). For example some raid controllers really prefer
noop.

Lastly, you didn't tell us if you're on i386-pae or x86_64 which may make a
big difference.

/Peter

> On 05/09/2011 06:01 PM, Peter Kjellstr÷m wrote:
> > On Saturday, May 07, 2011 09:35:48 PM Ali Ahsan wrote:
> >> Hi All
> >>
> >> I have xeon server with 16 Gb Ram and no Swap memory.I am running
> >> cassandra server on two node in cluster.When there is high load on
> >> server kswapd0 kicks inn and take 100% cpu and make machine very slow
> >> and we need to restart out cassandra server.I have latest kernel
> >> 2.6.18-238.9.1.el5.Please let me know how can i fix this issue .Its
> >> hurting us badly this our production server any quick help will be
> >> appreciated.
> >
> > There is more than one bug that causes this behaviour. A few related
> > memory managent situations (possibly responsible) may actually be
> > avoided if you add some swap (even if it's not used). My suggestion
> > would be to add some swap, set swappiness to 0 and see what happens.
> >
> > /Peter
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 03:49 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

On 05/09/2011 08:09 PM, Peter Kjellstr÷m wrote:
>
> This means that you're limiting the amount of RAM that can be used as
> pagecache (basically caching I/O) to 10%. This may not be an issue for you but
> may also severely limit your performance (all depending on work load).
>
> An alternative may be to simple set swappiness to 0.
>

Hmmm nice points,I am using Sata with LVM with 1 TB of two HD.
> You'll really have to try them to know what works for you. The two main
> variables that'll affect the choice is your I/O load and the device under
> (type of raid for example). For example some raid controllers really prefer
> noop.
>
> Lastly, you didn't tell us if you're on i386-pae or x86_64 which may make a
> big difference.

System is x86_64

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 04:45 PM
Lamar Owen
 
Default kswapd taking 100% cpu with no swap on system

On Monday, May 09, 2011 11:49:26 AM Ali Ahsan wrote:
> Hmmm nice points,I am using Sata with LVM with 1 TB of two HD.

What sort of SATA drives are you using? There are some known issues with some SATA drives in certain configurations and on some controllers. It shouldn't cause kswapd to hit high CPU, but it is worth checking out.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 05:08 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

On 05/09/2011 09:45 PM, Lamar Owen wrote:
>
> What sort of SATA drives are you using? There are some known issues with some SATA drives in certain configurations and on some controllers. It shouldn't cause kswapd to hit high CPU, but it is worth checking out.
>

these modules are loaded when i install centos 5.5.

lsmod | egrep 'ata_piix|libata'
ata_piix 57285 3
libata 208977 1 ata_piix
scsi_mod 199001 5 scsi_dh,sg,usb_storage,libata,sd_mod

--

S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ahsan@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 05-09-2011, 05:11 PM
Ali Ahsan
 
Default kswapd taking 100% cpu with no swap on system

On 05/09/2011 10:08 PM, Ali Ahsan wrote:
> On 05/09/2011 09:45 PM, Lamar Owen wrote:
>> What sort of SATA drives are you using? There are some known issues with some SATA drives in certain configurations and on some controllers. It shouldn't cause kswapd to hit high CPU, but it is worth checking out.
>>
> these modules are loaded when i install centos 5.5.
>
> lsmod | egrep 'ata_piix|libata'
> ata_piix 57285 3
> libata 208977 1 ata_piix
> scsi_mod 199001 5 scsi_dh,sg,usb_storage,libata,sd_mod
>
More info

SCSI subsystem initialized
libata version 3.00 loaded.
ata_piix 0000:00:1f.2: version 2.12
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 19 (level, low) -> IRQ 74
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x3158 ctl 0x316c bmdma 0x3130 irq 74
ata2: SATA max UDMA/133 cmd 0x3150 ctl 0x3168 bmdma 0x3138 irq 74
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-8: WDC WD10EARS-003BB1, 80.00A80, max UDMA/133
ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.00: configured for UDMA/133
ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-8: WDC WD10EARS-003BB1, 80.00A80, max UDMA/133
ata2.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata2.00: configured for UDMA/133
Vendor: ATA Model: WDC WD10EARS-003 Rev: 80.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
Vendor: ATA Model: WDC WD10EARS-003 Rev: 80.0
Type: Direct-Access ANSI SCSI revision: 05


--
S.Ali Ahsan

Senior System Engineer

e-Business (Pvt) Ltd

49-C Jail Road, Lahore, P.O. Box 676
Lahore 54000, Pakistan

Tel: +92 (0)42 3758 7140 Ext. 128

Mobile: +92 (0)345 831 8769

Fax: +92 (0)42 3758 0027

Email: ali.ahsan@panasiangroup.com



www.ebusiness-pg.com

www.panasiangroup.com

Confidentiality: This e-mail and any attachments may be confidential
and/or privileged. If you are not a named recipient, please notify the
sender immediately and do not disclose the contents to another person
use it for any purpose or store or copy the information in any medium.
Internet communications cannot be guaranteed to be timely, secure, error
or virus-free. We do not accept liability for any errors or omissions.

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 08:21 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ę2007 - 2008, www.linux-archive.org