FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > CentOS > CentOS

 
 
LinkBack Thread Tools
 
Old 03-16-2010, 01:12 PM
Ross Walker
 
Default disk i/o stalls with mptsas since upgrade to centos 5.4

On Mar 16, 2010, at 3:43 AM, Lennert Buytenhek
<buytenh@wantstofly.org> wrote:

> Hi!
>
> On two different machines, I've been experiencing disk I/O stalls
> after
> upgrading to the CentOS 5.4 kernel. Both machines have an LSI 1068E
> MPT SAS (mptsas) controller connected to a Chenbro CK13601 36-port SAS
> expander, with one machine having 16 1T WD disks hooked up to it, and
> the other having a mix of about 20 WD/Seagate/Samsung/Hitachi 1T and
> 2T
> disks.
>
> When there's a disk I/O stall, all reads and writes to any disk behind
> the SAS controller/expander just hang for a while (typically for
> almost
> exactly eight seconds), so not just the I/O to one particular disk
> or a
> subset of the disks. The disks on other (on-board SATA) controllers
> still pass I/O requests when the SAS I/O stalls.
>
> I hacked up the attached (dirty) perl script to demonstrate this
> effect
> -- it will read /proc/diskstats in a tight loop, and keep track of
> which request entered the request queue when, and when it completed,
> and
> it will WTF if a request took more than a second. (The same thing can
> probably be done with blktrace, but I was lazy.) New requests get
> submitted, but the pending ones fail to complete for a while, and then
> they all complete at once.
>
> This happens on kernel-2.6.18-164.11.1.el5, while reverting to the
> latest CentOS 5.3 kernel (kernel-2.6.18-128.7.1.el5) makes the issue
> go
> away again, i.e. no more stalls.
>
> It doesn't seem to matter whether the I/O load is high or not -- the
> stalls happen even under almost no load at all.
>
> Before I dig into this further, has anyone experienced anything
> similar?
> A quick google search didn't come up with much.

I would use iostat -x and see if there is a disk or group of disks
that show abnormal service times and/or utilization.

Are there any errors in the logs?

How are the disks configured? Software raid?

Is the adapter's firmware at the latest revision?

Was .128 kernel running stock drivers?Is .164 kernel running stock
drivers? (maybe weak-updates from .128 kernel?)

What IO scheduler is this? Default CFQ?

I would move this discussion to 'CentOS Users' as that is the more
appropriate list for this.

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 03-17-2010, 02:27 AM
Rajagopal Swaminathan
 
Default disk i/o stalls with mptsas since upgrade to centos 5.4

Greetings,

> I would move this discussion to 'CentOS Users' as that is the more
> appropriate list for this.

uh! I didn't know one such existed. Can you give some more details
about this please?

Regards

Rajagopal
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 03-17-2010, 02:46 AM
John R Pierce
 
Default disk i/o stalls with mptsas since upgrade to centos 5.4

Rajagopal Swaminathan wrote:
> Greetings,
>
>
>> I would move this discussion to 'CentOS Users' as that is the more
>> appropriate list for this.
>>
>
> uh! I didn't know one such existed. Can you give some more details
> about this please?
>

You crossposted to it (this list). I suspect the above comment was from
the Developers list, where this was off topic.


_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 03-17-2010, 03:10 AM
Christopher Chan
 
Default disk i/o stalls with mptsas since upgrade to centos 5.4

On Wednesday, March 17, 2010 11:46 AM, John R Pierce wrote:
> Rajagopal Swaminathan wrote:
>> Greetings,
>>
>>
>>> I would move this discussion to 'CentOS Users' as that is the more
>>> appropriate list for this.
>>>
>>
>> uh! I didn't know one such existed. Can you give some more details
>> about this please?
>>
>
> You crossposted to it (this list). I suspect the above comment was from
> the Developers list, where this was off topic.
>

Actually, Ross cross-posted.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 
Old 03-17-2010, 12:27 PM
Ross Walker
 
Default disk i/o stalls with mptsas since upgrade to centos 5.4

On Mar 17, 2010, at 12:10 AM, Christopher Chan <christopher.chan@bradbury.edu.hk
> wrote:

> On Wednesday, March 17, 2010 11:46 AM, John R Pierce wrote:
>> Rajagopal Swaminathan wrote:
>>> Greetings,
>>>
>>>
>>>> I would move this discussion to 'CentOS Users' as that is the more
>>>> appropriate list for this.
>>>>
>>>
>>> uh! I didn't know one such existed. Can you give some more details
>>> about this please?
>>>
>>
>> You crossposted to it (this list). I suspect the above comment was
>> from
>> the Developers list, where this was off topic.
>>
>
> Actually, Ross cross-posted.

Yes, actually thinking about it it would have been better to BCC
developer list to keep it out of follow-ups, but still wanted to leave
a marker there.

Oh well, next time.

-Ross

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
 

Thread Tools




All times are GMT. The time now is 01:58 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org