FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > EXT3 Users

 
 
LinkBack Thread Tools
 
Old 09-29-2011, 01:08 PM
"Ted Ts'o"
 
Default ext3 with maildir++ = huge disk latency and high load

On Thu, Sep 29, 2011 at 11:29:42AM +0400, Andrey wrote:
> Ok. Here are bonnie results on fresh installed Debian with 200GB
> FREE ext3 /home partitition (4 disks in RAID5 on HP Proliant DL380
> G4 server):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> debian 2G 243 97 22555 10 8794 2 1810 97 120444
> 11 317.0 5
> Latency 135ms 967ms 723ms 26526us 13143us
> 586ms
>
> Latency is also very bad according results. What is the reason?
> Hardware or ext3 itseld? Will try with xfs an ext4 and compare then.

Ext4 will do better than ext3; but I can predict to you up front that
xfs will do better than ext4, because it is better optimized for RAID
arrays at the moment. Ext4 has superblock fields to store RAID 5
parameters, but the code to fully take advantage of those RAID
parameters is not fully implemented.

You should also take a look at your RAID parameters. If your RAID
stripe size is too large, it will impact workloads such as mail
servers which typically involve small writes. Have you considered
using RAID 10 over RAID 5? It's not as efficient from a space
perspective but if you are primarily concerned about throughput and
latency, it's the way to go.

- Ted

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 09-29-2011, 02:36 PM
Andrey
 
Default ext3 with maildir++ = huge disk latency and high load

Let me to share some testing RAID 5 results with bonnie++:

ext3 (defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 242 96 22458 10 8826 2 1854 98 120985 11
317.1 3
Latency 211ms 896ms 720ms 22258us 18733us
622ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++
15404 35
Latency 12284us 992us 1029us 432us 140us
76us


ext3 (-T small,defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 229 98 4989 5 3862 1 1762 97 91111 9
266.6 6
Latency 79046us 22858ms 2577ms 19253us 12120us
767ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++
10347 26
Latency 9968us 977us 964us 482us 144us
178us


ext3 (-T news,defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 237 95 22807 11 8807 2 1897 99 121893 11
324.6 5
Latency 223ms 808ms 523ms 13765us 11049us
831ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++
15460 35
Latency 417us 984us 1024us 430us 140us
175us


ext4 (defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 256 98 21495 6 9896 2 1771 99 125775 11
349.7 5
Latency 37738us 992ms 3490ms 10811us 12045us
495ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++
17428 45
Latency 424us 982us 1023us 367us 139us
174us


xfs(defaults,noatime,logbufs=8,logbsize=131072):

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
debian 2G 476 96 35129 9 12524 3 1417 99 124716 12
445.9 9
Latency 19798us 420ms 721ms 14122us 9394us
131ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++
1346 8
Latency 104ms 291us 48604us 109ms 45us
227ms


It seems that latency is big in whole results, best is for XFS. It is
tempting me to think that there are some RAID 5 issues here. It's really
strange that block writing for SCSI server disks in RAID5 is no more
than 30MB/sec(XFS). I guess I should consider XFS file system or
different RAID configuration. May be someone can comment this strange
benchmark result? Will very appreciate that.


With regards, Andrey.

23.09.2011 11:31, Janne Pikkarainen пишет:

Hello,

On 09/23/2011 08:51 AM, Andrey wrote:

Hello,

I have a production mail server with maildir++ structure and about
250GB (~10 millions) of files on the ext3 partition on RAID5. It's
mounted with noatime option. These mail server is responsible to local
delivery and storing mail messages.

System has Debian Squeeze installed and Exim as MDA + Dovecot as
IMAP+POP3 server.

Bonnie results are terrible. Sequential output for Block and Rewrite
are 10722ms and 9232ms. So if there is a 1000 messages in the mail
queue load is extremely high, delivery time is very big and server can
hang. I did not see such problems with UFS on FreeBSD server.

As I understand ext3 file system is really bad for such configurations
with Maildir++ (many smaill files)? Is there a way to decrease disk
latency on ext3 or speed up it?

With regards, Andrey

___




_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 09-29-2011, 02:44 PM
Eric Sandeen
 
Default ext3 with maildir++ = huge disk latency and high load

On 9/29/11 9:36 AM, Andrey wrote:
> Let me to share some testing RAID 5 results with bonnie++:

What kernel version was this tested on?

Thanks,
-Eric

> ext3 (defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 242 96 22458 10 8826 2 1854 98 120985 11 317.1 3
> Latency 211ms 896ms 720ms 22258us 18733us 622ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++ 15404 35
> Latency 12284us 992us 1029us 432us 140us 76us
>
> ext3 (-T small,defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 229 98 4989 5 3862 1 1762 97 91111 9 266.6 6
> Latency 79046us 22858ms 2577ms 19253us 12120us 767ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++ 10347 26
> Latency 9968us 977us 964us 482us 144us 178us
>
> ext3 (-T news,defaults,noatime):
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 237 95 22807 11 8807 2 1897 99 121893 11 324.6 5
> Latency 223ms 808ms 523ms 13765us 11049us 831ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++ 15460 35
> Latency 417us 984us 1024us 430us 140us 175us
>
> ext4 (defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 256 98 21495 6 9896 2 1771 99 125775 11 349.7 5
> Latency 37738us 992ms 3490ms 10811us 12045us 495ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++ 17428 45
> Latency 424us 982us 1023us 367us 139us 174us
>
> xfs(defaults,noatime,logbufs=8,logbsize=131072):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 476 96 35129 9 12524 3 1417 99 124716 12 445.9 9
> Latency 19798us 420ms 721ms 14122us 9394us 131ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++ 1346 8
> Latency 104ms 291us 48604us 109ms 45us 227ms
>
> It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.
>
> With regards, Andrey.
>
> 23.09.2011 11:31, Janne Pikkarainen пишет:
>> Hello,
>>
>> On 09/23/2011 08:51 AM, Andrey wrote:
>>> Hello,
>>>
>>> I have a production mail server with maildir++ structure and about
>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>> mounted with noatime option. These mail server is responsible to local
>>> delivery and storing mail messages.
>>>
>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>> IMAP+POP3 server.
>>>
>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>> queue load is extremely high, delivery time is very big and server can
>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>
>>> As I understand ext3 file system is really bad for such configurations
>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>> latency on ext3 or speed up it?
>>>
>>> With regards, Andrey
>>>
>>> ___
>>
>
> _______________________________________________
> Ext3-users mailing list
> Ext3-users@redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 
Old 09-29-2011, 03:13 PM
Andrey
 
Default ext3 with maildir++ = huge disk latency and high load

Hello,

This is standard Debian Squeeze(6.0.2) kernel:

# uname -a
Linux debian 2.6.32-5-686 #1 SMP Fri Sep 9 20:51:05 UTC 2011 i686 GNU/Linux

29.09.2011 18:44, Eric Sandeen пишет:

On 9/29/11 9:36 AM, Andrey wrote:

Let me to share some testing RAID 5 results with bonnie++:


What kernel version was this tested on?

Thanks,
-Eric


ext3 (defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian 2G 242 96 22458 10 8826 2 1854 98 120985 11 317.1 3
Latency 211ms 896ms 720ms 22258us 18733us 622ms
Version 1.96 ------Sequential Create------ --------Random Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++ 15404 35
Latency 12284us 992us 1029us 432us 140us 76us

ext3 (-T small,defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian 2G 229 98 4989 5 3862 1 1762 97 91111 9 266.6 6
Latency 79046us 22858ms 2577ms 19253us 12120us 767ms
Version 1.96 ------Sequential Create------ --------Random Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++ 10347 26
Latency 9968us 977us 964us 482us 144us 178us

ext3 (-T news,defaults,noatime):
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian 2G 237 95 22807 11 8807 2 1897 99 121893 11 324.6 5
Latency 223ms 808ms 523ms 13765us 11049us 831ms
Version 1.96 ------Sequential Create------ --------Random Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++ 15460 35
Latency 417us 984us 1024us 430us 140us 175us

ext4 (defaults,noatime):

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian 2G 256 98 21495 6 9896 2 1771 99 125775 11 349.7 5
Latency 37738us 992ms 3490ms 10811us 12045us 495ms
Version 1.96 ------Sequential Create------ --------Random Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++ 17428 45
Latency 424us 982us 1023us 367us 139us 174us

xfs(defaults,noatime,logbufs=8,logbsize=131072):

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian 2G 476 96 35129 9 12524 3 1417 99 124716 12 445.9 9
Latency 19798us 420ms 721ms 14122us 9394us 131ms
Version 1.96 ------Sequential Create------ --------Random Create--------
debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++ 1346 8
Latency 104ms 291us 48604us 109ms 45us 227ms

It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.

With regards, Andrey.

23.09.2011 11:31, Janne Pikkarainen пишет:

Hello,

On 09/23/2011 08:51 AM, Andrey wrote:

Hello,

I have a production mail server with maildir++ structure and about
250GB (~10 millions) of files on the ext3 partition on RAID5. It's
mounted with noatime option. These mail server is responsible to local
delivery and storing mail messages.

System has Debian Squeeze installed and Exim as MDA + Dovecot as
IMAP+POP3 server.

Bonnie results are terrible. Sequential output for Block and Rewrite
are 10722ms and 9232ms. So if there is a 1000 messages in the mail
queue load is extremely high, delivery time is very big and server can
hang. I did not see such problems with UFS on FreeBSD server.

As I understand ext3 file system is really bad for such configurations
with Maildir++ (many smaill files)? Is there a way to decrease disk
latency on ext3 or speed up it?

With regards, Andrey

___




_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users





_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users
 

Thread Tools




All times are GMT. The time now is 12:06 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org