RAID config for mailboxes host
 

Hello,

Slightly OT because not Debian specific.

Let this physical server:

Dell R710
2x6 cores 2.3 GHz
48 GB RAM
12 disks 600 GB, 15 ktr/min
only one RAID card H700

for the usage of a rather small mailhost:

MDA
IMAP/POP for 1500 mailboxes (Maildir)
webmail
(MX are other machines)

We consider two RAID config for the 12 disks:

A 2 disks RAID 1 for system
10 disks RAID 6 for mailbox space

B 12 disks RAID 6 for the whole

In A, we lose 1.2 (raw) TB with respect to B. Advocates of A say that
in B, the heads read/write activity of the mailboxes will seriously
impact the system performance. In A, the system has its own axes,
separate from the mailboxes ones.

Side note: We are considering dovecot.

Do you think that the performance loss will be so visible that it
worth sacrifying 1.2 TB of mailbox space?

Facts are far better than guesses, but how to bench those two configs,
with which test protocol, and how realistic?

Thanks,

--
Emmanuel Halbwachs Observatoire de Paris-Meudon
Resp. Réseau/Sécurité 5 Place Jules Janssen
tel : +33 1 45 07 75 54 F 92195 MEUDON CEDEX
fax : +33 1 45 07 01 89 véhicules : 11 av. Marcellin Berthelot


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110909215213.GJ28359@ltserver1.obspm.fr">http://lists.debian.org/20110909215213.GJ28359@ltserver1.obspm.fr

RAID config for mailboxes host
 

In my opinion and experience with email systems off 100k+ users I would put all 12 disks in a raid 10 array. This is as long as you're using a decent MTA which doesn't beat up your disks before actually writing the message to the user's dir including any local virus/spam scanning.

Since you are planning on Dovecot I would also research using their new alternative to Maildir called dbox. I have only tested it in non production environments on top of GlusterFS.

So to sum it up: Postfix + Dspam + dovecot. Add in the dovecot dspam plugin so your users can train dspam just by moving messages into their spam folder via their imap client.

Cory
Sent from my U.S. Cellular BlackBerry® smartphone

-----Original Message-----
From: Emmanuel Halbwachs <Emmanuel.Halbwachs@obspm.fr>
Date: Fri, 9 Sep 2011 23:52:13
To: <debian-isp@lists.debian.org>
Subject: RAID config for mailboxes host

Hello,

Slightly OT because not Debian specific.

Let this physical server:

Dell R710
2x6 cores 2.3 GHz
48 GB RAM
12 disks 600 GB, 15 ktr/min
only one RAID card H700

for the usage of a rather small mailhost:

MDA
IMAP/POP for 1500 mailboxes (Maildir)
webmail
(MX are other machines)

We consider two RAID config for the 12 disks:

A 2 disks RAID 1 for system
10 disks RAID 6 for mailbox space

B 12 disks RAID 6 for the whole

In A, we lose 1.2 (raw) TB with respect to B. Advocates of A say that
in B, the heads read/write activity of the mailboxes will seriously
impact the system performance. In A, the system has its own axes,
separate from the mailboxes ones.

Side note: We are considering dovecot.

Do you think that the performance loss will be so visible that it
worth sacrifying 1.2 TB of mailbox space?

Facts are far better than guesses, but how to bench those two configs,
with which test protocol, and how realistic?

Thanks,

--
Emmanuel Halbwachs Observatoire de Paris-Meudon
Resp. Réseau/Sécurité 5 Place Jules Janssen
tel : +33 1 45 07 75 54 F 92195 MEUDON CEDEX
fax : +33 1 45 07 01 89 véhicules : 11 av. Marcellin Berthelot


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/20110909215213.GJ28359@ltserver1.obspm.fr

RAID config for mailboxes host
 

Emmanuel Halbwachs wrote (ao):
> Let this physical server:
>
> Dell R710
> 2x6 cores 2.3 GHz
> 48 GB RAM
> 12 disks 600 GB, 15 ktr/min
> only one RAID card H700
>
> for the usage of a rather small mailhost:
>
> MDA
> IMAP/POP for 1500 mailboxes (Maildir)
> webmail
> (MX are other machines)
>
> We consider two RAID config for the 12 disks:
>
> A 2 disks RAID 1 for system
> 10 disks RAID 6 for mailbox space
>
> B 12 disks RAID 6 for the whole
>
> In A, we lose 1.2 (raw) TB with respect to B. Advocates of A say that
> in B, the heads read/write activity of the mailboxes will seriously
> impact the system performance. In A, the system has its own axes,
> separate from the mailboxes ones.

I would go for scenario B. If your workload is too much for the system,
you are screwed in both scenario A and B anyway, and in B you have extra
spindles.

Btw, any reason you go with spinning disks and not with SSDs like Intel
710/720? SSDs are a good fit for maildir workload (random IO).

Sander

--
Humilis IT Services and Solutions
http://www.humilis.net


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110912092537.GA32671@attic.humilis.net">http://lists.debian.org/20110912092537.GA32671@attic.humilis.net

RAID config for mailboxes host
 

On 09/10/2011 05:52 AM, Emmanuel Halbwachs wrote:
> Side note: We are considering dovecot.

You are considering dovecot instead of what? If the other option was
Courier, don't consider, just do it. Courier is a dog, and Dovecot is
100 times faster (really 100 times, no jokes...).

As for your RAID, I would advice against having so many HDD on a RAID
array. It just increases the odds to have issues. If you go for the 12
disks on a RAID6, then do 2x a RAID6 array of 6 disks, and use LVM to
make it be seen as a single array (if you need to do that). This is
statistically a safer option, plus LVM will add striping, which will
make it faster (let me know if I'm wrong here, but I remember this is
what happens...).

Anyway, I think that 1500 mailboxes is really a low volume, and I don't
think you need that much of hardware! :)

Thomas


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E6E487D.4060602@debian.org">http://lists.debian.org/4E6E487D.4060602@debian.org

RAID config for mailboxes host
 

Hello,

cory.meyer@gmail.com a écrit (Sat, Sep 10, 2011 at 12:56:31AM +0000) :
> Since you are planning on Dovecot I would also research using their
> new alternative to Maildir called dbox.

For mail, which is critical service no 1 (users do not know what DNS
is ;-), we are rather conservative, I have to admit.

> I have only tested it in non production environments on top of
> GlusterFS.

Wow... We tested glusterfs some years ago on a HPC cluster and it was
far from stable. Perhaps it's production ready now.

Thanks,

--
Emmanuel Halbwachs Observatoire de Paris-Meudon
Resp. Réseau/Sécurité 5 Place Jules Janssen
tel : +33 1 45 07 75 54 F 92195 MEUDON CEDEX
fax : +33 1 45 07 01 89 véhicules : 11 av. Marcellin Berthelot


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110912213028.GA6879@ltserver1.obspm.fr">http://lists.debian.org/20110912213028.GA6879@ltserver1.obspm.fr

RAID config for mailboxes host
 

Hello,

Sander a écrit (Mon, Sep 12, 2011 at 11:25:37AM +0200) :
> Btw, any reason you go with spinning disks and not with SSDs like Intel
> 710/720? SSDs are a good fit for maildir workload (random IO).

Just not the feeling to be "adventurous" with the mail.

Thanks,

--
Emmanuel Halbwachs Observatoire de Paris-Meudon
Resp. Réseau/Sécurité 5 Place Jules Janssen
tel : +33 1 45 07 75 54 F 92195 MEUDON CEDEX
fax : +33 1 45 07 01 89 véhicules : 11 av. Marcellin Berthelot


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110912213214.GB6879@ltserver1.obspm.fr">http://lists.debian.org/20110912213214.GB6879@ltserver1.obspm.fr

RAID config for mailboxes host
 

Hello,

Thomas Goirand a écrit (Tue, Sep 13, 2011 at 01:59:25AM +0800) :
> You are considering dovecot instead of what? If the other option was
> Courier, don't consider, just do it. Courier is a dog,

Yes, we're currently running Courier on a RAID 10 setup with 15k.

> do 2x a RAID6 array of 6 disks, and use LVM to make it be seen as a
> single array (if you need to do that). This is statistically a safer
> option, plus LVM will add striping

That's interesting. But without doing the maths, I feel that LVM is
statistically the opposite.

> Anyway, I think that 1500 mailboxes is really a low volume,

Sure, but mail is vital for our demanding users.

> and I don't think you need that much of hardware! :)

Too late :-)

Thanks,

--
Emmanuel Halbwachs Observatoire de Paris-Meudon
Resp. Réseau/Sécurité 5 Place Jules Janssen
tel : +33 1 45 07 75 54 F 92195 MEUDON CEDEX
fax : +33 1 45 07 01 89 véhicules : 11 av. Marcellin Berthelot


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 20110912214233.GC6879@ltserver1.obspm.fr">http://lists.debian.org/20110912214233.GC6879@ltserver1.obspm.fr

RAID config for mailboxes host
 

On 09/09/2011 22:52, Emmanuel Halbwachs wrote:

We consider two RAID config for the 12 disks:

A 2 disks RAID 1 for system
10 disks RAID 6 for mailbox space

B 12 disks RAID 6 for the whole

In A, we lose 1.2 (raw) TB with respect to B. Advocates of A say that
in B, the heads read/write activity of the mailboxes will seriously
impact the system performance. In A, the system has its own axes,
separate from the mailboxes ones.


I still would go with option A -- you actually gain a lot of it. All
the logs and OS specific operations + MTA spools will go on a separate
set of spindles and in some situations that helps a lot.


Make sure you get battery for the PERC controller with ideally 1GB NV
cache -- it makes difference especially for massive amount of small
writes which I'm sure you are going to have.


Also, I am not sure how dovecot handles fsync() however this can be pain
in the ass if you are using ext3/4 file system which by the way is
definitelly not the right choice in this case ;-) (what you already know
I am sure).


I went through similar problem few months back - we were considering
upgrading our mail server for 200+ mail accounts (~ 400 mail addresses
in total) and we ended up... getting Google Apps for Businesses. Highly
recommend just having a look at it -- cost wise it looks a bit scary
initially but you are gaining *a lot*.



--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: 4E8B7931.9040706@timedout.org">http://lists.debian.org/4E8B7931.9040706@timedout.org

RAID config for mailboxes host
 

Hi,

Sander wrote:
> Emmanuel Halbwachs wrote (ao):
>> We consider two RAID config for the 12 disks:
>> A 2 disks RAID 1 for system
>> 10 disks RAID 6 for mailbox space
>> B 12 disks RAID 6 for the whole
>> In A, we lose 1.2 (raw) TB with respect to B. Advocates of A say that
in B, the heads read/write activity of the mailboxes will seriously
impact the system performance. In A, the system has its own axes,
separate from the mailboxes ones.
>
> Btw, any reason you go with spinning disks and not with SSDs like Intel
710/720? SSDs are a good fit for maildir workload (random IO).

Option A loses 1.2 (raw) TB ..... what size are the disks? Using SSDs at
that size will likely not be possible. If it is possible, then it would
be extremely expensive.

I would prefer the multi RAID6 idea myself.

--
Kind Regards
AndrewM

Andrew McGlashan
Broadband Solutions now including VoIP







--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: cec81f0516b7c14a5b841672c3920070.squirrel@www.affi nityvision.com.au">http://lists.debian.org/cec81f0516b7c14a5b841672c3920070.squirrel@www.affi nityvision.com.au