FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian ISP

 
 
LinkBack Thread Tools
 
Old 05-25-2008, 01:11 PM
Michelle Konzack
 
Default Backup of an courier-imap mailarchive

Hello,

On my "business" mailserver (courier-imap) I have 43 users (some of them
as On-Line-Archive), together over 42.000 mailfolders and 17.000.000
messages stored and each month we get 1800 mailfolders and 150.000
messages more. The total volume is arround 180 GByte. (real not blocks)

Currently I run a daily plus weekly cronjob with a selfmade script which
check first the /cur/ directories and if it does not match a previously
created MD5 + directory listing it will backuped. Daily incremental and
weekly full.

The daily incremental backup take arround 5 hours and the weekly full
backup arround 8 hours.

OK, the BaSH script works very fast but there is a problem with it...

It take 100% CPU resources... and it is nearly impossibel to connect
via IMAP to the server since the connection times out... :-(

Q1: Is there someone with a similar problem and can help out?

Q2: Would it be better, if I install 2-3 very small additional storage
servers (they need only 147 GByte Raid-1 storage each) which hold
only the huge Mailarchives and serv them over a separated GigaBit)
link? (I can install a second NIC in the imap server connect
to a 5 port Switch for the backup server and storage servers)

Note: The backup server is an Athlon XP1800+/512GB with Adaptec 29160
and has 6 x 74 GByte of storage for 6 weeks. (one HDD per week)

Thanks, Greetings and nice Day
Michelle Konzack
Systemadministrator
24V Electronic Engineer
Tamay Dogan Network
Debian GNU/Linux Consultant


--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
+49/177/9351947 50, rue de Soultz MSN LinuxMichi
+33/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)
 
Old 05-25-2008, 03:49 PM
Thomas Goirand
 
Default Backup of an courier-imap mailarchive

Michelle Konzack wrote:
> Hello,
>
> On my "business" mailserver (courier-imap) I have 43 users (some of them
> as On-Line-Archive), together over 42.000 mailfolders and 17.000.000
> messages stored and each month we get 1800 mailfolders and 150.000
> messages more. The total volume is arround 180 GByte. (real not blocks)
>
> Currently I run a daily plus weekly cronjob with a selfmade script which
> check first the /cur/ directories and if it does not match a previously
> created MD5 + directory listing it will backuped. Daily incremental and
> weekly full.

Why did you prefer such solution over something like dirvish for
example? It does rsync incremental backup every day, and I don't believe
it would take 100% of your CPU.

Thomas


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-25-2008, 05:51 PM
Michelle Konzack
 
Default Backup of an courier-imap mailarchive

Am 2008-05-25 23:49:31, schrieb Thomas Goirand:
> Why did you prefer such solution over something like dirvish for
> example? It does rsync incremental backup every day, and I don't believe
> it would take 100% of your CPU.

Because "rsync" has not only eaten my CPU to 100% but also the Memory...

Thanks, Greetings and nice Day
Michelle Konzack
Systemadministrator
24V Electronic Engineer
Tamay Dogan Network
Debian GNU/Linux Consultant


--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
+49/177/9351947 50, rue de Soultz MSN LinuxMichi
+33/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)
 
Old 05-25-2008, 06:24 PM
Joerg Backschues
 
Default Backup of an courier-imap mailarchive

Michelle Konzack wrote:


On my "business" mailserver (courier-imap) I have 43 users (some of them
as On-Line-Archive), together over 42.000 mailfolders and 17.000.000
messages stored and each month we get 1800 mailfolders and 150.000
messages more. The total volume is arround 180 GByte. (real not blocks)

Currently I run a daily plus weekly cronjob with a selfmade script which
check first the /cur/ directories and if it does not match a previously
created MD5 + directory listing it will backuped. Daily incremental and
weekly full.

The daily incremental backup take arround 5 hours and the weekly full
backup arround 8 hours.

OK, the BaSH script works very fast but there is a problem with it...

It take 100% CPU resources... and it is nearly impossibel to connect
via IMAP to the server since the connection times out... :-(


The best way is it to make backups of your filesystem based on block
level technologies e.g. with LVM. Snapshots based on block level doesn't
care about file and directory quantity.


There was an article - sorry, in German only - in the iX-Magazin in 2004
(<http://www.heise.de/kiosk/archiv/ix/04/10/136_Im_Blitzlicht>).


--
Greetings
Jörg Backschues


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-25-2008, 06:41 PM
Maarten Vink
 
Default Backup of an courier-imap mailarchive

Op 25 mei 2008, om 19:51 heeft Michelle Konzack het volgende geschreven:


Am 2008-05-25 23:49:31, schrieb Thomas Goirand:

Why did you prefer such solution over something like dirvish for
example? It does rsync incremental backup every day, and I don't
believe

it would take 100% of your CPU.


Because "rsync" has not only eaten my CPU to 100% but also the
Memory...


Rsync tends to use a lot of memory for larger filesystems because it
loads the entire filesystem tree in memory before connecting to the
destination host. There are two ways to fix this:


1) Build a script that splits up the backup in several rsync runs, for
example by starting rsync for each user instead of the entire mailspool
2) Try rsync 3.0; it has a "quickstart" mode that starts synchronizing
files before the entire filesystem is read, saving huge amounts of
memory. Rsync 3 is available in the backports collection.


Maarten


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-26-2008, 01:44 PM
Michelle Konzack
 
Default Backup of an courier-imap mailarchive

Hello Maarten,

Am 2008-05-25 20:41:03, schrieb Maarten Vink:
> Rsync tends to use a lot of memory for larger filesystems because it
> loads the entire filesystem tree in memory before connecting to the
> destination host. There are two ways to fix this:
>
> 1) Build a script that splits up the backup in several rsync runs, for
> example by starting rsync for each user instead of the entire mailspool

This is already done but there are 4 "archive" user which hold over 90%
of the whole Mailstorage...

> 2) Try rsync 3.0; it has a "quickstart" mode that starts synchronizing
> files before the entire filesystem is read, saving huge amounts of
> memory. Rsync 3 is available in the backports collection.

Thanks for the info, I will give it a try...

Thanks, Greetings and nice Day
Michelle Konzack

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
+49/177/9351947 50, rue de Soultz MSN LinuxMichi
+33/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)
 
Old 05-26-2008, 02:08 PM
Mike Bird
 
Default Backup of an courier-imap mailarchive

On Mon May 26 2008 06:44:51 Michelle Konzack wrote:
> Am 2008-05-25 20:41:03, schrieb Maarten Vink:
> > 1) Build a script that splits up the backup in several rsync runs, for
> > example by starting rsync for each user instead of the entire mailspool
>
> This is already done but there are 4 "archive" user which hold over 90%
> of the whole Mailstorage...

We have a lot of daily rsyncs, one of them 360GB, and they only
take a few minutes unless a lot has changed. Could this be an
ext3 filesystem built long ago without dir_index?

--Mike Bird


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-26-2008, 02:20 PM
"Andrew McGlashan"
 
Default Backup of an courier-imap mailarchive

Hi,

Mike Bird wrote:

On Mon May 26 2008 06:44:51 Michelle Konzack wrote:

Am 2008-05-25 20:41:03, schrieb Maarten Vink:

1) Build a script that splits up the backup in several rsync runs,
for example by starting rsync for each user instead of the entire
mailspool


This is already done but there are 4 "archive" user which hold over
90% of the whole Mailstorage...


We have a lot of daily rsyncs, one of them 360GB, and they only
take a few minutes unless a lot has changed. Could this be an
ext3 filesystem built long ago without dir_index?


Okay, I just checked mine with dumpe2fs and it showed that whilst I didn't
specify dir_index, it was a feature in use.


When did this start being a standard feature?

I know that I do rsync backups on the hour as it works extremely well, I may
increase the frequency given how well it performs for my system [which isn't
high end by any stretch]. I am thinking about doing 4 rsync backups per
hour with each one operating on a different backup area so that I can easily
go back to any file changed or deleted less than 1 hour ago, I may even do 8
over 2 hours. Certainly rsynce works very well for my needs on my ext3 file
system. I'm not currently concerned about using anything that is Maildir
specific (and aware), but that might come later.


Kind Regards
AndrewM

Andrew McGlashan
Broadband Solutions now including VoIP

Current Land Line No: 03 9912 0504
Mobile: 04 2574 1827 Fax: 03 9012 2178

National No: 1300 85 3804

Affinity Vision Australia Pty Ltd
http://www.affinityvision.com.au
http://adsl2choice.net.au

In Case of Emergency -- http://www.affinityvision.com.au/ice.html



--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-26-2008, 03:13 PM
Mike Bird
 
Default Backup of an courier-imap mailarchive

On Mon May 26 2008 07:20:32 Andrew McGlashan wrote:
> Okay, I just checked mine with dumpe2fs and it showed that whilst I didn't
> specify dir_index, it was a feature in use.
>
> When did this start being a standard feature?

e2fsprogs (1.39-1) unstable; urgency=low

* New upstream version
* Fix debugfs's dump_unused command so it will not core dump on
filesystems with a 64k blocksize
* Clarified and improved man pages, including spelling errors
(Closes: #368392, #368393, #368394, #368179)
* New filesystems are now created with directory indexing and
on-line resizing enabled by default
* Fix previously mangled wording in an older Debian changelog entry
* Fix doc-base pointer to the top-level html file (Closes: #362544, #362970)

-- Theodore Y. Ts'o <tytso@mit.edu> Mon, 29 May 2006 11:07:53 -0400

--Mike Bird


--
To UNSUBSCRIBE, email to debian-isp-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
 
Old 05-26-2008, 04:06 PM
Michelle Konzack
 
Default Backup of an courier-imap mailarchive

Am 2008-05-26 08:13:22, schrieb Mike Bird:
> e2fsprogs (1.39-1) unstable; urgency=low
>
> * New upstream version
> * Fix debugfs's dump_unused command so it will not core dump on
> filesystems with a 64k blocksize
> * Clarified and improved man pages, including spelling errors
> (Closes: #368392, #368393, #368394, #368179)
> * New filesystems are now created with directory indexing and
> on-line resizing enabled by default
> * Fix previously mangled wording in an older Debian changelog entry
> * Fix doc-base pointer to the top-level html file (Closes: #362544, #362970)
>
> -- Theodore Y. Ts'o <tytso@mit.edu> Mon, 29 May 2006 11:07:53 -0400

My filesystem was created under Sarge and even Etch does not
include this feature, unless you have installed a backport...

Maybe I will reinitializer the filesystem...

Thanks, Greetings and nice Day
Michelle Konzack
Systemadministrator
24V Electronic Engineer
Tamay Dogan Network
Debian GNU/Linux Consultant


--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack Apt. 917 ICQ #328449886
+49/177/9351947 50, rue de Soultz MSN LinuxMichi
+33/6/61925193 67100 Strasbourg/France IRC #Debian (irc.icq.com)
 

Thread Tools




All times are GMT. The time now is 04:46 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org