Backup solution to backup /var/spool/imap above 150GB data
 

Hi

We have* an issue with one of our clients , they have a mail server with the* /var/spool/imap partition as 150 GB . They need to take differential* backup on to /backup partition ( a different HDD* of total 250 GB space )* . We have tried dar , rsync, rdiff and impasync . But its is not sufficing the need as to take a lot of time and consumes a lot of I/O .


Is there any back up solution that you can think of , that can work in this situation* - open source or proprietary* * *
--
Regards
Agnello D'souza



_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

Dne 24.2.2010 10:00, Agnello George napsal(a):
> Hi
>
> We have an issue with one of our clients , they have a mail server with
> the /var/spool/imap partition as 150 GB . They need to take differential
> backup on to /backup partition ( a different HDD of total 250 GB space ) .
> We have tried dar , rsync, rdiff and impasync . But its is not sufficing the
> need as to take a lot of time and consumes a lot of I/O .
>
> Is there any back up solution that you can think of , that can work in this
> situation - open source or proprietary

Is seems to me, that you are using mbox format. So, differential backup
is hard to achieve. Migrate to maildir, every mail is a file, easy to
backup differentially.
David
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

On Wed, Feb 24, 2010 at 10:38:32AM +0100, David Hrbáč wrote:
> Dne 24.2.2010 10:00, Agnello George napsal(a):
> > We have an issue with one of our clients , they have a mail server with
> > the /var/spool/imap partition as 150 GB . They need to take differential
> > backup on to /backup partition ( a different HDD of total 250 GB space ) .
> > We have tried dar , rsync, rdiff and impasync . But its is not sufficing the
> > need as to take a lot of time and consumes a lot of I/O .
> >
> > Is there any back up solution that you can think of , that can work in this
> > situation - open source or proprietary
>
> Is seems to me, that you are using mbox format. So, differential backup
> is hard to achieve. Migrate to maildir, every mail is a file, easy to
> backup differentially.

rsync and rdiff should handle mbox format okay though. Though I agree Maildir
is generally nicer for differential backups.

Agnello, how long is "a lot of time"? A backup is always going to have to walk
the entire tree and checksum (or at least stat) every file, so there's a minimum
cost you're always going to have. How long does a 'find /var/spool/imap -ls'
take, for instance?

You might want to try brackup (http://code.google.com/p/brackup/). For very
large trees of relatively small files it seems to significantly out-perform
rsync-based backups. I've got brackup packages in my repository (see
http://www.openfusion.net/linux/openfusion_rpm_repository).

Cheers,
Gavin

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

On 02/24/2010 11:21 AM, Agnello George wrote:
> -rw------- 1 cyrus mail 4805500 Feb 24 16:23 4799.
> -rw------- 1 cyrus mail 22920 Feb 24 16:23 cyrus.cache
> -rw------- 1 cyrus mail 204 Dec 10 16:27 cyrus.header
> -rw------- 1 cyrus mail 896 Feb 24 16:23 cyrus.index
> -rw------- 1 cyrus mail 8669 Feb 24 11:28 cyrus.squat
>
> this is Just a very small user and a example

About 90% of your problem is already solved here, you are using cyrus
which has built in mail level replication. All you need to do is setup a
lvm volume away from this main store and run your mail replica over to
it. then just backup using whatever tools you want.

Free win you get is online failover, backup in whatever manner you want!

- KB
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

On Wed, Feb 24, 2010 at 05:06:08PM +0530, Agnello George wrote:
> On Wed, Feb 24, 2010 at 4:57 PM, Eero Volotinen <eero.volotinen@iki.fi>wrote:
> > 2010/2/24 Agnello George <agnello.dsouza@gmail.com>:
> > > On Wed, Feb 24, 2010 at 4:13 PM, Gavin Carr <gavin@openfusion.com.au>
> > wrote:
> > >> You might want to try brackup (http://code.google.com/p/brackup/). For
> > >> very
> > >> large trees of relatively small files it seems to significantly
> > >> out-perform
> > >> rsync-based backups. I've got brackup packages in my repository (see
> > >> http://www.openfusion.net/linux/openfusion_rpm_repository).
> > >>
> > >> Cheers,
> > >> Gavin
> > >>
> > > is it possible with " brackup " to back it up to a different server on
> > > the same lan instead of /backup . Is there any documentation on the
> > > same .
> >
> > rsync or rdiff-backup works on local disk or remote disk.(and other
> > backup methods too!)
>
> Does http://code.google.com/p/brackup/ also work in on remote machines .

Brackup will backup to local disk, or remotely to ftp, sftp, Amazon S3, or
Rackspace CloudFiles targets/servers. So yes, on a lan you can backup over
ftp or sftp just fine.

Re docs, install brackup, 'man Brackup::Manual::Overview'. I've also written
a few blog posts on it: http://www.openfusion.net/tags/brackup.

Cheers,
Gavin

_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

Agnello George wrote:
> Hi
>
> We have an issue with one of our clients , they have a mail server with
> the /var/spool/imap partition as 150 GB . They need to take
> differential backup on to /backup partition ( a different HDD of total
> 250 GB space ) . We have tried dar , rsync, rdiff and impasync . But
> its is not sufficing the need as to take a lot of time and consumes a
> lot of I/O .
>
> Is there any back up solution that you can think of , that can work in
> this situation - open source or proprietary

If you are just concerned about a single disk failure you could set up RAID1 on
the disks (with some downtime to rebuild...) to keep the copy in realtime with
little loss of speed.

Rsync should work as well as anything for snapshots but you might need to update
to a 3.x version to speed up handling large numbers of files. The 2.x version
included in Centos will read the entire directory tree into memory before
starting the comparisons and copies. The rpmforge repo has a packaged 3.0.7
version but I haven't tried it.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

On 02/24/2010 01:07 PM, Agnello George wrote:
> yes just spoke to my senior and confrimed that this was alreday tried
> out a delayed replication is possible .
> but the current suitation is we need to take backup on the same
> server on a different partition /backup :(

you can replicate to a local mail store as well. just make sure you put
it on a block device that is suiteable and fits in with the rest of your
backup strategy. And if you put it in an isolated enough place on the
block dev, it wont contest with the users access.

- KB
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

Agnello George wrote:
> Hi
>
> We have an issue with one of our clients , they have a mail server with
> the /var/spool/imap partition as 150 GB . They need to take differential
> backup on to /backup partition ( a different HDD of total 250 GB space ) .

You've stated things in terms of solutions. You may possibly get better
answers if you state your goal. There is some capability you are
trying to achieve. Tell us what that is, and you may make more progress.

IOW, what is the purpose of the backup? As one mentioned, RAID may
handle your needs.

Mike
--
p="p=%c%s%c;main(){printf(p,34,p,34);}";main(){pri ntf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

On 2/24/2010 1:31 PM, Eero Volotinen wrote:

>>>
>>> We have an issue with one of our clients , they have a mail server with
>>> the /var/spool/imap partition as 150 GB . They need to take differential
>>> backup on to /backup partition ( a different HDD of total 250 GB space ) .
>>
>> You've stated things in terms of solutions. You may possibly get better
>> answers if you state your goal. There is some capability you are
>> trying to achieve. Tell us what that is, and you may make more progress.
>>
>> IOW, what is the purpose of the backup? As one mentioned, RAID may
>> handle your needs.
>
> Err.. raid is NOT backup solution.

Neither is a snapshot in another location on the same machine. But both
will cover the most likely thing to fail, with raid doing it
transparently, the snapshot losing data from the time the last snapshot
copy happened. Usually what you want is raid _and_ a history of backups
kept elsewhere.

--
Les Mikesell
lesmikesell@gmail.com
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Backup solution to backup /var/spool/imap above 150GB data
 

Eero Volotinen wrote:
> 2010/2/24 Mike McCarty <Mike.McCarty@sbcglobal.net>:
>> Agnello George wrote:
>>> Hi
>>>
>>> We have an issue with one of our clients , they have a mail server with
>>> the /var/spool/imap partition as 150 GB . They need to take differential
>>> backup on to /backup partition ( a different HDD of total 250 GB space ) .
>> You've stated things in terms of solutions. You may possibly get better
>> answers if you state your goal. There is some capability you are
>> trying to achieve. Tell us what that is, and you may make more progress.
>>
>> IOW, what is the purpose of the backup? As one mentioned, RAID may
>> handle your needs.
>
> Err.. raid is NOT backup solution.

Of course not. RAID is a means to achieve availability,
which may be his goal. Karanbir already stated a means to do
what he seemed to want, but it seemed not to satisfy his needs.

Unless the query is placed in terms of requirements and goals,
instead of solutions, it'll be difficult to achieve satisfactory
results.

The purpose of backup is some degree of disaster recovery, and
perhaps also migration. If that's truly his goal, then ISTM
that Karanbir suggested a viable solution to achieving avialability
while also performing backup, by doing on-the-fly duplication
of the data onto another file system which can then be backed up
at liesure.

Doing so in a manner which ensures a true snapshot may be more
difficult to achieve, while still ensuring availability. I normally
do my backups in single user mode with all file systems mounted read
only, except the one to receive the backup. That of course precludes
availability during the backup procedure.

That's why I would like to see what he actually wants to achieve,
instead of how he has chosen to go about it.

Mike
--
p="p=%c%s%c;main(){printf(p,34,p,34);}";main(){pri ntf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos