Linux Archive

Linux Archive (http://www.linux-archive.org/)
-   CentOS (http://www.linux-archive.org/centos/)
-   -   s3 as mysql directory (http://www.linux-archive.org/centos/708853-s3-mysql-directory.html)

Nux! 10-01-2012 06:39 PM

s3 as mysql directory
 
On 01.10.2012 19:24, Tim Dunphy wrote:
> Hello list,
>
> I am soliciting opinion here as opposed technical help with an idea
> I
> have. I've setup a bacula backup system on an AWS volume. Bacula
> stores a
> LOT of information in it's mysql database (in my setup, you can also
> use
> postgres or sqlite if you chose). Since I've started doing this I
> notice
> that the mysql data directory has swelled to over 700GB! That's quite
> a lot
> and its' easting up valuable disk space.
>
> So I had an idea. What about uses the fuse based s3fs to mount an S3
> bucket on the local filesystem and use that as your mysql data dir?
> In
> other words mount your s3 bucket on /var/lib/mysql
>
> I used this article to setup the s3fs file system
>
>
> http://benjisimon.blogspot.com/2011/01/setting-up-s3-backup-solution-on-centos.html
>
> And everything went as planned. So my question to you dear listers is
> if I
> do start using a locally mounted s3 bucket as my mysqld data dir,
> will
> performance of the database be acceptable? If so, why? If not are
> there any
> other reasons why it would NOT be a good idea to do this?
>
> The steps I have in mind are basically this:
>
> 1) mysqldump --all-databases > alldb.sql
> 2) stop mysql
> 3) rm -rf /var/lib/mysql/*
> 4) mount the s3 bucket on /var/lib/mysql
> 5) start mysql
> 6) restore the alldb.sql dump
>
>
> Thanks for your opinions on this!
>
> Tim
> 4)

What a wild idea! :-)
I don't think it will work; S3 is not really a filesystem afaik.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

Tim Dunphy 10-01-2012 06:52 PM

s3 as mysql directory
 
>
> What a wild idea! :-)
> I don't think it will work; S3 is not really a filesystem afaik.
>

Hey, well thanks for the compliment! Thing is once you mount your s3 bucket
to your local file system via fuse and the s3fs app, your S3 bucket will
ACT like a local filesystem in every way. If you read the link that I
included in the OP you'll see that the guy uses it as his backup
filesystem. The only complaint is that it can be a tad slow to do things
this way. Now there are clever ways around this and speed things up. So
addressing the speed issue, my question to the list is there any reason it
won't work to use a locally mounted S3 bucket as your mysql data dir?
Considering that s3fs makes your bucket act like a regular volume.

Any thoughts?

Thanks!

Tim


On Mon, Oct 1, 2012 at 2:39 PM, Nux! <nux@li.nux.ro> wrote:

> On 01.10.2012 19:24, Tim Dunphy wrote:
> > Hello list,
> >
> > I am soliciting opinion here as opposed technical help with an idea
> > I
> > have. I've setup a bacula backup system on an AWS volume. Bacula
> > stores a
> > LOT of information in it's mysql database (in my setup, you can also
> > use
> > postgres or sqlite if you chose). Since I've started doing this I
> > notice
> > that the mysql data directory has swelled to over 700GB! That's quite
> > a lot
> > and its' easting up valuable disk space.
> >
> > So I had an idea. What about uses the fuse based s3fs to mount an S3
> > bucket on the local filesystem and use that as your mysql data dir?
> > In
> > other words mount your s3 bucket on /var/lib/mysql
> >
> > I used this article to setup the s3fs file system
> >
> >
> >
> http://benjisimon.blogspot.com/2011/01/setting-up-s3-backup-solution-on-centos.html
> >
> > And everything went as planned. So my question to you dear listers is
> > if I
> > do start using a locally mounted s3 bucket as my mysqld data dir,
> > will
> > performance of the database be acceptable? If so, why? If not are
> > there any
> > other reasons why it would NOT be a good idea to do this?
> >
> > The steps I have in mind are basically this:
> >
> > 1) mysqldump --all-databases > alldb.sql
> > 2) stop mysql
> > 3) rm -rf /var/lib/mysql/*
> > 4) mount the s3 bucket on /var/lib/mysql
> > 5) start mysql
> > 6) restore the alldb.sql dump
> >
> >
> > Thanks for your opinions on this!
> >
> > Tim
> > 4)
>
> What a wild idea! :-)
> I don't think it will work; S3 is not really a filesystem afaik.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>



--
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos

joel billy 10-01-2012 07:11 PM

s3 as mysql directory
 
Hi,

700 GB is quite large. What was the amount of data you would have
backed up till this point ? Could be that the catalog data is building
up, have you looked at
http://www.bacula.org/5.0.x-manuals/en/main/main/Catalog_Maintenance.html

While your setup in theory should work, have you tested restores.
Search through catalogs might be slower than you expect for 700 GB

- jb

On 10/1/12, Tim Dunphy <bluethundr@gmail.com> wrote:
> Hello list,
>
> I am soliciting opinion here as opposed technical help with an idea I
> have. I've setup a bacula backup system on an AWS volume. Bacula stores a
> LOT of information in it's mysql database (in my setup, you can also use
> postgres or sqlite if you chose). Since I've started doing this I notice
> that the mysql data directory has swelled to over 700GB! That's quite a lot
> and its' easting up valuable disk space.
>
> So I had an idea. What about uses the fuse based s3fs to mount an S3
> bucket on the local filesystem and use that as your mysql data dir? In
> other words mount your s3 bucket on /var/lib/mysql
>
> I used this article to setup the s3fs file system
>
> http://benjisimon.blogspot.com/2011/01/setting-up-s3-backup-solution-on-centos.html
>
> And everything went as planned. So my question to you dear listers is if I
> do start using a locally mounted s3 bucket as my mysqld data dir, will
> performance of the database be acceptable? If so, why? If not are there any
> other reasons why it would NOT be a good idea to do this?
>
> The steps I have in mind are basically this:
>
> 1) mysqldump --all-databases > alldb.sql
> 2) stop mysql
> 3) rm -rf /var/lib/mysql/*
> 4) mount the s3 bucket on /var/lib/mysql
> 5) start mysql
> 6) restore the alldb.sql dump
>
>
> Thanks for your opinions on this!
>
> Tim
> 4)
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


All times are GMT. The time now is 08:42 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.