Experts,

What happens if the disk hosting an external journal of a filesytem
running with data=journal goes bust.

The Backstory ...

I have been batteling with filesystem performance for some time
now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems.

Recently we added an SSD to our setup and have moved all the journals
to this ssd. This has dramatically improved performance and
especially reduced the interdependence between performance of
different partitions hosted on the same RAID.

http://insights.oetiker.ch/linux/external-journal-on-ssd.html

I realy like the performance of this new setup, but I am not all
that sure about the data security aspects of it. Especially after
reading

http://www.cs.wisc.edu/adsl/Publications/sfa-dsn05.pdf

which suggests that damaged journals are the worst that can happen
to ext3.

any insights on this?

cheers
tobi


--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

On Wed, 10 Sep 2008 13:30:45 +0200, Tobias Oetiker wrote:

> What happens if the disk hosting an external journal of a filesytem
> running with data=journal goes bust.

Probably the same as if the journal was on the same disk, going bust. :-)
Or rather :-( as this can indeed get pretty ugly.
With ext3 you can always fall back to mounting as ext2 and at least try to
recover as much as possible.

> Recently we added an SSD to our setup and have moved all the journals to
> this ssd. This has dramatically improved performance and especially
> reduced the interdependence between performance of different partitions
> hosted on the same RAID.

That is one of the great SSD uses, yes.

> http://insights.oetiker.ch/linux/external-journal-on-ssd.html

Very interesting, thanks! I was planning to do the same but waiting for
the Intel SSDs to come to market or the large OZCs to come down in price,
whatever happened first..

> I realy like the performance of this new setup, but I am not all that sure
> about the data security aspects of it. Especially after reading
>
> http://www.cs.wisc.edu/adsl/Publications/sfa-dsn05.pdf
>
> which suggests that damaged journals are the worst that can happen to
> ext3.

True, a borked journal is bad but with the SSD you should actually have
*less* chance of corruption (of the type mentioned in the paper), since
the wear-leveling should keep the journal blocks alive without the
file system/block layer noticing. At least in theory.. :-D

You may also find this interesting:
http://labs.google.com/papers/disk_failures.html

Holger


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Tobias Oetiker wrote:
> Experts,
>
> What happens if the disk hosting an external journal of a filesytem
> running with data=journal goes bust.
>
> The Backstory ...
>
> I have been batteling with filesystem performance for some time
> now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems.
>
> Recently we added an SSD to our setup and have moved all the journals
> to this ssd. This has dramatically improved performance and
> especially reduced the interdependence between performance of
> different partitions hosted on the same RAID.
>
> http://insights.oetiker.ch/linux/external-journal-on-ssd.html

How does this compare to putting journals on a separate non-ssd device?

-Eric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Another followup..

On Wed, 10 Sep 2008 13:30:45 +0200, Tobias Oetiker wrote:

> Recently we added an SSD to our setup and have moved all the journals to
> this ssd. This has dramatically improved performance and especially
> reduced the interdependence between performance of different partitions
> hosted on the same RAID.
>
> http://insights.oetiker.ch/linux/external-journal-on-ssd.html

You mention that you chose data=journal, i.e. full journaling. Have you
tried ordered mode as well? This should still yield a significant
performance win because of reduced head movement and faster metadata
writes. It may or may not be faster depending on the size of the
written data itself..I'm just curious if you tested this.

thanks
Holger


_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Hi Eric,

I have not tested this, but since we are putting about 16 different
journals on this one ssd, I would assume that the loss through
seeking between the journals would be pretty bad, and again bring
back that inter-filesystem-dependency we were trying to loose with
this measure.

cheers
tobi

Today Eric Sandeen wrote:

> Tobias Oetiker wrote:
> > Experts,
> >
> > What happens if the disk hosting an external journal of a filesytem
> > running with data=journal goes bust.
> >
> > The Backstory ...
> >
> > I have been batteling with filesystem performance for some time
> > now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems.
> >
> > Recently we added an SSD to our setup and have moved all the journals
> > to this ssd. This has dramatically improved performance and
> > especially reduced the interdependence between performance of
> > different partitions hosted on the same RAID.
> >
> > http://insights.oetiker.ch/linux/external-journal-on-ssd.html
>
> How does this compare to putting journals on a separate non-ssd device?
>
> -Eric
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Tobias Oetiker wrote:
> Hi Eric,
>
> I have not tested this, but since we are putting about 16 different
> journals on this one ssd, I would assume that the loss through
> seeking between the journals would be pretty bad, and again bring
> back that inter-filesystem-dependency we were trying to loose with
> this measure.

Ah, ok - I missed that you had several journals on one device.

Thanks,
-Eric

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Look at:*http://www.fusionio.com/Products.aspx
At 120K IOPS @1K blocks, it should make for a very good journaling device.

It's not an SSD per se; it bypasses old disk controllers altogether (very innovative block device design).

The block device layer and hardware are tailored for NAND failure*idiosyncrasies... which results in their data loss is less than any available SSD or rotating disk.

Put two together in a RAID1 configuration to compensate for device failures (assure you have 2 PCIe x8 slots available).
Chris
On Wed, Sep 10, 2008 at 10:05 AM, Tobias Oetiker <tobi@oetiker.ch> wrote:

Hi Eric,



I have not tested this, but since we are putting about 16 different

journals on this one ssd, I would assume that the loss through

seeking between the journals would be pretty bad, and again bring

back that inter-filesystem-dependency we were trying to loose with

this measure.



cheers

tobi



Today Eric Sandeen wrote:



> Tobias Oetiker wrote:

> > Experts,

> >

> > What happens if the disk hosting an external journal of a filesytem

> > running with data="" goes bust.

> >

> > The Backstory ...

> >

> > I have been batteling with filesystem performance for some time

> > now. Our setup is a HW Raid(6) with LVM on top and ext3 filesytems.

> >

> > Recently we added an SSD to our setup and have moved all the journals

> > to this ssd. This has dramatically improved performance and

> > especially reduced the interdependence between performance of

> > different partitions hosted on the same RAID.

> >

> > *http://insights.oetiker.ch/linux/external-journal-on-ssd.html

>

> How does this compare to putting journals on a separate non-ssd device?

>

> -Eric

>

>



--

Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland

http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900



_______________________________________________

Ext3-users mailing list

Ext3-users@redhat.com

https://www.redhat.com/mailman/listinfo/ext3-users



_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Hi Chris,

Yesterday Chris Worley wrote:

> Note that I do have one to experiment with.
> What's a good way to measure journal performance, and/or in what cases do
> you need a faster journal (i.e. an EXT3 atop an MD device with big block
> stripes)?
>
> Chris

Well, the 'problem' we had to solve was the following:

setup:

- large HW raid6 array
- lvm on top
- many ext3 partitions

when there was a lot of write or meta data update activity on one
partition, performance on all other partitions went to 0.
(processes hanging for 10-20 seconds as soon as they accessed the
filesystem). I am sure that there is a bad-bad bug in the linux
kernel somewhere which is causing this, but all the upgrading and
patching did not help, the condition remained.

Until we moved the journals off to that external ssd.

Now I can copy partition A over to partition B and the server
remains nicely responsive. I am atributing that to the external
journal.

Obviously I would like to know how bad we are going to be had when
the ssd dies.

cheers
tobi

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

On Sep 10, 2008 18:05 +0200, Tobias Oetiker wrote:
> I have not tested this, but since we are putting about 16 different
> journals on this one ssd, I would assume that the loss through
> seeking between the journals would be pretty bad, and again bring
> back that inter-filesystem-dependency we were trying to loose with
> this measure.

The cost of putting the journals on 16 separate, relatively small
disk devices would probably be comparable to the cost of the SSD
and not have a single point of failure. The journal does mostly
linear IO, so performance is probably equal or better.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users

Folks,

Yesterday Andreas Dilger wrote:

> On Sep 10, 2008 18:05 +0200, Tobias Oetiker wrote:
> > I have not tested this, but since we are putting about 16 different
> > journals on this one ssd, I would assume that the loss through
> > seeking between the journals would be pretty bad, and again bring
> > back that inter-filesystem-dependency we were trying to loose with
> > this measure.
>
> The cost of putting the journals on 16 separate, relatively small
> disk devices would probably be comparable to the cost of the SSD
> and not have a single point of failure. The journal does mostly
> linear IO, so performance is probably equal or better.

You are telling me things that I am aware of. The reason I wrote to
this group is to figure what would happen to an ext3 fs when the
external journal was lost, especially what happens when it is lost
on a filesystem where 'data=journal' is set.

Because if it is catastrophic, then it basically means that the
journal has to reside on a device that is as secure as to rest of
the data, meaning that if the data is on RAID6 then the journal
should be on RAID6 too.

What I am hoping for, is that someone tells me, that in the case of
'data=journal' the loss would only be the material that is still in
the journal (eg 30 seconds worth of data) and the rest of the fs
would have a fair chance of being recoverd with fsck.

cheers
tobi


--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch tobi@oetiker.ch ++41 62 775 9902 / sb: -9900

_______________________________________________
Ext3-users mailing list
Ext3-users@redhat.com
https://www.redhat.com/mailman/listinfo/ext3-users