Ext4 and xfs problems in dm-thin on allocation and discard
 

Hello all

I am doing some testing of dm-thin on kernel 3.4.2 and latest lvm
from source (the rest is Ubuntu Precise 12.04).

There are a few problems with ext4 and (different ones with) xfs



I am doing this:

dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync

lvs

rm zeroes #optional

dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync* #again

lvs

rm zeroes #optional

...

dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync* #again

lvs

rm zeroes

fstrim /mnt/mountpoint

lvs



On ext4 the problem is that it always reallocates blocks at
different places, so you can see from lvs that space occupation in
the pool and thinlv increases at each iteration of dd, again and
again, until it has allocated the whole thin device (really 100% of
it). And this is true regardless of me doing rm or not between one
dd and the other.

The other problem is that by doing this, ext4 always gets the worst
performance from thinp, about 140MB/sec on my system, because it is
constantly allocating blocks, instead of 350MB/sec which should have
been with my system if it used already allocated regions (see below
compared to xfs). I am on an MD raid-5 of 5 hdds.

I could suggest to add a "thinp mode" mount option to ext4 affecting
the allocator, so that it tries to reallocate recently used and
freed areas and not constantly new areas. Note that mount -o discard
does work and prevents allocation bloating, but it still always gets
the worst write performances from thinp. Alternatively thinp could
be improved so that block allocation is fast :-P (*)

However, good news is that fstrim works correctly on ext4, and is
able to drop all space allocated by all dd's. Also mount -o discard
works.



On xfs there is a different problem.

Xfs apparently correctly re-uses the same blocks so that after the
first write at 140MB/sec, subsequent overwrites of the same file are
at full speed such as 350MB/sec (same speed as with non-thin lvm),
and also you don't see space occupation going up at every iteration
of dd, either with or without rm in-between the dd's. [ok actually
now retrying it needed 3 rewrites to stabilize allocation...
probably an AG count thing.]

However the problem with XFS is that discard doesn't appear to work.
Fstrim doesn't work, and neither does "mount -o discard ... + rm
zeroes" . There is apparently no way to drop the allocated blocks,
as seen from lvs. This is in contrast to what it is written here

http://xfs.org/index.php/FITRIM/discard
which declare fstrim and mount -o discard to be working.

Please note that since I am above MD raid5 (I believe this is the
reason), the passdown of discards does not work, as my dmesg says:

[160508.497879] device-mapper: thin: Discard unsupported by data
device (dm-1): Disabling discard passdown.

but AFAIU, unless there is a thinp bug, this should not affect the
unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
able to do that.



(*) Strange thing is that write performance appears to be roughly
the same for default thin chunksize and for 1MB thin chunksize. I
would have expected thinp allocation to be faster with larger thin
chunksizes but instead it is actually slower (note that there are no
snapshots here and hence no CoW). This is also true if I set the
thinpool to not zero newly allocated blocks: performances are about
240 MB/sec then, but again they don't increase with larger
chunksizes, they actually decrease slightly with very large
chunksizes such as 16MB. Why is that?



Thanks for your help

S.



--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Mon, Jun 18, 2012 at 11:33:50PM +0200, Spelic wrote:
> Hello all
> I am doing some testing of dm-thin on kernel 3.4.2 and latest lvm
> from source (the rest is Ubuntu Precise 12.04).
> There are a few problems with ext4 and (different ones with) xfs
>
> I am doing this:
> dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync
> lvs
> rm zeroes #optional
> dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync #again
> lvs
> rm zeroes #optional
> ...
> dd if=/dev/zero of=zeroes bs=1M count=1000 conv=fsync #again
> lvs
> rm zeroes
> fstrim /mnt/mountpoint
> lvs

[snip ext4 problems]

> On xfs there is a different problem.
> Xfs apparently correctly re-uses the same blocks so that after the
> first write at 140MB/sec, subsequent overwrites of the same file are
> at full speed such as 350MB/sec (same speed as with non-thin lvm),
> and also you don't see space occupation going up at every iteration
> of dd, either with or without rm in-between the dd's. [ok actually
> now retrying it needed 3 rewrites to stabilize allocation...
> probably an AG count thing.]

That's just a characteristic of the allocation algorithm. It's not
something that you see in day-to-day operation of the filesystem,
though, because you rarely remove and rewrite a file like this
repeatedly. So in the real world, performance will be more like ext4
when you are running workloads where you actually store data for
longer than a millisecond...

Expect that the 140MB/s number is the normal performance case,
because as soon as you take a snapshot, the overwrite requires new
blocks to be allocated in dm-thinp. You don't get thinp for nothing
- it has an associated performance cost as you are now finding
out....

> However the problem with XFS is that discard doesn't appear to work.
> Fstrim doesn't work, and neither does "mount -o discard ... + rm
> zeroes" . There is apparently no way to drop the allocated blocks,
> as seen from lvs. This is in contrast to what it is written here
> http://xfs.org/index.php/FITRIM/discard which declare fstrim and
> mount -o discard to be working.

I don't see why it wouldnt be if the underlying device supports it.
Have you looked at a block trace or an xfs event trace to see if
discards are being issued by XFS?

Are you getting messages like:

XFS: (dev) discard failed for extent [0x123,4096], error -5

in dmesg, or is fstrim seeing errors returned from the trim ioctl?

> Please note that since I am above MD raid5 (I believe this is the
> reason), the passdown of discards does not work, as my dmesg says:
> [160508.497879] device-mapper: thin: Discard unsupported by data
> device (dm-1): Disabling discard passdown.
> but AFAIU, unless there is a thinp bug, this should not affect the
> unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
> able to do that.

Does ext4 report that same error?

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Mon, Jun 18 2012 at 9:57pm -0400,
Dave Chinner <david@fromorbit.com> wrote:

> On Mon, Jun 18, 2012 at 11:33:50PM +0200, Spelic wrote:
>
> > Please note that since I am above MD raid5 (I believe this is the
> > reason), the passdown of discards does not work, as my dmesg says:
> > [160508.497879] device-mapper: thin: Discard unsupported by data
> > device (dm-1): Disabling discard passdown.
> > but AFAIU, unless there is a thinp bug, this should not affect the
> > unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
> > able to do that.
>
> Does ext4 report that same error?

That message says the underlying device doesn't support discards
(because it is an MD device). But the thinp device still has discards
enabled -- it just won't pass the discards down to the underlying data
device.

So yes, it'll happen with ext4 -- it is generated when the thin-pool
device is loaded (which happens independent of the filesystem that is
layered ontop).

The discards still inform the thin-pool that the corresponding extents
are no longer allocated.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Mon, 18 Jun 2012, Mike Snitzer wrote:

> Date: Mon, 18 Jun 2012 23:12:42 -0400
> From: Mike Snitzer <snitzer@redhat.com>
> To: Dave Chinner <david@fromorbit.com>
> Cc: Spelic <spelic@shiftmail.org>,
> device-mapper development <dm-devel@redhat.com>,
> linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
>
> On Mon, Jun 18 2012 at 9:57pm -0400,
> Dave Chinner <david@fromorbit.com> wrote:
>
> > On Mon, Jun 18, 2012 at 11:33:50PM +0200, Spelic wrote:
> >
> > > Please note that since I am above MD raid5 (I believe this is the
> > > reason), the passdown of discards does not work, as my dmesg says:
> > > [160508.497879] device-mapper: thin: Discard unsupported by data
> > > device (dm-1): Disabling discard passdown.
> > > but AFAIU, unless there is a thinp bug, this should not affect the
> > > unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
> > > able to do that.
> >
> > Does ext4 report that same error?
>
> That message says the underlying device doesn't support discards
> (because it is an MD device). But the thinp device still has discards
> enabled -- it just won't pass the discards down to the underlying data
> device.
>
> So yes, it'll happen with ext4 -- it is generated when the thin-pool
> device is loaded (which happens independent of the filesystem that is
> layered ontop).
>
> The discards still inform the thin-pool that the corresponding extents
> are no longer allocated.

So do I understand correctly that even though the discard came
through and thinp took advantage of it it still returns EOPNOTSUPP ?
This seems rather suboptimal. IIRC there was a discussion to add an
option to enable/disable sending discard in thinp target down
to the device.

So maybe it might be a bit smarter than that and actually
enable/disable discard pass through depending on the underlying
support, so we do not blindly send discard down to the device even
though it does not support it.

So we'll have three options:

pass through - always send discard down
backstop - never send discard down to the device
auto - send discard only if the underlying device supports it

What do you think ?

-Lukas

> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On 06/19/12 08:32, Lukáš Czerner wrote:


So do I understand correctly that even though the discard came
through and thinp took advantage of it it still returns EOPNOTSUPP ?
This seems rather suboptimal. IIRC there was a discussion to add an
option to enable/disable sending discard in thinp target down
to the device.


I'll ask this too...
do I understand correctly that dm-thin returns EOPNOTSUPP to the
filesystem layer even though it is using the discard to unmap blocks,
and at that point XFS stops sending discards down there (while ext4
keeps sending them)?


This looks like a bug of dm-thin to me. Discards are "supported" in such
a scenario.


Do you have a patch for dm-thin so to prevent it sending EOPTNOTSUPP ?

Thank you
S.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Tue, 19 Jun 2012, Spelic wrote:

> Date: Tue, 19 Jun 2012 13:29:55 +0200
> From: Spelic <spelic@shiftmail.org>
> To: Lukáš Czerner <lczerner@redhat.com>
> Cc: Mike Snitzer <snitzer@redhat.com>, Dave Chinner <david@fromorbit.com>,
> Spelic <spelic@shiftmail.org>,
> device-mapper development <dm-devel@redhat.com>,
> linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
>
> On 06/19/12 08:32, Lukáš Czerner wrote:
> >
> > So do I understand correctly that even though the discard came
> > through and thinp took advantage of it it still returns EOPNOTSUPP ?
> > This seems rather suboptimal. IIRC there was a discussion to add an
> > option to enable/disable sending discard in thinp target down
> > to the device.
>
> I'll ask this too...
> do I understand correctly that dm-thin returns EOPNOTSUPP to the filesystem
> layer even though it is using the discard to unmap blocks, and at that point
> XFS stops sending discards down there (while ext4 keeps sending them)?
>
> This looks like a bug of dm-thin to me. Discards are "supported" in such a
> scenario.
>
> Do you have a patch for dm-thin so to prevent it sending EOPTNOTSUPP ?

Yes, definitely this behaviour need to change in dm-thin. I do not
have a path, it was merely a proposal how thing could be done. Not
sure what Mike and rest of the dm folks think about this.

-Lukas

>
> Thank you
> S.
> --
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Tue, Jun 19 2012 at 2:32am -0400,
Lukáš Czerner <lczerner@redhat.com> wrote:

> On Mon, 18 Jun 2012, Mike Snitzer wrote:
>
> > Date: Mon, 18 Jun 2012 23:12:42 -0400
> > From: Mike Snitzer <snitzer@redhat.com>
> > To: Dave Chinner <david@fromorbit.com>
> > Cc: Spelic <spelic@shiftmail.org>,
> > device-mapper development <dm-devel@redhat.com>,
> > linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> > Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
> >
> > On Mon, Jun 18 2012 at 9:57pm -0400,
> > Dave Chinner <david@fromorbit.com> wrote:
> >
> > > On Mon, Jun 18, 2012 at 11:33:50PM +0200, Spelic wrote:
> > >
> > > > Please note that since I am above MD raid5 (I believe this is the
> > > > reason), the passdown of discards does not work, as my dmesg says:
> > > > [160508.497879] device-mapper: thin: Discard unsupported by data
> > > > device (dm-1): Disabling discard passdown.
> > > > but AFAIU, unless there is a thinp bug, this should not affect the
> > > > unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
> > > > able to do that.
> > >
> > > Does ext4 report that same error?
> >
> > That message says the underlying device doesn't support discards
> > (because it is an MD device). But the thinp device still has discards
> > enabled -- it just won't pass the discards down to the underlying data
> > device.
> >
> > So yes, it'll happen with ext4 -- it is generated when the thin-pool
> > device is loaded (which happens independent of the filesystem that is
> > layered ontop).
> >
> > The discards still inform the thin-pool that the corresponding extents
> > are no longer allocated.
>
> So do I understand correctly that even though the discard came
> through and thinp took advantage of it it still returns EOPNOTSUPP ?

No, not correct. Why are you assuming this? I must be missing
something from this discussion that led you there.

> This seems rather suboptimal. IIRC there was a discussion to add an
> option to enable/disable sending discard in thinp target down
> to the device.
>
> So maybe it might be a bit smarter than that and actually
> enable/disable discard pass through depending on the underlying
> support, so we do not blindly send discard down to the device even
> though it does not support it.

Yes, that is what we did.

Discards are enabled my default (including discard passdown), but if the
underlying data device doesn't support discards then the discards will
not be passed down.

And here are the feature controls that can be provided when loading the
thin-pool's DM table:

ignore_discard: disable discard
no_discard_passdown: don't pass discards down to the data device

-EOPNOTSUPP is only ever returned if 'ignore_discard' is provided.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Tue, 19 Jun 2012, Mike Snitzer wrote:

> Date: Tue, 19 Jun 2012 09:16:49 -0400
> From: Mike Snitzer <snitzer@redhat.com>
> To: Lukáš Czerner <lczerner@redhat.com>
> Cc: Dave Chinner <david@fromorbit.com>, Spelic <spelic@shiftmail.org>,
> device-mapper development <dm-devel@redhat.com>,
> linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
>
> On Tue, Jun 19 2012 at 2:32am -0400,
> Lukáš Czerner <lczerner@redhat.com> wrote:
>
> > On Mon, 18 Jun 2012, Mike Snitzer wrote:
> >
> > > Date: Mon, 18 Jun 2012 23:12:42 -0400
> > > From: Mike Snitzer <snitzer@redhat.com>
> > > To: Dave Chinner <david@fromorbit.com>
> > > Cc: Spelic <spelic@shiftmail.org>,
> > > device-mapper development <dm-devel@redhat.com>,
> > > linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> > > Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
> > >
> > > On Mon, Jun 18 2012 at 9:57pm -0400,
> > > Dave Chinner <david@fromorbit.com> wrote:
> > >
> > > > On Mon, Jun 18, 2012 at 11:33:50PM +0200, Spelic wrote:
> > > >
> > > > > Please note that since I am above MD raid5 (I believe this is the
> > > > > reason), the passdown of discards does not work, as my dmesg says:
> > > > > [160508.497879] device-mapper: thin: Discard unsupported by data
> > > > > device (dm-1): Disabling discard passdown.
> > > > > but AFAIU, unless there is a thinp bug, this should not affect the
> > > > > unmapping of thin blocks by fstrimming xfs... and in fact ext4 is
> > > > > able to do that.
> > > >
> > > > Does ext4 report that same error?
> > >
> > > That message says the underlying device doesn't support discards
> > > (because it is an MD device). But the thinp device still has discards
> > > enabled -- it just won't pass the discards down to the underlying data
> > > device.
> > >
> > > So yes, it'll happen with ext4 -- it is generated when the thin-pool
> > > device is loaded (which happens independent of the filesystem that is
> > > layered ontop).
> > >
> > > The discards still inform the thin-pool that the corresponding extents
> > > are no longer allocated.
> >
> > So do I understand correctly that even though the discard came
> > through and thinp took advantage of it it still returns EOPNOTSUPP ?
>
> No, not correct. Why are you assuming this? I must be missing
> something from this discussion that led you there.

Those two paragraphs led me to that conclusion:

That message says the underlying device doesn't support discards
(because it is an MD device). But the thinp device still has discards
enabled -- it just won't pass the discards down to the underlying data
device.

The discards still inform the thin-pool that the corresponding extents
are no longer allocated.

so I am a bit confused now. Why the dm-thin returned EOPNOTSUPP then
? Is that because it has been configured to ignore_discard, or it
actually takes advantage of the discard but underlying device does
not support it (and no_discard_passdown is not set) so it return
EOPNOTSUPP ?

>
> > This seems rather suboptimal. IIRC there was a discussion to add an
> > option to enable/disable sending discard in thinp target down
> > to the device.
> >
> > So maybe it might be a bit smarter than that and actually
> > enable/disable discard pass through depending on the underlying
> > support, so we do not blindly send discard down to the device even
> > though it does not support it.
>
> Yes, that is what we did.
>
> Discards are enabled my default (including discard passdown), but if the
> underlying data device doesn't support discards then the discards will
> not be passed down.
>
> And here are the feature controls that can be provided when loading the
> thin-pool's DM table:
>
> ignore_discard: disable discard
> no_discard_passdown: don't pass discards down to the data device
>
> -EOPNOTSUPP is only ever returned if 'ignore_discard' is provided.

Ok, so in this case 'ignore_discard' has been configured ?

Thanks!
-Lukas--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Tue, Jun 19 2012 at 9:25am -0400,
Lukáš Czerner <lczerner@redhat.com> wrote:

> On Tue, 19 Jun 2012, Mike Snitzer wrote:
>
> > Date: Tue, 19 Jun 2012 09:16:49 -0400
> > From: Mike Snitzer <snitzer@redhat.com>
> > To: Lukáš Czerner <lczerner@redhat.com>
> > Cc: Dave Chinner <david@fromorbit.com>, Spelic <spelic@shiftmail.org>,
> > device-mapper development <dm-devel@redhat.com>,
> > linux-ext4@vger.kernel.org, xfs@oss.sgi.com
> > Subject: Re: Ext4 and xfs problems in dm-thin on allocation and discard
> >
> > On Tue, Jun 19 2012 at 2:32am -0400,
> > Lukáš Czerner <lczerner@redhat.com> wrote:
> >
> > > So do I understand correctly that even though the discard came
> > > through and thinp took advantage of it it still returns EOPNOTSUPP ?
> >
> > No, not correct. Why are you assuming this? I must be missing
> > something from this discussion that led you there.
>
> Those two paragraphs led me to that conclusion:
>
> That message says the underlying device doesn't support discards
> (because it is an MD device). But the thinp device still has discards
> enabled -- it just won't pass the discards down to the underlying data
> device.
>
> The discards still inform the thin-pool that the corresponding extents
> are no longer allocated.
>
> so I am a bit confused now. Why the dm-thin returned EOPNOTSUPP then
> ? Is that because it has been configured to ignore_discard, or it
> actually takes advantage of the discard but underlying device does
> not support it (and no_discard_passdown is not set) so it return
> EOPNOTSUPP ?
>
> >
> > > This seems rather suboptimal. IIRC there was a discussion to add an
> > > option to enable/disable sending discard in thinp target down
> > > to the device.
> > >
> > > So maybe it might be a bit smarter than that and actually
> > > enable/disable discard pass through depending on the underlying
> > > support, so we do not blindly send discard down to the device even
> > > though it does not support it.
> >
> > Yes, that is what we did.
> >
> > Discards are enabled my default (including discard passdown), but if the
> > underlying data device doesn't support discards then the discards will
> > not be passed down.
> >
> > And here are the feature controls that can be provided when loading the
> > thin-pool's DM table:
> >
> > ignore_discard: disable discard
> > no_discard_passdown: don't pass discards down to the data device
> >
> > -EOPNOTSUPP is only ever returned if 'ignore_discard' is provided.
>
> Ok, so in this case 'ignore_discard' has been configured ?

I don't recall Spelic saying anything about EOPNOTSUPP. So what has
made you zero in on an -EOPNOTSUPP return (which should not be
happening)?

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Ext4 and xfs problems in dm-thin on allocation and discard
 

On Tue, Jun 19 2012 at 7:29am -0400,
Spelic <spelic@shiftmail.org> wrote:

> On 06/19/12 08:32, Lukáš Czerner wrote:
> >
> >So do I understand correctly that even though the discard came
> >through and thinp took advantage of it it still returns EOPNOTSUPP ?
> >This seems rather suboptimal. IIRC there was a discussion to add an
> >option to enable/disable sending discard in thinp target down
> >to the device.
>
> I'll ask this too...
> do I understand correctly that dm-thin returns EOPNOTSUPP to the
> filesystem layer even though it is using the discard to unmap
> blocks, and at that point XFS stops sending discards down there
> (while ext4 keeps sending them)?

Are you actually seeing that? Or are you just seizing on Lukas'
misunderstanding?

> This looks like a bug of dm-thin to me. Discards are "supported" in
> such a scenario.
>
> Do you have a patch for dm-thin so to prevent it sending EOPTNOTSUPP ?

thinp should _not_ be sending -EOPNOTSUPP unless 'ignore_discard' is
provided as a feature when loading thin-pool's DM table.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel