FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 09-02-2011, 09:34 PM
Mikulas Patocka
 
Default New dm-bufio with shrinker API

Hi

I created new dm-bufio that uses shrinker API and placed it here:
http://people.redhat.com/mpatocka/patches/kernel/dm-thinp-bufio/

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-05-2011, 09:04 AM
Joe Thornber
 
Default New dm-bufio with shrinker API

On Fri, Sep 02, 2011 at 05:34:09PM -0400, Mikulas Patocka wrote:
> Hi
>
> I created new dm-bufio that uses shrinker API and placed it here:
> http://people.redhat.com/mpatocka/patches/kernel/dm-thinp-bufio/

Thanks Mikulas, I'll merge over the next day or two.

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-05-2011, 02:49 PM
Joe Thornber
 
Default New dm-bufio with shrinker API

On Mon, Sep 05, 2011 at 10:04:29AM +0100, Joe Thornber wrote:
> On Fri, Sep 02, 2011 at 05:34:09PM -0400, Mikulas Patocka wrote:
> > Hi
> >
> > I created new dm-bufio that uses shrinker API and placed it here:
> > http://people.redhat.com/mpatocka/patches/kernel/dm-thinp-bufio/
>
> Thanks Mikulas, I'll merge over the next day or two.

Mikulas,

It's merged and pushed to my github repo.

I changed the test suite to reset the peak_allocated parameter before
each test and record it at the end of each test. It's very hard to
say what is right and wrong when talking about cache sizes, since you
always have to qualify anything by saying 'for this particular load'.
However, I think bufio could be more aggressive about recycling cache
entries. With the old block manager the test suite ran nicely with
less than 256k, from memory I think I started seeing slow down around
128k. With bufio I'm seeing consistently larger cache sizes for the
same performance.

For instance the test_overwriting_various_thin_devices scenario from
here
(https://github.com/jthornber/thinp-test-suite/blob/master/basic_tests.rb)
has a peak use of ~1100k, if I change from using dt with random io
pattern to plain old dd then the usage drops to ~900k. Setting the
max_age parameter to 1 second had very little effect.

With my bm I would trigger a flush if there were fewer than a 1/4 of
the blocks free, and at that point would try and flush half the blocks
(I think; would have to check exact numbers). Presumably you're doing
something very similar, except with different numbers. Do you think
we could publish these params to allow some experimentation please?

The allocated_* parameter files always seem to contain 0. Even when
cache_size is non-zero.

I don't understand the correspondence between 'cache_size' and
'peak_allocated'. I just ran a test and got these numbers:

cache_size: 7475200
peak_allocated: 974848

Is the cache_size correct? 7M seems an awful lot.

Can we really not avoid using dm-io to submit the ios? I was suprised
to see that in there when scanning the code for parameter names.

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-05-2011, 03:07 PM
Christoph Hellwig
 
Default New dm-bufio with shrinker API

On Mon, Sep 05, 2011 at 03:49:14PM +0100, Joe Thornber wrote:
> I changed the test suite to reset the peak_allocated parameter before
> each test and record it at the end of each test. It's very hard to
> say what is right and wrong when talking about cache sizes, since you
> always have to qualify anything by saying 'for this particular load'.
> However, I think bufio could be more aggressive about recycling cache
> entries. With the old block manager the test suite ran nicely with
> less than 256k, from memory I think I started seeing slow down around
> 128k. With bufio I'm seeing consistently larger cache sizes for the
> same performance.

IS there any reason you'll need a fixed size? This is fairly similar in
concept to the XFS buffercache, which does perfectly well by allocation
memory as needed, and letting the shrinker reclaim buffers when under
memory pressure.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-05-2011, 04:01 PM
Mikulas Patocka
 
Default New dm-bufio with shrinker API

> I don't understand the correspondence between 'cache_size' and
> 'peak_allocated'. I just ran a test and got these numbers:
>
> cache_size: 7475200
> peak_allocated: 974848
>
> Is the cache_size correct? 7M seems an awful lot.

"cache_size" is the value that you set as a maximum cache size. The
default is 2% of memory or 25% of vmalloc area.

"cache_size" doesn't change with benchmark that you run. You can set
cache size manually by writing the value to the file.

"peak_allocated" is the maximum number of bytes that was actually in use.
"peak_allocated" grows as more cache is allocated, but it is never shrunk.

> With the old block manager the test suite ran nicely with
> less than 256k, from memory I think I started seeing slow down around
> 128k. With bufio I'm seeing consistently larger cache sizes for the
> same performance.

So, reduce cache_size to 256k (or whatever value you want to test) and see
how it performs.

> For instance the test_overwriting_various_thin_devices scenario from
> here
> (https://github.com/jthornber/thinp-test-suite/blob/master/basic_tests.rb)
> has a peak use of ~1100k, if I change from using dt with random io
> pattern to plain old dd then the usage drops to ~900k. Setting the
> max_age parameter to 1 second had very little effect.

Reduce cache_size and try it.

> Can we really not avoid using dm-io to submit the ios? I was suprised
> to see that in there when scanning the code for parameter names.

If we didn't use dm-io, then we'd have to submit several bios in parallel.
It is possible to avoid dm-io, but it makes no sense, because we would be
duplicating dm-io logic in dm-bufio then.

Mikulas

> - Joe
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-06-2011, 08:50 AM
Joe Thornber
 
Default New dm-bufio with shrinker API

On Mon, Sep 05, 2011 at 11:07:15AM -0400, Christoph Hellwig wrote:
> On Mon, Sep 05, 2011 at 03:49:14PM +0100, Joe Thornber wrote:
> > I changed the test suite to reset the peak_allocated parameter before
> > each test and record it at the end of each test. It's very hard to
> > say what is right and wrong when talking about cache sizes, since you
> > always have to qualify anything by saying 'for this particular load'.
> > However, I think bufio could be more aggressive about recycling cache
> > entries. With the old block manager the test suite ran nicely with
> > less than 256k, from memory I think I started seeing slow down around
> > 128k. With bufio I'm seeing consistently larger cache sizes for the
> > same performance.
>
> IS there any reason you'll need a fixed size? This is fairly similar in
> concept to the XFS buffercache, which does perfectly well by allocation
> memory as needed, and letting the shrinker reclaim buffers when under
> memory pressure.

This is exactly what we're trying to do.

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-06-2011, 09:33 AM
Joe Thornber
 
Default New dm-bufio with shrinker API

On Mon, Sep 05, 2011 at 12:01:28PM -0400, Mikulas Patocka wrote:
> "cache_size" is the value that you set as a maximum cache size. The
> default is 2% of memory or 25% of vmalloc area.

ah, could you rename this variable to 'max_allocated' then please, to
match with the 'total_allocated' field (which I presume gives the
current cache size?).

> > With the old block manager the test suite ran nicely with
> > less than 256k, from memory I think I started seeing slow down around
> > 128k. With bufio I'm seeing consistently larger cache sizes for the
> > same performance.
>
> So, reduce cache_size to 256k (or whatever value you want to test) and see
> how it performs.

But then I'm limited to 256k, my point is we want scaling _and_ to use
less memory. We cannot tell our users to experiment to find the right
setting for this depending on the number of pools they're running and
the usage of each pool.

> > For instance the test_overwriting_various_thin_devices scenario from
> > here
> > (https://github.com/jthornber/thinp-test-suite/blob/master/basic_tests.rb)
> > has a peak use of ~1100k, if I change from using dt with random io
> > pattern to plain old dd then the usage drops to ~900k. Setting the
> > max_age parameter to 1 second had very little effect.
>
> Reduce cache_size and try it.

Here are the numbers (best of 3 runs):

| Test | 256k cache (M/s) | 2M cache (M/s) |
| unprovisioned thin | 74.4 | 75 |
| provisioned thin | 72.8 | 72.6 |
| new snap (complete sharing) | 73.7 | 73.8 |
| old snap (no sharing) | 72.2 | 72.8 |

So I think that proves my point. We're getting no benefit from that
extra memory, is there a subsystem that could be making better use of
it? (eg, page cache?). Or are you telling me that nobody else would
have been using that memory?

(This is all just tweaking, bufio is working very well).

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-06-2011, 09:53 AM
Joe Thornber
 
Default New dm-bufio with shrinker API

On Mon, Sep 05, 2011 at 11:07:15AM -0400, Christoph Hellwig wrote:
> On Mon, Sep 05, 2011 at 03:49:14PM +0100, Joe Thornber wrote:
> > I changed the test suite to reset the peak_allocated parameter before
> > each test and record it at the end of each test. It's very hard to
> > say what is right and wrong when talking about cache sizes, since you
> > always have to qualify anything by saying 'for this particular load'.
> > However, I think bufio could be more aggressive about recycling cache
> > entries. With the old block manager the test suite ran nicely with
> > less than 256k, from memory I think I started seeing slow down around
> > 128k. With bufio I'm seeing consistently larger cache sizes for the
> > same performance.
>
> IS there any reason you'll need a fixed size? This is fairly similar in
> concept to the XFS buffercache, which does perfectly well by allocation
> memory as needed, and letting the shrinker reclaim buffers when under
> memory pressure.

Well if the shrinker does such a good job, do we really need to set a
maximum value for the cache size at all? (perhaps this was your
question and I'm being slow).

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-06-2011, 03:57 PM
Mikulas Patocka
 
Default New dm-bufio with shrinker API

On Mon, 5 Sep 2011, Christoph Hellwig wrote:

> On Mon, Sep 05, 2011 at 03:49:14PM +0100, Joe Thornber wrote:
> > I changed the test suite to reset the peak_allocated parameter before
> > each test and record it at the end of each test. It's very hard to
> > say what is right and wrong when talking about cache sizes, since you
> > always have to qualify anything by saying 'for this particular load'.
> > However, I think bufio could be more aggressive about recycling cache
> > entries. With the old block manager the test suite ran nicely with
> > less than 256k, from memory I think I started seeing slow down around
> > 128k. With bufio I'm seeing consistently larger cache sizes for the
> > same performance.
>
> IS there any reason you'll need a fixed size? This is fairly similar in
> concept to the XFS buffercache, which does perfectly well by allocation
> memory as needed, and letting the shrinker reclaim buffers when under
> memory pressure.

It is possible to make unlimited size. --- the question: is the shrinker
run when we exhaust vmalloc arena?

dm-bufio cache uses vmalloc arena under some circumstances. On some
architectures (for example i386), vmalloc arena is smaller than main
memory, therefore it may overflow before main memory does.

What does XFS do when vmalloc arena is exhausted?

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 09-06-2011, 04:08 PM
Christoph Hellwig
 
Default New dm-bufio with shrinker API

On Tue, Sep 06, 2011 at 11:57:00AM -0400, Mikulas Patocka wrote:
> > IS there any reason you'll need a fixed size? This is fairly similar in
> > concept to the XFS buffercache, which does perfectly well by allocation
> > memory as needed, and letting the shrinker reclaim buffers when under
> > memory pressure.
>
> It is possible to make unlimited size. --- the question: is the shrinker
> run when we exhaust vmalloc arena?
>
> dm-bufio cache uses vmalloc arena under some circumstances. On some
> architectures (for example i386), vmalloc arena is smaller than main
> memory, therefore it may overflow before main memory does.
>
> What does XFS do when vmalloc arena is exhausted?

At this point shrinkers do not handle vmalloc space, although we could
add them. In the default configuration XFS uses very little vmalloc
space in the buffer cache - only the 8 log buffers are vmapped, and
those can't be reclaimed anyway. During log recovery or if using the
non-standard larger directory block mkfs option it can consume a larger
amount of vmalloc space, and we have run into problems because of that,
e.g. take a look at the loop around vm_map_ram() in _xfs_buf_map_pages()
that we had to add as a workaround, and the commit introducing it for
more details (a19fb380).

Just curious, why do you need the buffers to be vmapped? If we'd design
the dir2 format these days we'd make sure it is aligned in a way that
we could deal with individually mapped pages.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 03:08 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org