FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 06-10-2011, 08:44 AM
Joe Thornber
 
Default dm-kcopyd: introduce per-module throttle structure

On Thu, Jun 09, 2011 at 12:08:08PM -0400, Mikulas Patocka wrote:
>
>
> On Thu, 9 Jun 2011, Joe Thornber wrote:
> > What we're trying to do is avoid kcopyd issuing so much io that it
> > interferes with userland io.
>
> But you don't know if there is some userland IO or not to the same disk.

None the less, this was the motivation Alasdair gave for wanting this
throttling.

> > i) If there is lots of memory available can your throttling patch
> > still manage to issue too much io in the time that kcopyd is active?
>
> It issues as much IO as it can in the active period.

Exactly, it can issue too much.

> > ii) If there is little memory available few ios will be issued. But
> > your throttling will still occur, slowing things down even more.
>
> Yes. Memory pressure and throttiling are independent things.

True, but if kcopyd has only managed to submit 50k of io in it's last
timeslice why on earth would you decide to put it to sleep rather than
try and issue some more? I don't believe your time based throttling
behaves the same under different memory pressure situations. So the
sys admin could set up your throttle parameters under one set of
conditions. Then these conditions could change and result in either
too much or too little throttling.

>
> > I think it makes much more sense to throttle based on amount of io
> > issued by kcopyd. Either tracking throughput,
>
> You don't know what is the throughput of the device. So throttling to
> something like "50% throughput" can't be done.

I agree we don't know what the throughput on the devices is. What I
meant was to throttle the volume of io that kcopyd generates against
an absolute value. eg. "The mirror kcopyd client cannot submit more
than 100M of io per second." So you don't have to measure and
calculate any theoretical maximum throughput and calculate percentages
off that.

> > or even just putting a
> > limit on the amount of io that can be in flight at any one time.
>
> Which is much less reliable throttling than time slots.
> the resync spee is:
> 8 sub jobs --- 76MB/s
> 2 sub jobs --- 74MB/s
> 1 sub job --- 65MB/s

I really don't understand these figures. Why doesn't it scale
linearly with the number of sub jobs? Are the sub jobs all the same
size in these cases? Is this with your throttling? Are the sub jobs
so large that memory pressure is imposing a max limit of in flight io?

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-10-2011, 09:28 AM
Lars Ellenberg
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, Jun 10, 2011 at 09:44:25AM +0100, Joe Thornber wrote:
> On Thu, Jun 09, 2011 at 12:08:08PM -0400, Mikulas Patocka wrote:
> >
> >
> > On Thu, 9 Jun 2011, Joe Thornber wrote:
> > > What we're trying to do is avoid kcopyd issuing so much io that it
> > > interferes with userland io.
> >
> > But you don't know if there is some userland IO or not to the same disk.
>
> None the less, this was the motivation Alasdair gave for wanting this
> throttling.

Not sure if it helps,
but are you familiar with the MD raid rebuild throttling?

see is_mddev_idle() in drivers/md/md.c

It basically looks at read/write sector part_stats, and tracks its
own resync IO issued to the disk in question (in the special
gendisk member sync_io -- you'd probably need to use your own
counter in some other kcopyd context struct).

Then it figures, if there is "significant" increase in the
partition stats that cannot be accounted for by its own resync IO,
that should be userland IO, which means the device is not idle.

Then it has minimum and maximum bandwidth limits.
It won't use up more than maximum bandwidth,
even if the disk(s) seem to be idle.

If the disk seems to be NOT idle, it still would only throttle,
if its current estimated resync bandwidth is above the minimum
bandwidth limit.

Maybe a similar algorithm could be used for kcopyd?


Lars

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-10-2011, 10:14 AM
Joe Thornber
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, Jun 10, 2011 at 11:28:34AM +0200, Lars Ellenberg wrote:
> On Fri, Jun 10, 2011 at 09:44:25AM +0100, Joe Thornber wrote:
> > On Thu, Jun 09, 2011 at 12:08:08PM -0400, Mikulas Patocka wrote:
> > >
> > >
> > > On Thu, 9 Jun 2011, Joe Thornber wrote:
> > > > What we're trying to do is avoid kcopyd issuing so much io that it
> > > > interferes with userland io.
> > >
> > > But you don't know if there is some userland IO or not to the same disk.
> >
> > None the less, this was the motivation Alasdair gave for wanting this
> > throttling.
>
> Not sure if it helps,
> but are you familiar with the MD raid rebuild throttling?

Lars,

This is very helpful, thankyou. Any thoughts on this Mikulas?

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-10-2011, 01:41 PM
Mikulas Patocka
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, 10 Jun 2011, Joe Thornber wrote:

> On Fri, Jun 10, 2011 at 11:28:34AM +0200, Lars Ellenberg wrote:
> > On Fri, Jun 10, 2011 at 09:44:25AM +0100, Joe Thornber wrote:
> > > On Thu, Jun 09, 2011 at 12:08:08PM -0400, Mikulas Patocka wrote:
> > > >
> > > >
> > > > On Thu, 9 Jun 2011, Joe Thornber wrote:
> > > > > What we're trying to do is avoid kcopyd issuing so much io that it
> > > > > interferes with userland io.
> > > >
> > > > But you don't know if there is some userland IO or not to the same disk.
> > >
> > > None the less, this was the motivation Alasdair gave for wanting this
> > > throttling.
> >
> > Not sure if it helps,
> > but are you familiar with the MD raid rebuild throttling?
>
> Lars,
>
> This is very helpful, thankyou. Any thoughts on this Mikulas?
>
> - Joe

This works if the data is directly on the partition, but it won't work on
the device mapper (if MD is on the device mapper, it won't work too).

The device mapper has no function to tell where the bio is finally
remapped, so you can't read disk statistics for that particular disk.

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-10-2011, 01:48 PM
Joe Thornber
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, Jun 10, 2011 at 09:41:50AM -0400, Mikulas Patocka wrote:
> This works if the data is directly on the partition, but it won't work on
> the device mapper (if MD is on the device mapper, it won't work too).
>
> The device mapper has no function to tell where the bio is finally
> remapped, so you can't read disk statistics for that particular disk.

ok, thx

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-10-2011, 04:13 PM
Lars Ellenberg
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, Jun 10, 2011 at 02:48:00PM +0100, Joe Thornber wrote:
> On Fri, Jun 10, 2011 at 09:41:50AM -0400, Mikulas Patocka wrote:
> > This works if the data is directly on the partition, but it won't work on
> > the device mapper (if MD is on the device mapper, it won't work too).
> >
> > The device mapper has no function to tell where the bio is finally
> > remapped, so you can't read disk statistics for that particular disk.
>
> ok, thx

Right, you do not know which phyical device the IO may end up on,
if it passes through several stacking layers.

But even if that would be desirable in the long term,
it may not be necessary right now?[*]

Jobs on one mapping would not notice that the same physical device may
be busy via some other mapping.

But at least it would notice user (ok, non-kcopyd) IO on the same source
or destination mapping(s), which is a big improvement already, compared
with completely ignoring what is going on?

struct kcopyd_job {
...
struct dm_io_region source;
struct dm_io_region dests[...];
...
}

struct dm_io_region {
struct block_device *bdev;
...
}

struct block_device {
...
struct gendisk *bd_disk;
...
}

struct gendisk {
...
struct hd_struct part0; <--- there live the stats.
...
}
[*]
/me wildly handwaving
That could be solved, if we want to.
Add to block_device_operations:
struct gendisk * (*get_lower_level_gendisk)(struct gendisk*);


struct gendisk *appropriate_function_name(struct gendisk *d)
{
while (d->fops->get_lower_level_gendisk)
d = d->fops->get_lower_level_gendisk(d);
return d;
}

Then have some convenience functions part_stat_read_lowest_level(...).

May have come up before, probably was rejected because of uglyness?

;-)


Lars

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-11-2011, 08:27 PM
Mikulas Patocka
 
Default dm-kcopyd: introduce per-module throttle structure

On Fri, 10 Jun 2011, Joe Thornber wrote:

> On Thu, Jun 09, 2011 at 12:08:08PM -0400, Mikulas Patocka wrote:
> >
> >
> > On Thu, 9 Jun 2011, Joe Thornber wrote:
> > > What we're trying to do is avoid kcopyd issuing so much io that it
> > > interferes with userland io.
> >
> > But you don't know if there is some userland IO or not to the same disk.
>
> None the less, this was the motivation Alasdair gave for wanting this
> throttling.
>
> > > i) If there is lots of memory available can your throttling patch
> > > still manage to issue too much io in the time that kcopyd is active?
> >
> > It issues as much IO as it can in the active period.
>
> Exactly, it can issue too much.

As shown in the numbers below, submitting one IO reduces throughput from
76MB/s to 65MB/s. Reducing number of IOs doesn't help much.

> > > ii) If there is little memory available few ios will be issued. But
> > > your throttling will still occur, slowing things down even more.
> >
> > Yes. Memory pressure and throttiling are independent things.
>
> True, but if kcopyd has only managed to submit 50k of io in it's last
> timeslice why on earth would you decide to put it to sleep rather than
> try and issue some more?

Because the issues of memory pressure and throttling are idependent.

> I don't believe your time based throttling
> behaves the same under different memory pressure situations. So the
> sys admin could set up your throttle parameters under one set of
> conditions. Then these conditions could change and result in either
> too much or too little throttling.

I tried it and it thottled with one sub job as well as with many sub jobs.

I think you are tackling multiple independent things here. In order to
have understandable code and understandable behaviour, the throttle code
must manage *JUST*ONE*THING* (the time when i/os are submitted, in this
case). If you start making exceptions such as "what if there is a memory
pressure" or "apart from restricting the time, send less i/o", then the
logic behind the code becomes unintelligible.

It is much easier to explain to the users "if you set X value in
/sys/module/dm_mirror/parameters/raid1_resync_throttle, then the copying
will be done in X% of time, leaving the disk idle 100-X% of time", then to
invent some magic mechanisms that change multiple things based on X and
other conditions.

> > > I think it makes much more sense to throttle based on amount of io
> > > issued by kcopyd. Either tracking throughput,
> >
> > You don't know what is the throughput of the device. So throttling to
> > something like "50% throughput" can't be done.
>
> I agree we don't know what the throughput on the devices is. What I
> meant was to throttle the volume of io that kcopyd generates against
> an absolute value. eg. "The mirror kcopyd client cannot submit more
> than 100M of io per second." So you don't have to measure and
> calculate any theoretical maximum throughput and calculate percentages
> off that.

And what if the beginning of the disk has 130MB/s and the end 80MB/s? Then
throughput-based throttle would behave differently based on the part of
the disk that is being resynchronized.

> > > or even just putting a
> > > limit on the amount of io that can be in flight at any one time.
> >
> > Which is much less reliable throttling than time slots.
> > the resync spee is:
> > 8 sub jobs --- 76MB/s
> > 2 sub jobs --- 74MB/s
> > 1 sub job --- 65MB/s
>
> I really don't understand these figures. Why doesn't it scale
> linearly with the number of sub jobs? Are the sub jobs all the same
> size in these cases? Is this with your throttling? Are the sub jobs
> so large that memory pressure is imposing a max limit of in flight io?

It is without throttling. It doesn't scale linearly because the disk has
just one mechanical arm with all the heads, so it processes all jobs
one-by-one anyway.

If the disk had multiple arms, it would scale linearly.

> - Joe

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-13-2011, 09:17 AM
Joe Thornber
 
Default dm-kcopyd: introduce per-module throttle structure

On Sat, Jun 11, 2011 at 04:27:02PM -0400, Mikulas Patocka wrote:
> It is much easier to explain to the users "if you set X value in
> /sys/module/dm_mirror/parameters/raid1_resync_throttle, then the copying
> will be done in X% of time, leaving the disk idle 100-X% of time", then to
> invent some magic mechanisms that change multiple things based on X and
> other conditions.

Yes, that is very easy to explain to the users. This sort of
description is what I was after when I kept asking you to tell me how
to set it.

It's not clear to me that setting the throttle to 80% will copy at 80%
of the speed, or leave the disk idle for 20% of the time. Perhaps you
have some benchmarks to back this up? (with different memory pressure
situations please).

Restricting the cpu available to issue io is not a great way to
throttle io. There will be plenty of situations where 20% of the cpu
is enough to swamp the devices.

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-13-2011, 09:06 PM
Mikulas Patocka
 
Default dm-kcopyd: introduce per-module throttle structure

On Mon, 13 Jun 2011, Joe Thornber wrote:

> On Sat, Jun 11, 2011 at 04:27:02PM -0400, Mikulas Patocka wrote:
> > It is much easier to explain to the users "if you set X value in
> > /sys/module/dm_mirror/parameters/raid1_resync_throttle, then the copying
> > will be done in X% of time, leaving the disk idle 100-X% of time", then to
> > invent some magic mechanisms that change multiple things based on X and
> > other conditions.
>
> Yes, that is very easy to explain to the users. This sort of
> description is what I was after when I kept asking you to tell me how
> to set it.
>
> It's not clear to me that setting the throttle to 80% will copy at 80%
> of the speed, or leave the disk idle for 20% of the time. Perhaps you
> have some benchmarks to back this up? (with different memory pressure
> situations please).

See below. It is not exatly linear, but it is approximately linear well
enough.

> Restricting the cpu available to issue io is not a great way to
> throttle io. There will be plenty of situations where 20% of the cpu
> is enough to swamp the devices.
>
> - Joe

You misunderstand the patch. The patch doesn't restrict cpu time. The
patch measures global time and time when one or more i/os was in flight
(it doesn't count the number of i/os in flight). Every second, both these
variables are divided by 2 (to provide some decay). The proportion of
these two variables is the percentage of actual disk activity. When this
proportion is greater than the percentage set by the user, we sleep.



Resync speed is this:

throttle | resync speed MB/s
100% 75
90% 66
80% 60
70% 49
60% 44
50% 33
40% 25
30% 17
20% 10
10% 5
0% 0.26

If I simulate memory pressure (i.e. don't allow allocations and process
just one sub job), the speed is this:

100% 70
90% 62
80% 55
70% 47
60% 37
50% 29
40% 25
30% 16
20% 9
10% 2
0% 0.12

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 06-14-2011, 08:34 AM
Joe Thornber
 
Default dm-kcopyd: introduce per-module throttle structure

On Mon, Jun 13, 2011 at 05:06:29PM -0400, Mikulas Patocka wrote:
> See below. It is not exatly linear, but it is approximately linear well
> enough.

Thanks Mikulas, these results are very nice. I'll merge this set into
my tree and give it a go.

- Joe

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 01:48 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org