FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 08-05-2008, 09:27 AM
Andrea Righi
 
Default Too many I/O controller patches

Paul Menage wrote:

On Mon, Aug 4, 2008 at 1:44 PM, Andrea Righi <righi.andrea@gmail.com> wrote:

A safer approach IMHO is to force the tasks to wait synchronously on
each operation that directly or indirectly generates i/o.

In particular the solution used by the io-throttle controller to limit
the dirty-ratio in memory is to impose a sleep via
schedule_timeout_killable() in balance_dirty_pages() when a generic
process exceeds the limits defined for the belonging cgroup.

Limiting read operations is a lot more easy, because they're always
synchronized with i/o requests.


I think that you're conflating two issues:

- controlling how much dirty memory a cgroup can have at any given
time (since dirty memory is much harder/slower to reclaim than clean
memory)

- controlling how much effect a cgroup can have on a given I/O device.

By controlling the rate at which a task can generate dirty pages,
you're not really limiting either of these. I think you'd have to set
your I/O limits artificially low to prevent a case of a process
writing a large data file and then doing fsync() on it, which would
then hit the disk with the entire file at once, and blow away any QoS
guarantees for other groups.


Anyway, dirty pages ratio is directly proportional to the IO that will
be performed on the real device, isn't it? this wouldn't prevent IO
bursts as you correctly say, but IMHO it is a simple and quite effective
way to measure the IO write activity of each cgroup on each affected
device.

To prevent the IO peaks I usually reduce the vm_dirty_ratio, but, ok,
this is a workaround, not the solution to the problem either.

IMHO, based on the dirty-page rate measurement, we should apply both
limiting methods: throttle dirty-pages ratio to prevent too many dirty
pages in the system (harde to reclaim and generating
unpredictable/unpleasant/unresponsiveness behaviour), and throttle the
dispatching of IO requests at the device-mapper/IO-scheduler layer to
smooth IO peaks/bursts, generated by fsync() and similar scenarios.

Another different approach could be to implement the measurement in the
elevator, looking at the elapsed between the IO request is issued to the
drive and the request is served. So, look at the start time T1,
completion time T2, take the difference (T2 - T1) and say: cgroup C1
consumed an amount of IO of (T2 - T1), and also use a token-bucket
policy to fill/reduce the "credits" of each IO cgroup in terms of IO
time slots. This would be a more precise measurement, instead of trying
to predict how expensive the IO operation will be, only looking at the
dirty-page ratio. Then throttle both dirty-page ratio *and* the
dispatching of the IO requests submitted by the cgroup that exceeds the
limits.



As Dave suggested, I think it would make more sense to have your
page-dirtying throttle points hook into the memory controller instead,
and allow the memory controller to track/limit dirty pages for a
cgroup, and potentially do throttling as part of that.

Paul


Yes, implementing page-drity throttling in memory controller seems
absolutely reasonable. I can try to move in this direction, merge the
page-dirty throttling in memory controller and also post the RFC.

Thanks,
-Andrea

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 09:28 AM
Andrea Righi
 
Default Too many I/O controller patches

Satoshi UCHIDA wrote:
> Andrea's requirement is
> * to be able to set and control by absolute(direct) performance.

* improve IO performance predictability of each cgroup
(try to guarantee more precise IO performance values)

> And, he gave a advice "Can't a framework which organized each way,
> such as I/O elevator, be made?".
> I try to consider such framework (in elevator layer or block layer).

It would be probably the best place to evaluate the "cost" of each
IO operation.

> I think that OOM problems caused by memory/cache systems.
> So, it will be better that I/O controller created out of these problems
> first, although a lateness of the I/O device would be related.
> If these problem can be resolved, its technique should be applied into
> normal I/O control as well as cgroups.
>
> Buffered write I/O is also related with cache system.
> We must consider this problem as I/O control.

Agree. At least, maybe we should consider if an IO controller could be
a valid solution also for these problems.

>> I did some experiments trying to implement minimum bandwidth requirements
>> for my io-throttle controller, mapping the requirements to CFQ prio and
>> using the Satoshi's controller. But this needs additional work and
>> testing right now, so I've not posted anything yet, just informed
>> Satoshi about this.
>
> I'm very interested in this results.

I'll collect some numbers and keep you informed.

-Andrea

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 09:31 AM
Andrea Righi
 
Default Too many I/O controller patches

Hirokazu Takahashi wrote:

Hi, Andrea,

I'm working with Ryo on dm-ioband and other stuff.


On Mon, 2008-08-04 at 20:22 +0200, Andrea Righi wrote:

But I'm not yet convinced that limiting the IO writes at the device
mapper layer is the best solution. IMHO it would be better to throttle
applications' writes when they're dirtying pages in the page cache (the
io-throttle way), because when the IO requests arrive to the device
mapper it's too late (we would only have a lot of dirty pages that are
waiting to be flushed to the limited block devices, and maybe this could
lead to OOM conditions). IOW dm-ioband is doing this at the wrong level
(at least for my requirements). Ryo, correct me if I'm wrong or if I've
not understood the dm-ioband approach.

The avoid-lots-of-page-dirtying problem sounds like a hard one. But, if
you look at this in combination with the memory controller, they would
make a great team.

The memory controller keeps you from dirtying more than your limit of
pages (and pinning too much memory) even if the dm layer is doing the
throttling and itself can't throttle the memory usage.

mmh... but in this way we would just move the OOM inside the cgroup,
that is a nice improvement, but the main problem is not resolved...


The concept of dm-ioband includes it should be used with cgroup memory
controller as well as the bio cgroup. The memory controller is supposed
to control memory allocation and dirty-page ratio inside each cgroup.

Some guys of cgroup memory controller team just started to implement
the latter mechanism. They try to make each cgroup have a threshold
to limit the number of dirty pages in the group.


Interesting, they also post a patch or RFC?

-Andrea

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 10:01 AM
Hirokazu Takahashi
 
Default Too many I/O controller patches

Hi,

> > Hi, Andrea,
> >
> > I'm working with Ryo on dm-ioband and other stuff.
> >
> >>> On Mon, 2008-08-04 at 20:22 +0200, Andrea Righi wrote:
> >>>> But I'm not yet convinced that limiting the IO writes at the device
> >>>> mapper layer is the best solution. IMHO it would be better to throttle
> >>>> applications' writes when they're dirtying pages in the page cache (the
> >>>> io-throttle way), because when the IO requests arrive to the device
> >>>> mapper it's too late (we would only have a lot of dirty pages that are
> >>>> waiting to be flushed to the limited block devices, and maybe this could
> >>>> lead to OOM conditions). IOW dm-ioband is doing this at the wrong level
> >>>> (at least for my requirements). Ryo, correct me if I'm wrong or if I've
> >>>> not understood the dm-ioband approach.
> >>> The avoid-lots-of-page-dirtying problem sounds like a hard one. But, if
> >>> you look at this in combination with the memory controller, they would
> >>> make a great team.
> >>>
> >>> The memory controller keeps you from dirtying more than your limit of
> >>> pages (and pinning too much memory) even if the dm layer is doing the
> >>> throttling and itself can't throttle the memory usage.
> >> mmh... but in this way we would just move the OOM inside the cgroup,
> >> that is a nice improvement, but the main problem is not resolved...
> >
> > The concept of dm-ioband includes it should be used with cgroup memory
> > controller as well as the bio cgroup. The memory controller is supposed
> > to control memory allocation and dirty-page ratio inside each cgroup.
> >
> > Some guys of cgroup memory controller team just started to implement
> > the latter mechanism. They try to make each cgroup have a threshold
> > to limit the number of dirty pages in the group.
>
> Interesting, they also post a patch or RFC?

You can take a look at the thread start from
http://www.ussg.iu.edu/hypermail/linux/kernel/0807.1/0472.html,
whose subject is "[PATCH][RFC] dirty balancing for cgroups."

This project has just started, so it would be a good time to
discuss it with them.

Thanks,
Hirokazu Takahashi.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 12:01 PM
Hirokazu Takahashi
 
Default Too many I/O controller patches

Hi,

> I think that OOM problems caused by memory/cache systems.
> So, it will be better that I/O controller created out of these problems
> first, although a lateness of the I/O device would be related.
> If these problem can be resolved, its technique should be applied into
> normal I/O control as well as cgroups.

Yes, this is one of the problems linux kernel still has, which should
be solved.

But I believe this should be done in the linux memory management layer
including the cgroup memory controller, which has to work correctly
on any type of device with various access speeds.

I think it's better that I/O controllers should only focus on flow of
I/O requests. This approach will keep the implementation of linux
kernel simple.

> Buffered write I/O is also related with cache system.
> We must consider this problem as I/O control.
> I don't have a good way which can resolve this problems.
>

Thank you,
Hirokazu Takahashi.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 01:17 PM
Ryo Tsuruta
 
Default Too many I/O controller patches

Hi Andrea, Satoshi and all,

Thanks for giving a chance to discuss.

> Mr. Andrew gave a advice "Should discuss about design more and more"
> to me.
> And, in Containers Mini-summit (and Linux Symposium 2008 in Ottawa),
> Paul said that a necessary to us is to decide a requirement first.
> So, we must discuss requirement and design.

We've implemented dm-ioband and bio-cgroup to meet the following requirements:
* Assign some bandwidth to each group on the same device.
A group is a set of processes, which may be a cgroup.
* Assign some bandwidth to each partition on the same device.
It can work with the process group based bandwidth control.
ex) With this feature, you can assign 40% of the bandwidth of a
disk to /root and 60% of them to /usr.
* It can work with virtual machines such as Xen and KVM.
I/O requests issued from virtual machines have to be controlled.
* It should work any type of I/O scheduler, including ones which
will be released in the future.
* Support multiple devices which share the same bandwidth such as
raid disks and LVM.
* Handle asynchronous I/O requests such as AIO request and delayed
write requests.
- This can be done with bio-cgroup, which uses the page-tracking
mechanism the cgroup memory controller has.
* Control dirty page ratio.
- This can be done with the cgroup memory controller in the near
feature. It would be great that you can also use other features
the memory controller is going to have with dm-ioband.
* Make it easy to enhance.
- The current implementation of dm-ioband has an interface to
add a new policy to control I/O requests. You can easily add
I/O throttling policy if you want.
* Fine grained bandwidth control.
* Keep I/O throughput.
* Make it scalable.
* It should work correctly if the I/O load is quite high,
even when the io-request queue of a certain disk is overflowed.

> Ryo, do you have other documentation besides the info reported in the
> dm-ioband website?

I don't have any documentation besides in the website.

Thanks,
Ryo Tsuruta

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 04:20 PM
Dave Hansen
 
Default Too many I/O controller patches

On Tue, 2008-08-05 at 11:28 +0200, Andrea Righi wrote:
> > Buffered write I/O is also related with cache system.
> > We must consider this problem as I/O control.
>
> Agree. At least, maybe we should consider if an IO controller could be
> a valid solution also for these problems.

Isn't this one of the core points that we keep going back and forth
over? It seems like people are arguing in circles over this:

Do we:
1. control potential memory usage by throttling I/O
or
2. Throttle I/O when memory is full

I might lean toward (1) if we didn't already have a memory controller.
But, we have one, and it works. Also, we *already* do (2) in the
kernel, so it would seem to graft well onto existing mechanisms that we
have.

I/O controllers should not worry about memory. They're going to have a
hard enough time getting the I/O part right.

Or, am I over-simplifying this?

-- Dave


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-05-2008, 04:25 PM
Dave Hansen
 
Default Too many I/O controller patches

On Mon, 2008-08-04 at 22:55 -0700, Paul Menage wrote:
>
> As Dave suggested, I think it would make more sense to have your
> page-dirtying throttle points hook into the memory controller instead,
> and allow the memory controller to track/limit dirty pages for a
> cgroup, and potentially do throttling as part of that.

Yeah, I'm sure we're going to have to get to setting the dirty ratio

$ cat /proc/sys/vm/dirty_ratio
40

on a per-container basis at *some* point. We might as well do it
earlier rather than later.

-- Dave

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-06-2008, 02:44 AM
KAMEZAWA Hiroyuki
 
Default Too many I/O controller patches

On Tue, 05 Aug 2008 09:20:18 -0700
Dave Hansen <dave@linux.vnet.ibm.com> wrote:

> On Tue, 2008-08-05 at 11:28 +0200, Andrea Righi wrote:
> > > Buffered write I/O is also related with cache system.
> > > We must consider this problem as I/O control.
> >
> > Agree. At least, maybe we should consider if an IO controller could be
> > a valid solution also for these problems.
>
> Isn't this one of the core points that we keep going back and forth
> over? It seems like people are arguing in circles over this:
>
> Do we:
> 1. control potential memory usage by throttling I/O
> or
> 2. Throttle I/O when memory is full
>
> I might lean toward (1) if we didn't already have a memory controller.
> But, we have one, and it works. Also, we *already* do (2) in the
> kernel, so it would seem to graft well onto existing mechanisms that we
> have.
>
> I/O controllers should not worry about memory.
I agree here

>They're going to have a hard enough time getting the I/O part right.
>
memcg have more problems now ;(

Only a difficult thing to limit dirty-ratio in memcg is how-to-count dirty
pages. If I/O controller's hook helps, it's good.

My small concern is "What happens if we throttole I/O bandwidth too small
under some memcg." In such cgroup, we may see more OOMs because I/O will
not finish in time.
A system admin have to find some way to avoid this.

But please do I/O control first. Dirty-page control is related but different
layer's problem, I think.

Thanks,
-Kame

> Or, am I over-simplifying this?
>



--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 08-06-2008, 03:30 AM
Balbir Singh
 
Default Too many I/O controller patches

KAMEZAWA Hiroyuki wrote:
> On Tue, 05 Aug 2008 09:20:18 -0700
> Dave Hansen <dave@linux.vnet.ibm.com> wrote:
>
>> On Tue, 2008-08-05 at 11:28 +0200, Andrea Righi wrote:
>>>> Buffered write I/O is also related with cache system.
>>>> We must consider this problem as I/O control.
>>> Agree. At least, maybe we should consider if an IO controller could be
>>> a valid solution also for these problems.
>> Isn't this one of the core points that we keep going back and forth
>> over? It seems like people are arguing in circles over this:
>>
>> Do we:
>> 1. control potential memory usage by throttling I/O
>> or
>> 2. Throttle I/O when memory is full
>>
>> I might lean toward (1) if we didn't already have a memory controller.
>> But, we have one, and it works. Also, we *already* do (2) in the
>> kernel, so it would seem to graft well onto existing mechanisms that we
>> have.
>>
>> I/O controllers should not worry about memory.
> I agree here
>
>> They're going to have a hard enough time getting the I/O part right.
>>
> memcg have more problems now ;(
>
> Only a difficult thing to limit dirty-ratio in memcg is how-to-count dirty
> pages. If I/O controller's hook helps, it's good.
>
> My small concern is "What happens if we throttole I/O bandwidth too small
> under some memcg." In such cgroup, we may see more OOMs because I/O will
> not finish in time.
> A system admin have to find some way to avoid this.
>
> But please do I/O control first. Dirty-page control is related but different
> layer's problem, I think.

Yes, please solve the I/O control problem first.

--
Warm Regards,
Balbir Singh
Linux Technology Center
IBM, ISTL


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 02:25 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org