FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.

» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

LinkBack Thread Tools
Old 02-15-2008, 09:26 PM
Kiyoshi Ueda
Default request stacking and request-based dm-multipath

Hi Jens, list,

I'm working on device-mapper multipath (dm-multipath) and trying to
move its layer down below the I/O scheduler so that dm-multipath can
make better multipathing decision.
(This is subjected as request-based dm-multipath.)

On that design, dm-multipath treats and submits struct request to
underlying devices instead of struct bio.
So requests are stacked while currently dm and md are stacking bios.

The following 3 patches are implementation of the request stacking
1/3: add rq->complete_io hook for request stacking
2/3: move block-layer internal request completion to kblockd
3/3: export driver's busy state and an example of scsi

The first 2 patches enable request stacking.
The last 1 patch is necessary to avoid a performance problem,
that requests are sent down too early while the underlying device
is busy.

I also attach the patch-set of request-based dm-multipath
as appendix to show how the request stacking interfaces are used.
(With this version of the patch-set, request-based dm-multipath
works without userspace (multipath-tools and libdevmapper) change.)

This patch-set should be applied on top of 2.6.25-rc1.

Although I'm planning to discuss this topic in the Storage Workshop, too,
any comments and feedbacks in advance are very appreciated and helpful.

Below is the explanation about needs and details of the request stacking

Request-based dm-multipath needs this patch-set to hook in before
completing each chunk of request, check errors for it and retry it
using another path if error is detected.

The patch-set, which adds the rq->complete_io() hook, is necessary
to allow device stacking at request level, that is request-based
device-mapper multipath.
Currently, device-mapper is implemented as a stacking block device
at BIO level. OTOH, request-based dm will stack at request level
for better multipathing decision.
To stack devices at request level, the completion procedure needs
a hook for it.
For example, dm-multipath has to check errors and retry with other
paths if necessary before returning the I/O result to the upper layer.
struct request has 'end_io' hook currently. But it's called at
the very late stage of completion handling where the I/O result
is already returned to the upper layer.
So we need something here.

The first approach to hook in completion of each chunk of request
was adding a new rq->end_io_first() hook and calling it on the top
of __end_that_request_first().
- http://marc.info/?l=linux-kernel&m=116656637425880&w=2
- http://marc.info/?l=linux-scsi&m=115520444515914&w=2
However, Jens pointed out that redesigning rq->end_io() as a full
completion handler would be better:

On Thu, 21 Dec 2006 08:49:47 +0100, Jens Axboe wrote:
> Ok, I see what you are getting at. The current ->end_io() is called when
> the request has fully completed, you want notification for each chunk
> potentially completed.
> I think a better design here would be to use ->end_io() as the full
> completion handler, similar to how bio->bi_end_io() works. A request
> originating from __make_request() would set something ala:
> instead of calling the functions manually. That would allow you to get
> notification right at the beginning and do what you need, without adding
> a special hook for this.

I thought his comment was reasonable, and I modified the patches
based on his suggestion:
- http://marc.info/?l=linux-kernel&m=118860086901958&w=2

But the patch was ugly.
After considering the patch, I come to think changing the role
of ->end_io() is not feasible.
The role of ->end_io() is a destructor, and current users only
want to do something at the destruction of struct request, don't
want to do other works like data completion.
So I think it's reasonable to add another hook for request stacking.
(or rename the current ->end_io() to ->dtor() and make ->end_io()
a request stacking hook.)

Kiyoshi Ueda

dm-devel mailing list
Old 03-19-2008, 10:05 PM
Kiyoshi Ueda
Default request stacking and request-based dm-multipath

Hi Jens, Mike, Hannes, Alasdair and list,

This is a new version of request-based dm-multipath patches,
updated based on discussions and comments at LSF'08.
The patches are created on top of 2.6.25-rc5.

Major changes from the previous version (*) are:
- Request stacking drivers clone both request and bio, and
use both rq->end_io and bio->bi_end_io for request stacking
- Request stacking drivers use blk_complete_request() and
complete request in a softirq context, which doesn't have
queue lock, to avoid some deadlock issues
(*) http://lkml.org/lkml/2008/2/15/411

Some basic function/performance testings are done with NEC iStorage,
which is active-active multipath, and no problem was found.

The first 4 patches are for the block layer. Are they acceptable?

Request stacking design affects dm implementaion a lot.
If you have any objection on the design of dm side, please let me
know on this version since I'd like to fix the design and proceed
further dm implementation.

Summary of the patch-set:
01/13: block: add request data completion interface
02/13: block: add request submission interface
03/13: block: export driver's busy state via queue flag (and an example)
04/13: block: export blk_register_queue()
05/13: dm: remove dead code (preparation for request-based dm)
06/13: dm: tidy local_init (preparation for request-based dm)
07/13: dm: prepare mempools on module init for request-based dm
08/13: dm: add target interface for request-based dm
09/13: dm: add core functions for request-based dm
10/13: dm: add a switch to enable request-based dm if target is ready
11/13: dm: reject bad table load for request-based dm
12/13: dm-mpath: add hw-handler interface for request-based dm
13/13: dm-mpath: convert to request-based from bio-based

Summary of the design and request-based dm-multipath are below.

Currently, device-mapper (dm) is implemented as a stacking block device
at bio level. This bio-based implementation has an issue below
on dm-multipath.

Because hook for I/O mapping is above block layer __make_request(),
contiguous bios can be mapped to different underlying devices
and these bios aren't merged into a request.
Dynamic load balancing could happen this situation, though
it has not been implemented yet.
Therefore, I/O mapping after bio merging is needed for better
dynamic load balancing.

The basic idea to resolve the issue is to move multipathing layer
down below the I/O scheduler, and it was proposed from Mike Christie
as the block layer (request-based) multipath:

Mike's patch added new block layer device for multipath and didn't
have dm interface. So I modified his patch to be used from dm.
It is request-based dm-multipath.

While currently dm and md stacks block devices at bio level,
request-based dm stacks at request level and submits/completes
struct request instead of struct bio.

Overview of the request-based dm patch:
- Mapping is done in a unit of struct request, instead of struct bio
- Hook for I/O mapping is at q->request_fn() after merging and
sorting by I/O scheduler, instead of q->make_request_fn().
- Hook for I/O completion is at bio->bi_end_io() and rq->end_io(),
instead of only bio->bi_end_io()
bio-based (current) request-based (this patch)
submission q->make_request_fn() q->request_fn()
completion bio->bi_end_io() bio->bi_end_io(), rq->end_io()
- Whether the dm device is bio-based or request-based is determined
at table loading time
- Keep user interface same (table/message/status)
- Any bio-based devices (like dm/md) can be stacked on request-based
dm device.
Request-based dm device *cannot* be stacked on any bio-based device.

Expected benefit:
- better load balancing

Additional explanations:

Why does request-based dm use bio->bi_end_io(), too?
- dm needs to keep not only the request but also bios of the request,
if dm target drivers want to retry or do something on the request.
For example, dm-multipath has to check errors and retry with other
paths if necessary before returning the I/O result to the upper layer.

- But rq->end_io() is called at the very late stage of completion
handling where all bios in the request have been completed and
the I/O results are already returned to the upper layer.
So request-based dm hooks bio->bi_end_io() and doesn't complete the bio
in error cases, and gives over the error handling to rq->end_io() hook.

- dm generic interface for dynamic load balancing
- dynamic load balancer using the interface

Kiyoshi Ueda

dm-devel mailing list

Thread Tools

All times are GMT. The time now is 04:36 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org