Linux Archive - Prep work for immutable bio vecs

Linux Archive (http://www.linux-archive.org/)

- Device-mapper Development (http://www.linux-archive.org/device-mapper-development/)

- - Prep work for immutable bio vecs (http://www.linux-archive.org/device-mapper-development/701593-prep-work-immutable-bio-vecs.html)

Kent Overstreet

09-07-2012 10:59 PM

Prep work for immutable bio vecs

Random assortment of refactoring and trivial cleanups.

Immutable bio vecs and efficient bio splitting require auditing and removing
pretty much all bi_idx uses, among other things.

The reason is that with immutable bio vecs we can't use the bvec array
directly; if we have a partially completed bvec, that'll be indicated with a
field in struct bvec_iter (which gets embedded in struct bio) - bi_bvec_done.

bio_for_each_segments() will handle this transparently, so code needs to be
converted to use it or some other generic accessor.

Also, bio splitting means that when a driver gets a bio, bi_idx and
bi_bvec_done may both be nonzero. Again, just need to use generic accessors.

Kent Overstreet (9):
block: Convert integrity to bvec_alloc_bs(), and a bugfix
block: Add bio_advance()
block: Refactor blk_update_request()
md: Convert md_trim_bio() to use bio_advance()
block: Add bio_end()
block: Use bio_sectors() more consistently
block: Don't use bi_idx in bio_split() or require it to be 0
block: Remove bi_idx references
block: Remove some unnecessary bi_vcnt usage

block/blk-core.c | 86 ++++-----------------
block/cfq-iosched.c | 7 +-
block/deadline-iosched.c | 2 +-
drivers/block/aoe/aoeblk.c | 2 +-
drivers/block/aoe/aoecmd.c | 2 +-
drivers/block/brd.c | 3 +-
drivers/block/drbd/drbd_req.c | 8 +-
drivers/block/floppy.c | 1 -
drivers/block/pktcdvd.c | 8 +-
drivers/block/ps3vram.c | 2 +-
drivers/md/dm-raid1.c | 2 +-
drivers/md/dm-stripe.c | 2 +-
drivers/md/dm-verity.c | 4 +-
drivers/md/faulty.c | 6 +-
drivers/md/linear.c | 3 +-
drivers/md/md.c | 19 ++---
drivers/md/raid0.c | 9 +--
drivers/md/raid1.c | 21 +++--
drivers/md/raid10.c | 28 +++----
drivers/md/raid5.c | 22 +++---
drivers/message/fusion/mptsas.c | 6 +-
drivers/s390/block/dcssblk.c | 3 +-
drivers/scsi/libsas/sas_expander.c | 6 +-
drivers/scsi/mpt2sas/mpt2sas_transport.c | 10 +--
fs/bio-integrity.c | 128 ++++++++++---------------------
fs/bio.c | 48 +++++++++++-
fs/btrfs/extent_io.c | 3 +-
fs/buffer.c | 1 -
fs/gfs2/lops.c | 2 +-
fs/jfs/jfs_logmgr.c | 2 -
fs/logfs/dev_bdev.c | 5 --
include/linux/bio.h | 8 +-
include/trace/events/block.h | 10 +--
mm/page_io.c | 1 -
34 files changed, 192 insertions(+), 278 deletions(-)

--
1.7.12

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Tejun Heo

09-08-2012 07:41 PM

Prep work for immutable bio vecs

Hello, Kent.

On Fri, Sep 07, 2012 at 03:59:11PM -0700, Kent Overstreet wrote:
> Random assortment of refactoring and trivial cleanups.
>
> Immutable bio vecs and efficient bio splitting require auditing and removing
> pretty much all bi_idx uses, among other things.
>
> The reason is that with immutable bio vecs we can't use the bvec array
> directly; if we have a partially completed bvec, that'll be indicated with a
> field in struct bvec_iter (which gets embedded in struct bio) - bi_bvec_done.
>
> bio_for_each_segments() will handle this transparently, so code needs to be
> converted to use it or some other generic accessor.
>
> Also, bio splitting means that when a driver gets a bio, bi_idx and
> bi_bvec_done may both be nonzero. Again, just need to use generic accessors.

There are three pending patchsets and I don't know how they're
supposed to come together. Please explain on which tree the patches
are based and how they stack and preferably provide a git branch.

Thanks.

--
tejun

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Kent Overstreet

09-10-2012 01:21 AM

Prep work for immutable bio vecs

On Sat, Sep 08, 2012 at 12:41:15PM -0700, Tejun Heo wrote:
> Hello, Kent.
>
> On Fri, Sep 07, 2012 at 03:59:11PM -0700, Kent Overstreet wrote:
> > Random assortment of refactoring and trivial cleanups.
> >
> > Immutable bio vecs and efficient bio splitting require auditing and removing
> > pretty much all bi_idx uses, among other things.
> >
> > The reason is that with immutable bio vecs we can't use the bvec array
> > directly; if we have a partially completed bvec, that'll be indicated with a
> > field in struct bvec_iter (which gets embedded in struct bio) - bi_bvec_done.
> >
> > bio_for_each_segments() will handle this transparently, so code needs to be
> > converted to use it or some other generic accessor.
> >
> > Also, bio splitting means that when a driver gets a bio, bi_idx and
> > bi_bvec_done may both be nonzero. Again, just need to use generic accessors.
>
> There are three pending patchsets and I don't know how they're
> supposed to come together. Please explain on which tree the patches
> are based and how they stack and preferably provide a git branch.

Both of the new ones depend on the block cleanups series, but they
shouldn't depend on each other (bar perhaps some trivial merge
conflicts in bio.h)

Git repo - block_stuff branch on http://evilpiepirate.org/git/linux-bcache.git/

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Kent Overstreet

09-11-2012 12:22 AM

Prep work for immutable bio vecs

Random assortment of refactoring and trivial cleanups;

Immutable bio vecs and efficient bio splitting require auditing and
removing pretty much all bi_idx uses, among other things.

The reason is that with immutable bio vecs we can't use the bvec array
directly; if we have a partially completed bvec, that'll be indicated
with a field in struct bvec_iter (which gets embedded in struct bio) -
bi_bvec_done.

bio_for_each_segments() will handle this transparently, so code needs to
be converted to use it or some other generic accessor.

Also, bio splitting means that when a driver gets a bio, bi_idx and
bi_bvec_done may both be nonzero. Again, just need to use generic
accessors.

v2: Patch series now has all the prep work to be done before abstracting
out the bio iterator, I think.

Kent Overstreet (26):
block: Convert integrity to bvec_alloc_bs(), and a bugfix
block: Add bio_advance()
block: Refactor blk_update_request()
md: Convert md_trim_bio() to use bio_advance()
block: Add bio_end()
block: Use bio_sectors() more consistently
block: Don't use bi_idx in bio_split() or require it to be 0
block: Remove bi_idx references
block: Remove some unnecessary bi_vcnt usage
block: Add submit_bio_wait(), remove from md
raid10: Use bio_reset()
raid1: use bio_reset()
raid5: use bio_reset()
raid1: Refactor narrow_write_error() to not use bi_idx
block: Add bio_copy_data()
pktcdvd: use bio_copy_data()
pktcdvd: Use bio_reset() in disabled code to kill bi_idx usage
raid1: use bio_copy_data()
bounce: Refactor __blk_queue_bounce to not use bi_io_vec
block: Add bio_for_each_segment_all()
block: Convert some code to bio_for_each_segment_all()
block: Add bio_alloc_pages()
raid1: use bio_alloc_pages()
block: Add an explicit bio flag for bios that own their bvec
bio-integrity: Add explicit field for owner of bip_buf
block: Add BIO_SUBMITTED flag, kill BIO_CLONED

block/blk-core.c | 88 +++---------
block/cfq-iosched.c | 7 +-
block/deadline-iosched.c | 2 +-
drivers/block/aoe/aoeblk.c | 2 +-
drivers/block/aoe/aoecmd.c | 2 +-
drivers/block/brd.c | 3 +-
drivers/block/drbd/drbd_req.c | 8 +-
drivers/block/floppy.c | 1 -
drivers/block/pktcdvd.c | 102 ++++----------
drivers/block/ps3vram.c | 2 +-
drivers/md/dm-crypt.c | 3 +-
drivers/md/dm-raid1.c | 2 +-
drivers/md/dm-stripe.c | 2 +-
drivers/md/dm-verity.c | 4 +-
drivers/md/dm.c | 1 -
drivers/md/faulty.c | 6 +-
drivers/md/linear.c | 3 +-
drivers/md/md.c | 19 +--
drivers/md/raid0.c | 9 +-
drivers/md/raid1.c | 131 ++++++------------
drivers/md/raid10.c | 78 +++--------
drivers/md/raid5.c | 50 +++----
drivers/message/fusion/mptsas.c | 6 +-
drivers/s390/block/dcssblk.c | 3 +-
drivers/scsi/libsas/sas_expander.c | 6 +-
drivers/scsi/mpt2sas/mpt2sas_transport.c | 10 +-
fs/bio-integrity.c | 134 ++++++------------
fs/bio.c | 226 +++++++++++++++++++++++++++----
fs/btrfs/extent_io.c | 3 +-
fs/buffer.c | 1 -
fs/direct-io.c | 8 +-
fs/exofs/ore.c | 2 +-
fs/exofs/ore_raid.c | 2 +-
fs/gfs2/lops.c | 2 +-
fs/jfs/jfs_logmgr.c | 2 -
fs/logfs/dev_bdev.c | 5 -
include/linux/bio.h | 34 +++--
include/linux/blk_types.h | 3 +-
include/trace/events/block.h | 10 +-
mm/bounce.c | 75 +++-------
mm/page_io.c | 1 -
41 files changed, 471 insertions(+), 587 deletions(-)

--
1.7.12

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

NeilBrown

09-11-2012 05:22 AM

Prep work for immutable bio vecs

On Mon, 10 Sep 2012 17:22:11 -0700 Kent Overstreet <koverstreet@google.com>
wrote:

> Random assortment of refactoring and trivial cleanups;
>
> Immutable bio vecs and efficient bio splitting require auditing and
> removing pretty much all bi_idx uses, among other things.
>
> The reason is that with immutable bio vecs we can't use the bvec array
> directly; if we have a partially completed bvec, that'll be indicated
> with a field in struct bvec_iter (which gets embedded in struct bio) -
> bi_bvec_done.
>
> bio_for_each_segments() will handle this transparently, so code needs to
> be converted to use it or some other generic accessor.
>
> Also, bio splitting means that when a driver gets a bio, bi_idx and
> bi_bvec_done may both be nonzero. Again, just need to use generic
> accessors.
>
> v2: Patch series now has all the prep work to be done before abstracting
> out the bio iterator, I think.

Hi Kent,
this looks pretty good to me. I've only really looked closely at the md
bits, but they all seem to make sense (with the minor issues that I reported
separately).

If/when there is another posting I'll try to allocate some time to testing
them and looking more closely.

thanks!

NeilBrown

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Tejun Heo

09-20-2012 11:22 PM

Prep work for immutable bio vecs

On Mon, Sep 10, 2012 at 05:22:11PM -0700, Kent Overstreet wrote:
> Random assortment of refactoring and trivial cleanups;
>
> Immutable bio vecs and efficient bio splitting require auditing and
> removing pretty much all bi_idx uses, among other things.
>
> The reason is that with immutable bio vecs we can't use the bvec array
> directly; if we have a partially completed bvec, that'll be indicated
> with a field in struct bvec_iter (which gets embedded in struct bio) -
> bi_bvec_done.
>
> bio_for_each_segments() will handle this transparently, so code needs to
> be converted to use it or some other generic accessor.
>
> Also, bio splitting means that when a driver gets a bio, bi_idx and
> bi_bvec_done may both be nonzero. Again, just need to use generic
> accessors.
>
> v2: Patch series now has all the prep work to be done before abstracting
> out the bio iterator, I think.

Cc'ing Vivek. Kent, can you please add Vivek to Cc on block layer
patches?

Vivek, can you please review this series? It's generic block stuff
and definitely can use your review.

http://thread.gmane.org/gmane.linux.kernel.bcache.devel/1055

Thanks.

--
tejun

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Kent Overstreet

09-24-2012 10:34 PM

Prep work for immutable bio vecs

Random assortment of refactoring and trivial cleanups;

Previous posting: http://thread.gmane.org/gmane.linux.kernel.bcache.devel/1055

No big changes since v2 - incorporated review feedback - mostly documentation,
did more testing, and fixed the deadlock Vivek noticed in my "convert integrity
to bvec_alloc_bs()" patch.

Patch series is on top of my patches for fixing a bio allocation deadlock with
stacked blocked devices, which is on top of Jens' tree.

Git tree is at:
http://evilpiepirate.org/git/linux-bcache.git block_stuff

v2: Patch series now has all the prep work to be done before abstracting
out the bio iterator, I think.

Kent Overstreet (26):
block: Fix a buffer overrun in bio_integrity_split()
block: Convert integrity to bvec_alloc_bs()
block: Add bio_advance()
block: Refactor blk_update_request()
md: Convert md_trim_bio() to use bio_advance()
block: Add bio_end_sector()
block: Use bio_sectors() more consistently
block: Change bio_split() to respect the current value of bi_idx
block: Remove bi_idx references
block: Remove some unnecessary bi_vcnt usage
block: Add submit_bio_wait(), remove from md
raid10: Use bio_reset()
raid1: use bio_reset()
raid5: use bio_reset()
raid1: Refactor narrow_write_error() to not use bi_idx
block: Add bio_copy_data()
pktcdvd: use bio_copy_data()
pktcdvd: Use bio_reset() in disabled code to kill bi_idx usage
raid1: use bio_copy_data()
bounce: Refactor __blk_queue_bounce to not use bi_io_vec
block: Add bio_for_each_segment_all()
block: Convert some code to bio_for_each_segment_all()
block: Add bio_alloc_pages()
block: Add an explicit bio flag for bios that own their bvec
bio-integrity: Add explicit field for owner of bip_buf
block: Add BIO_SUBMITTED flag, kill BIO_CLONED

block/blk-core.c | 84 ++--------
block/cfq-iosched.c | 7 +-
block/deadline-iosched.c | 2 +-
drivers/block/aoe/aoeblk.c | 2 +-
drivers/block/aoe/aoecmd.c | 2 +-
drivers/block/brd.c | 3 +-
drivers/block/drbd/drbd_req.c | 8 +-
drivers/block/floppy.c | 1 -
drivers/block/pktcdvd.c | 102 +++---------
drivers/block/ps3vram.c | 2 +-
drivers/md/dm-crypt.c | 3 +-
drivers/md/dm-raid1.c | 2 +-
drivers/md/dm-stripe.c | 2 +-
drivers/md/dm-verity.c | 4 +-
drivers/md/dm.c | 1 -
drivers/md/faulty.c | 6 +-
drivers/md/linear.c | 3 +-
drivers/md/md.c | 19 +--
drivers/md/raid0.c | 9 +-
drivers/md/raid1.c | 133 +++++-----------
drivers/md/raid10.c | 78 +++-------
drivers/md/raid5.c | 49 +++---
drivers/message/fusion/mptsas.c | 6 +-
drivers/s390/block/dcssblk.c | 3 +-
drivers/scsi/libsas/sas_expander.c | 6 +-
drivers/scsi/mpt2sas/mpt2sas_transport.c | 10 +-
fs/bio-integrity.c | 143 +++++++----------
fs/bio.c | 260 +++++++++++++++++++++++++------
fs/btrfs/extent_io.c | 3 +-
fs/buffer.c | 1 -
fs/direct-io.c | 8 +-
fs/exofs/ore.c | 2 +-
fs/exofs/ore_raid.c | 2 +-
fs/gfs2/lops.c | 2 +-
fs/jfs/jfs_logmgr.c | 2 -
fs/logfs/dev_bdev.c | 5 -
include/linux/bio.h | 48 +++---
include/linux/blk_types.h | 3 +-
include/trace/events/block.h | 10 +-
mm/bounce.c | 75 +++------
mm/page_io.c | 1 -
41 files changed, 499 insertions(+), 613 deletions(-)

--
1.7.12

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Kent Overstreet

10-15-2012 08:08 PM

Prep work for immutable bio vecs

Since last time - split bio integrity stuff out into its own 2 patch series.

Also changed bio_advance() a bit - it now explicitly special cases discard and
WRITE_SAME requests, so that partial completions will work. Previously, partial
completions worked for other requests but would've silently blown up for those
- this'll be of use later too, for generic bio splitting.

I've got 3 patch series queued up but they all should be independent - git tree
is at

http://evilpiepirate.org/git/linux-bcache.git for-jens

It's on top of jens' for-next branch.

Kent Overstreet (24):
block: Add bio_advance()
block: Refactor blk_update_request()
md: Convert md_trim_bio() to use bio_advance()
block: Add bio_end_sector()
block: Use bio_sectors() more consistently
block: Change bio_split() to respect the current value of bi_idx
block: Remove bi_idx references
block: Remove some unnecessary bi_vcnt usage
block: Add submit_bio_wait(), remove from md
raid10: Use bio_reset()
raid1: use bio_reset()
raid5: use bio_reset()
raid1: Refactor narrow_write_error() to not use bi_idx
block: Add bio_copy_data()
pktcdvd: use bio_copy_data()
pktcdvd: Use bio_reset() in disabled code to kill bi_idx usage
raid1: use bio_copy_data()
bounce: Refactor __blk_queue_bounce to not use bi_io_vec
block: Add bio_for_each_segment_all()
block: Convert some code to bio_for_each_segment_all()
block: Add bio_alloc_pages()
block: Add an explicit bio flag for bios that own their bvec
bio-integrity: Add explicit field for owner of bip_buf
block: Add BIO_SUBMITTED flag, kill BIO_CLONED

block/blk-core.c | 84 ++----------
block/cfq-iosched.c | 7 +-
block/deadline-iosched.c | 2 +-
drivers/block/aoe/aoeblk.c | 2 +-
drivers/block/aoe/aoecmd.c | 2 +-
drivers/block/brd.c | 3 +-
drivers/block/drbd/drbd_req.c | 8 +-
drivers/block/floppy.c | 1 -
drivers/block/pktcdvd.c | 102 ++++----------
drivers/md/dm-crypt.c | 3 +-
drivers/md/dm-raid1.c | 2 +-
drivers/md/dm-stripe.c | 2 +-
drivers/md/dm-verity.c | 4 +-
drivers/md/dm.c | 1 -
drivers/md/faulty.c | 6 +-
drivers/md/linear.c | 3 +-
drivers/md/md.c | 17 +--
drivers/md/raid0.c | 9 +-
drivers/md/raid1.c | 133 ++++++------------
drivers/md/raid10.c | 78 +++--------
drivers/md/raid5.c | 49 ++++---
drivers/message/fusion/mptsas.c | 6 +-
drivers/s390/block/dcssblk.c | 3 +-
drivers/scsi/libsas/sas_expander.c | 6 +-
drivers/scsi/mpt2sas/mpt2sas_transport.c | 10 +-
fs/bio-integrity.c | 10 +-
fs/bio.c | 226 +++++++++++++++++++++++++++----
fs/btrfs/extent_io.c | 3 +-
fs/buffer.c | 1 -
fs/direct-io.c | 8 +-
fs/exofs/ore.c | 2 +-
fs/exofs/ore_raid.c | 2 +-
fs/gfs2/lops.c | 2 +-
fs/jfs/jfs_logmgr.c | 2 -
fs/logfs/dev_bdev.c | 5 -
include/linux/bio.h | 29 ++--
include/linux/blk_types.h | 5 +-
include/trace/events/block.h | 10 +-
mm/bounce.c | 75 +++-------
mm/page_io.c | 1 -
40 files changed, 424 insertions(+), 500 deletions(-)

--
1.7.12

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Kent Overstreet

10-15-2012 08:09 PM

Prep work for immutable bio vecs

All times are GMT. The time now is 11:14 PM.