FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor


 
 
LinkBack Thread Tools
 
Old 07-26-2010, 09:53 PM
Mikulas Patocka
 
Default BIO_RW_SYNCIO

Hi Jens

I found out that when I remove BIO_RW_SYNCIO from dm-kcopyd.c, performance
when writing to the snapshot origin is increased twice. In this workload,
snapshot driver reads a lot of 8k chunks from one place on the disk and
writes them to the other place on the same disk. Without BIO_RW_SYNCIO, it
is twice faster with CFQ scheduler.

What does BIO_RW_SYNCIO exactly do? Does it attempt to decrease latency
and decrease throughput? (so that disk head seeks more between the source
and destination and it causes performance degradation) Is there some
general guidance, when use BIO_RW_SYNCIO and when not?

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-27-2010, 02:09 AM
Vivek Goyal
 
Default BIO_RW_SYNCIO

On Mon, Jul 26, 2010 at 05:53:40PM -0400, Mikulas Patocka wrote:
> Hi Jens
>
[ Jens's mail id has changed. Ccing him on different mail id ]

> I found out that when I remove BIO_RW_SYNCIO from dm-kcopyd.c, performance
> when writing to the snapshot origin is increased twice. In this workload,
> snapshot driver reads a lot of 8k chunks from one place on the disk and
> writes them to the other place on the same disk. Without BIO_RW_SYNCIO, it
> is twice faster with CFQ scheduler.
>
> What does BIO_RW_SYNCIO exactly do? Does it attempt to decrease latency
> and decrease throughput? (so that disk head seeks more between the source
> and destination and it causes performance degradation) Is there some
> general guidance, when use BIO_RW_SYNCIO and when not?

BIO_RW_SYNC marks a request as SYNC. reads are by default sync. Writes
can be marked as SYNC so that they get priority of ASYNC WRITES.

Marking writes as SYNC has advantage that it gets its own queue, it can
preempt ASYNC WRITES and CFQ also idles on the cfq queue waiting for
more requests.

I suspect if it is idling on SYNC WRITES which is a problem here.

- Is it the same thread which is submitting both read and write requests?
- Are you interleaving reads and writes. Like reading some data, then
writting it and reading again...

I guess after writting some data, CFQ is idling on WRITE_SYNC and next
READ does not get dispatched immediately to the disk.

Is it possible to provide some 30 second blktrace of the underlying device
where CFQ is running. You can also try setting slice_idle to 0 and see if
it helps.

If setting slice_idle=0 helps you, can you please also give a try to
following patch.

https://patchwork.kernel.org/patch/113061/

Thanks
Vivek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-27-2010, 07:48 PM
Mikulas Patocka
 
Default BIO_RW_SYNCIO

On Mon, 26 Jul 2010, Vivek Goyal wrote:

> On Mon, Jul 26, 2010 at 05:53:40PM -0400, Mikulas Patocka wrote:
> > Hi Jens
> >
> [ Jens's mail id has changed. Ccing him on different mail id ]
>
> > I found out that when I remove BIO_RW_SYNCIO from dm-kcopyd.c, performance
> > when writing to the snapshot origin is increased twice. In this workload,
> > snapshot driver reads a lot of 8k chunks from one place on the disk and
> > writes them to the other place on the same disk. Without BIO_RW_SYNCIO, it
> > is twice faster with CFQ scheduler.
> >
> > What does BIO_RW_SYNCIO exactly do? Does it attempt to decrease latency
> > and decrease throughput? (so that disk head seeks more between the source
> > and destination and it causes performance degradation) Is there some
> > general guidance, when use BIO_RW_SYNCIO and when not?
>
> BIO_RW_SYNC marks a request as SYNC. reads are by default sync. Writes
> can be marked as SYNC so that they get priority of ASYNC WRITES.
>
> Marking writes as SYNC has advantage that it gets its own queue, it can
> preempt ASYNC WRITES and CFQ also idles on the cfq queue waiting for
> more requests.
>
> I suspect if it is idling on SYNC WRITES which is a problem here.
>
> - Is it the same thread which is submitting both read and write requests?

Yes.

> - Are you interleaving reads and writes. Like reading some data, then
> writting it and reading again...

I issue write immediatelly when read finishes.

> I guess after writting some data, CFQ is idling on WRITE_SYNC and next
> READ does not get dispatched immediately to the disk.
>
> Is it possible to provide some 30 second blktrace of the underlying device
> where CFQ is running. You can also try setting slice_idle to 0 and see if
> it helps.
>
> If setting slice_idle=0 helps you, can you please also give a try to
> following patch.
>
> https://patchwork.kernel.org/patch/113061/
>
> Thanks
> Vivek

I took the traces and placed them at
http://people.redhat.com/mpatocka/data/blktrace/

It shows that WRITE requests are merged without SYNCIO flags and are not
merged if SYNCIO is used.

slice_idle=0 or the patch don't help.


BTW. blkparse ignores the input file, when I use "blkparse -i
sda.blktrace.0.sync", blkparse still displays content of "sda.blktrace.0"
and not "sda.blktrace.0.sync".

I have to use "cat sda.blktrace.0.sync|blkparse -i -" to read th trace. Is
it a blkparse bug? I use blkparse from Debian Lenny.

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-27-2010, 11:09 PM
Vivek Goyal
 
Default BIO_RW_SYNCIO

On Tue, Jul 27, 2010 at 03:48:52PM -0400, Mikulas Patocka wrote:
>
>
> On Mon, 26 Jul 2010, Vivek Goyal wrote:
>
> > On Mon, Jul 26, 2010 at 05:53:40PM -0400, Mikulas Patocka wrote:
> > > Hi Jens
> > >
> > [ Jens's mail id has changed. Ccing him on different mail id ]
> >
> > > I found out that when I remove BIO_RW_SYNCIO from dm-kcopyd.c, performance
> > > when writing to the snapshot origin is increased twice. In this workload,
> > > snapshot driver reads a lot of 8k chunks from one place on the disk and
> > > writes them to the other place on the same disk. Without BIO_RW_SYNCIO, it
> > > is twice faster with CFQ scheduler.
> > >
> > > What does BIO_RW_SYNCIO exactly do? Does it attempt to decrease latency
> > > and decrease throughput? (so that disk head seeks more between the source
> > > and destination and it causes performance degradation) Is there some
> > > general guidance, when use BIO_RW_SYNCIO and when not?
> >
> > BIO_RW_SYNC marks a request as SYNC. reads are by default sync. Writes
> > can be marked as SYNC so that they get priority of ASYNC WRITES.
> >
> > Marking writes as SYNC has advantage that it gets its own queue, it can
> > preempt ASYNC WRITES and CFQ also idles on the cfq queue waiting for
> > more requests.
> >
> > I suspect if it is idling on SYNC WRITES which is a problem here.
> >
> > - Is it the same thread which is submitting both read and write requests?
>
> Yes.
>
> > - Are you interleaving reads and writes. Like reading some data, then
> > writting it and reading again...
>
> I issue write immediatelly when read finishes.
>
> > I guess after writting some data, CFQ is idling on WRITE_SYNC and next
> > READ does not get dispatched immediately to the disk.
> >
> > Is it possible to provide some 30 second blktrace of the underlying device
> > where CFQ is running. You can also try setting slice_idle to 0 and see if
> > it helps.
> >
> > If setting slice_idle=0 helps you, can you please also give a try to
> > following patch.
> >
> > https://patchwork.kernel.org/patch/113061/
> >
> > Thanks
> > Vivek
>
> I took the traces and placed them at
> http://people.redhat.com/mpatocka/data/blktrace/
>
> It shows that WRITE requests are merged without SYNCIO flags and are not
> merged if SYNCIO is used.

Yes you are right. So I think following is happening.

Key is that requests don't get merged once they are on dispatch list. They
get merged only when they are still sitting in some cfq queue and are with
elevator.

In case of sync IO, both reads and writes are on single cfq queue. We are
driving good dispatch list depth (drv=65). That means there are 65 reads
and writes on dispatch list and none of the new requests can be merged with
those.

In case of async IO, reads and writes are going on different cfq queues.
While reads are being dispatched from one queue, writes are sitting in
CFQ and are open to merge. That's the reason we are seeing lot more WRITE
merging with async case.

Not sure what we can do about it though. But had a couple of questions.

- You seem to be issuing lots of 4K size adjacent READS and WRITES. Is
there a way that you can club these together and issue a bigger request.

- What kind of device this is where request queue depth is 65. Can you
try reducing request queue depth to say 16 and see if things improve
a bit. (/sys/block/<dev>/device/queue_depth).

>
> slice_idle=0 or the patch don't help.

So it is a case of single cfq queue, so slice_idle will not help.

Thanks
Vivek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-28-2010, 01:33 AM
Vivek Goyal
 
Default BIO_RW_SYNCIO

On Tue, Jul 27, 2010 at 07:09:50PM -0400, Vivek Goyal wrote:
> On Tue, Jul 27, 2010 at 03:48:52PM -0400, Mikulas Patocka wrote:
> >
> >
> > On Mon, 26 Jul 2010, Vivek Goyal wrote:
> >
> > > On Mon, Jul 26, 2010 at 05:53:40PM -0400, Mikulas Patocka wrote:
> > > > Hi Jens
> > > >
> > > [ Jens's mail id has changed. Ccing him on different mail id ]
> > >
> > > > I found out that when I remove BIO_RW_SYNCIO from dm-kcopyd.c, performance
> > > > when writing to the snapshot origin is increased twice. In this workload,
> > > > snapshot driver reads a lot of 8k chunks from one place on the disk and
> > > > writes them to the other place on the same disk. Without BIO_RW_SYNCIO, it
> > > > is twice faster with CFQ scheduler.
> > > >
> > > > What does BIO_RW_SYNCIO exactly do? Does it attempt to decrease latency
> > > > and decrease throughput? (so that disk head seeks more between the source
> > > > and destination and it causes performance degradation) Is there some
> > > > general guidance, when use BIO_RW_SYNCIO and when not?
> > >
> > > BIO_RW_SYNC marks a request as SYNC. reads are by default sync. Writes
> > > can be marked as SYNC so that they get priority of ASYNC WRITES.
> > >
> > > Marking writes as SYNC has advantage that it gets its own queue, it can
> > > preempt ASYNC WRITES and CFQ also idles on the cfq queue waiting for
> > > more requests.
> > >
> > > I suspect if it is idling on SYNC WRITES which is a problem here.
> > >
> > > - Is it the same thread which is submitting both read and write requests?
> >
> > Yes.
> >
> > > - Are you interleaving reads and writes. Like reading some data, then
> > > writting it and reading again...
> >
> > I issue write immediatelly when read finishes.
> >
> > > I guess after writting some data, CFQ is idling on WRITE_SYNC and next
> > > READ does not get dispatched immediately to the disk.
> > >
> > > Is it possible to provide some 30 second blktrace of the underlying device
> > > where CFQ is running. You can also try setting slice_idle to 0 and see if
> > > it helps.
> > >
> > > If setting slice_idle=0 helps you, can you please also give a try to
> > > following patch.
> > >
> > > https://patchwork.kernel.org/patch/113061/
> > >
> > > Thanks
> > > Vivek
> >
> > I took the traces and placed them at
> > http://people.redhat.com/mpatocka/data/blktrace/
> >
> > It shows that WRITE requests are merged without SYNCIO flags and are not
> > merged if SYNCIO is used.
>
> Yes you are right. So I think following is happening.
>
> Key is that requests don't get merged once they are on dispatch list. They
> get merged only when they are still sitting in some cfq queue and are with
> elevator.
>
> In case of sync IO, both reads and writes are on single cfq queue. We are
> driving good dispatch list depth (drv=65). That means there are 65 reads
> and writes on dispatch list and none of the new requests can be merged with
> those.
>
> In case of async IO, reads and writes are going on different cfq queues.
> While reads are being dispatched from one queue, writes are sitting in
> CFQ and are open to merge. That's the reason we are seeing lot more WRITE
> merging with async case.
>
> Not sure what we can do about it though. But had a couple of questions.
>
> - You seem to be issuing lots of 4K size adjacent READS and WRITES. Is
> there a way that you can club these together and issue a bigger request.
>
> - What kind of device this is where request queue depth is 65. Can you
> try reducing request queue depth to say 16 and see if things improve
> a bit. (/sys/block/<dev>/device/queue_depth).
>

One more thing. Have you change the "quantum" tunable of CFQ? Looking
at the traces of SYNC case, CFQ seems to be allowing up to 65 requests
from a single cfq queue. But that does not happen. It is limited to 8
requests max from a cfq queue.

Which kernel version are you using?

Thanks
Vivek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-28-2010, 12:35 PM
Mikulas Patocka
 
Default BIO_RW_SYNCIO

> One more thing. Have you change the "quantum" tunable of CFQ? Looking
> at the traces of SYNC case, CFQ seems to be allowing up to 65 requests
> from a single cfq queue. But that does not happen. It is limited to 8
> requests max from a cfq queue.
>
> Which kernel version are you using?
>
> Thanks
> Vivek

I haven't changed it. It's 8 --- the default. I'm using 2.6.34.

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-28-2010, 12:42 PM
Mikulas Patocka
 
Default BIO_RW_SYNCIO

> > I took the traces and placed them at
> > http://people.redhat.com/mpatocka/data/blktrace/
> >
> > It shows that WRITE requests are merged without SYNCIO flags and are not
> > merged if SYNCIO is used.
>
> Yes you are right. So I think following is happening.
>
> Key is that requests don't get merged once they are on dispatch list. They
> get merged only when they are still sitting in some cfq queue and are with
> elevator.
>
> In case of sync IO, both reads and writes are on single cfq queue. We are
> driving good dispatch list depth (drv=65). That means there are 65 reads
> and writes on dispatch list and none of the new requests can be merged with
> those.
>
> In case of async IO, reads and writes are going on different cfq queues.
> While reads are being dispatched from one queue, writes are sitting in
> CFQ and are open to merge. That's the reason we are seeing lot more WRITE
> merging with async case.
>
> Not sure what we can do about it though. But had a couple of questions.
>
> - You seem to be issuing lots of 4K size adjacent READS and WRITES. Is
> there a way that you can club these together and issue a bigger request.

It is possible, but it would mean major code size increase (replicate the
merge functionality in dm-kcopyd). We don't have problems with CPU time
consumption, so we are not planning it now.

It just simpler to turn off BIO_RW_SYNCIO. I also turned off BIO_RW_UNPLUG
and unplug the queue after more requests. It improves performance to
22MB/s.

> - What kind of device this is where request queue depth is 65. Can you
> try reducing request queue depth to say 16 and see if things improve
> a bit. (/sys/block/<dev>/device/queue_depth).

Seagate U320 SCSI disk on MPT controller. It has 64 tags.

When I reduced the number of tags, it improved performance, 16 was good
(19MB/s), reducing it to 4 or 1 improved it even more (22MB/s).

Mikulas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-28-2010, 03:44 PM
Vivek Goyal
 
Default BIO_RW_SYNCIO

On Wed, Jul 28, 2010 at 08:42:06AM -0400, Mikulas Patocka wrote:
> > > I took the traces and placed them at
> > > http://people.redhat.com/mpatocka/data/blktrace/
> > >
> > > It shows that WRITE requests are merged without SYNCIO flags and are not
> > > merged if SYNCIO is used.
> >
> > Yes you are right. So I think following is happening.
> >
> > Key is that requests don't get merged once they are on dispatch list. They
> > get merged only when they are still sitting in some cfq queue and are with
> > elevator.
> >
> > In case of sync IO, both reads and writes are on single cfq queue. We are
> > driving good dispatch list depth (drv=65). That means there are 65 reads
> > and writes on dispatch list and none of the new requests can be merged with
> > those.
> >
> > In case of async IO, reads and writes are going on different cfq queues.
> > While reads are being dispatched from one queue, writes are sitting in
> > CFQ and are open to merge. That's the reason we are seeing lot more WRITE
> > merging with async case.
> >
> > Not sure what we can do about it though. But had a couple of questions.
> >
> > - You seem to be issuing lots of 4K size adjacent READS and WRITES. Is
> > there a way that you can club these together and issue a bigger request.
>
> It is possible, but it would mean major code size increase (replicate the
> merge functionality in dm-kcopyd). We don't have problems with CPU time
> consumption, so we are not planning it now.
>
> It just simpler to turn off BIO_RW_SYNCIO. I also turned off BIO_RW_UNPLUG
> and unplug the queue after more requests. It improves performance to
> 22MB/s.
>

I am not very sure how effective unplug thing is because it works only
if there is no IO happening in device. From blktraces it looks that device
is continuously busy.

I guess that not marking writes as sync is probably best in this case.
Though it will be interesting how does this look in presence of other
buffered writers on some other partition on device. I am not sure how
common case that is.

Are you seeing same issue with deadline also? I guess deadline might
run into same issue and beacause there is no idling logic there, I
think even turning off BIO_RW_SYNCIO is not going to help.

> > - What kind of device this is where request queue depth is 65. Can you
> > try reducing request queue depth to say 16 and see if things improve
> > a bit. (/sys/block/<dev>/device/queue_depth).
>
> Seagate U320 SCSI disk on MPT controller. It has 64 tags.
>
> When I reduced the number of tags, it improved performance, 16 was good
> (19MB/s), reducing it to 4 or 1 improved it even more (22MB/s).

Ok. I looked at the code again and realized that cfq allows unlimited dispatch
from a queue if there are no other competing queues. That's the reason in this
case we are allowing 64 request dispatch from single queue at a time.

Vivek

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 05:19 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org