FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 03-18-2009, 09:57 PM
Christopher Chen
 
Default Shell Scripts or Arbitrary Priority Callouts?

Hello, running Centos 5.2 here--

The multipathd daemon is very unhappy with any arbritrary script I
provide for determining priorities. I see some fuzz in the syslog
about ramfs and static binaries.

How do I use shell scripts or arbitrary programs for multipathd? I
compiled a simple program that spits out "1" and it seems to return
appropriately.

Also, why does multipath -ll return the appropriate data, namely
prio=1 (when using my custom statically compiled callout) and
multipath -l always returns prio=0? Is this an indication of a broken
configuration or something else?

Cheers

cc

--
Chris Chen <muffaleta@gmail.com>
"I want the kind of six pack you can't drink."
-- Micah

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-19-2009, 10:04 AM
"John A. Sullivan III"
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Wed, 2009-03-18 at 15:57 -0700, Christopher Chen wrote:
> Hello, running Centos 5.2 here--
>
> The multipathd daemon is very unhappy with any arbritrary script I
> provide for determining priorities. I see some fuzz in the syslog
> about ramfs and static binaries.
>
> How do I use shell scripts or arbitrary programs for multipathd? I
> compiled a simple program that spits out "1" and it seems to return
> appropriately.
>
> Also, why does multipath -ll return the appropriate data, namely
> prio=1 (when using my custom statically compiled callout) and
> multipath -l always returns prio=0? Is this an indication of a broken
> configuration or something else?
>
> Cheers
>
> cc
>
I had the exact same problem and someone kindly explained it on this
list so thanks to them.

If I understand it correctly, multipathd must be prepared to function if
it loses access to disk. Therefore, it is designed to not read from
disk but caches everything it needs in memory. Apparently, it can only
cache binaries.

To use a shell script, call it via the shell, i.e., rather than
shell.script call sh shell.script.

That worked perfectly fine for me. However, I do not know if multipathd
actually caches shell.script or if it still must read it from disk when
invoking sh and hence remains vulnerable to loss of disk access. Does
anyone know? Thanks - John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-20-2009, 04:11 AM
Christopher Chen
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Thu, Mar 19, 2009 at 4:04 AM, John A. Sullivan III
<jsullivan@opensourcedevel.com> wrote:
> On Wed, 2009-03-18 at 15:57 -0700, Christopher Chen wrote:
>> Hello, running Centos 5.2 here--
>>
>> The multipathd daemon is very unhappy with any arbritrary script I
>> provide for determining priorities. I see some fuzz in the syslog
>> about ramfs and static binaries.
>>
>> How do I use shell scripts or arbitrary programs for multipathd? I
>> compiled a simple program that spits out "1" and it seems to return
>> appropriately.
>>
>> Also, why does multipath -ll return the appropriate data, namely
>> prio=1 (when using my custom statically compiled callout) and
>> multipath -l always returns prio=0? Is this an indication of a broken
>> configuration or something else?
>>
>> Cheers
>>
>> cc
>>
> I had the exact same problem and someone kindly explained it on this
> list so thanks to them.
>
> If I understand it correctly, multipathd must be prepared to function if
> it loses access to disk. *Therefore, it is designed to not read from
> disk but caches everything it needs in memory. *Apparently, it can only
> cache binaries.
>
> To use a shell script, call it via the shell, i.e., rather than
> shell.script call sh shell.script.
>
> That worked perfectly fine for me. *However, I do not know if multipathd
> actually caches shell.script or if it still must read it from disk when
> invoking sh and hence remains vulnerable to loss of disk access. *Does
> anyone know? Thanks - John

John:

Thanks for the reply.

I ended up writing a small C program to do the priority computation for me.

I have two sets of FC-AL shelves attached to two dual-channel Qlogic
cards. That gives me two paths to each disk. I have about 56 spindles
in the current configuration, and am tying them together with md
software raid.

Now, even though each disk says it handles concurrent I/O on each
port, my testing indicates that throughput suffers when using multibus
by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
when using multibus).

However, with failover, I am effectively using only one channel on
each card. With my custom priority callout, I more or less match the
disks with even numbers to the even numbered scsi channels with a
higher priority. Same with the odd numbered disks and odd numbered
channels. The odds are 2ndary on even and vice versa. It seems to work
rather well, and appears to spread the load nicely.

Thanks again for your help!

--
Chris Chen <muffaleta@gmail.com>
"I want the kind of six pack you can't drink."
-- Micah

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-20-2009, 09:01 AM
"John A. Sullivan III"
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Thu, 2009-03-19 at 22:11 -0700, Christopher Chen wrote:
> On Thu, Mar 19, 2009 at 4:04 AM, John A. Sullivan III
> <jsullivan@opensourcedevel.com> wrote:
> > On Wed, 2009-03-18 at 15:57 -0700, Christopher Chen wrote:
> >> Hello, running Centos 5.2 here--
> >>
> >> The multipathd daemon is very unhappy with any arbritrary script I
> >> provide for determining priorities. I see some fuzz in the syslog
> >> about ramfs and static binaries.
> >>
> >> How do I use shell scripts or arbitrary programs for multipathd? I
> >> compiled a simple program that spits out "1" and it seems to return
> >> appropriately.
> >>
> >> Also, why does multipath -ll return the appropriate data, namely
> >> prio=1 (when using my custom statically compiled callout) and
> >> multipath -l always returns prio=0? Is this an indication of a broken
> >> configuration or something else?
> >>
> >> Cheers
> >>
> >> cc
> >>
> > I had the exact same problem and someone kindly explained it on this
> > list so thanks to them.
> >
> > If I understand it correctly, multipathd must be prepared to function if
> > it loses access to disk. Therefore, it is designed to not read from
> > disk but caches everything it needs in memory. Apparently, it can only
> > cache binaries.
> >
> > To use a shell script, call it via the shell, i.e., rather than
> > shell.script call sh shell.script.
> >
> > That worked perfectly fine for me. However, I do not know if multipathd
> > actually caches shell.script or if it still must read it from disk when
> > invoking sh and hence remains vulnerable to loss of disk access. Does
> > anyone know? Thanks - John
>
> John:
>
> Thanks for the reply.
>
> I ended up writing a small C program to do the priority computation for me.
>
> I have two sets of FC-AL shelves attached to two dual-channel Qlogic
> cards. That gives me two paths to each disk. I have about 56 spindles
> in the current configuration, and am tying them together with md
> software raid.
>
> Now, even though each disk says it handles concurrent I/O on each
> port, my testing indicates that throughput suffers when using multibus
> by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
> when using multibus).
>
> However, with failover, I am effectively using only one channel on
> each card. With my custom priority callout, I more or less match the
> disks with even numbers to the even numbered scsi channels with a
> higher priority. Same with the odd numbered disks and odd numbered
> channels. The odds are 2ndary on even and vice versa. It seems to work
> rather well, and appears to spread the load nicely.
>
> Thanks again for your help!
>
I'm really glad you brought up the performance problem. I had posted
about it a few days ago but it seems to have gotten lost. We are really
struggling with performance issues when attempting to combine multiple
paths (in the case of multipath to one big target) or targets (in the
case of software RAID0 across several targets) rather than using, in
effect, JBODs. In our case, we are using iSCSI.

Like you, we found that using multibus caused almost a linear drop in
performance. Round robin across two paths was half as much as aggregate
throughput to two separate disks, four paths, one fourth.

We also tried striping across the targets with software RAID0 combined
with failover multipath - roughly the same effect.

We really don't want to be forced to treated SAN attached disks as
JDOBs. Has anyone cracked this problem of using them in either multibus
or RAID0 so we can present them as a single device to the OS and still
load balance multiple paths. This is a HUGE problem for us so any help
is greatly appreciated. Thanks- John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-22-2009, 02:27 PM
Pasi Kärkkäinen
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Fri, Mar 20, 2009 at 06:01:23AM -0400, John A. Sullivan III wrote:
> >
> > John:
> >
> > Thanks for the reply.
> >
> > I ended up writing a small C program to do the priority computation for me.
> >
> > I have two sets of FC-AL shelves attached to two dual-channel Qlogic
> > cards. That gives me two paths to each disk. I have about 56 spindles
> > in the current configuration, and am tying them together with md
> > software raid.
> >
> > Now, even though each disk says it handles concurrent I/O on each
> > port, my testing indicates that throughput suffers when using multibus
> > by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
> > when using multibus).
> >
> > However, with failover, I am effectively using only one channel on
> > each card. With my custom priority callout, I more or less match the
> > disks with even numbers to the even numbered scsi channels with a
> > higher priority. Same with the odd numbered disks and odd numbered
> > channels. The odds are 2ndary on even and vice versa. It seems to work
> > rather well, and appears to spread the load nicely.
> >
> > Thanks again for your help!
> >
> I'm really glad you brought up the performance problem. I had posted
> about it a few days ago but it seems to have gotten lost. We are really
> struggling with performance issues when attempting to combine multiple
> paths (in the case of multipath to one big target) or targets (in the
> case of software RAID0 across several targets) rather than using, in
> effect, JBODs. In our case, we are using iSCSI.
>
> Like you, we found that using multibus caused almost a linear drop in
> performance. Round robin across two paths was half as much as aggregate
> throughput to two separate disks, four paths, one fourth.
>
> We also tried striping across the targets with software RAID0 combined
> with failover multipath - roughly the same effect.
>
> We really don't want to be forced to treated SAN attached disks as
> JDOBs. Has anyone cracked this problem of using them in either multibus
> or RAID0 so we can present them as a single device to the OS and still
> load balance multiple paths. This is a HUGE problem for us so any help
> is greatly appreciated. Thanks- John

Hello.

Hmm.. just a guess, but could this be related to the fact that if your paths
to the storage are different iSCSI sessions (open-iscsi _doesn't_ support
multiple connections per session aka MC/s), then there is a separate SCSI
command queue per path.. and if SCSI requests are split across those queues
they can get out-of-order and that causes performance drop?

See:
http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html

Especially the reply from Ross (CC). Maybe he has some comments

-- Pasi

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-22-2009, 03:50 PM
"John A. Sullivan III"
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Sun, 2009-03-22 at 17:27 +0200, Pasi Kärkkäinen wrote:
> <snip>> > Now, even though each disk says it handles concurrent I/O on each
> > > port, my testing indicates that throughput suffers when using multibus
> > > by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
> > > when using multibus).
> > >
> > > However, with failover, I am effectively using only one channel on
> > > each card. With my custom priority callout, I more or less match the
> > > disks with even numbers to the even numbered scsi channels with a
> > > higher priority. Same with the odd numbered disks and odd numbered
> > > channels. The odds are 2ndary on even and vice versa. It seems to work
> > > rather well, and appears to spread the load nicely.
> > >
> > > Thanks again for your help!
> > >
> > I'm really glad you brought up the performance problem. I had posted
> > about it a few days ago but it seems to have gotten lost. We are really
> > struggling with performance issues when attempting to combine multiple
> > paths (in the case of multipath to one big target) or targets (in the
> > case of software RAID0 across several targets) rather than using, in
> > effect, JBODs. In our case, we are using iSCSI.
> >
> > Like you, we found that using multibus caused almost a linear drop in
> > performance. Round robin across two paths was half as much as aggregate
> > throughput to two separate disks, four paths, one fourth.
> >
> > We also tried striping across the targets with software RAID0 combined
> > with failover multipath - roughly the same effect.
> >
> > We really don't want to be forced to treated SAN attached disks as
> > JDOBs. Has anyone cracked this problem of using them in either multibus
> > or RAID0 so we can present them as a single device to the OS and still
> > load balance multiple paths. This is a HUGE problem for us so any help
> > is greatly appreciated. Thanks- John
>
> Hello.
>
> Hmm.. just a guess, but could this be related to the fact that if your paths
> to the storage are different iSCSI sessions (open-iscsi _doesn't_ support
> multiple connections per session aka MC/s), then there is a separate SCSI
> command queue per path.. and if SCSI requests are split across those queues
> they can get out-of-order and that causes performance drop?
>
> See:
> http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html
>
> Especially the reply from Ross (CC). Maybe he has some comments
<snip>
That makes sense and would explain what we are seeing with multipath but
why would we see the same thing with dmadm and using RAID0 to stripe
across multiple iSCSI targets? I would think that, just like increasing
physical spindles increases performance, increasing iSCSI sessions would
also increase performance.

On a side note, we did discover quite a bit about the influence of the
I/O scheduler last night. We found that there was only marginal
difference between cfq, deadline, anticipatory, and noop when running a
single thread. However, when running multiple threads, cfq did not
scale at all; performance for 10 threads was the same as for one - in
our case, roughly 6900 IOPS at 512 bytes block size for sequential read.
The other schedulers scaled almost linearly (at least at first). 10
threads shot up to 42000 IOPS, 40 threads to over 60000 IOPS.

We did find that RAID0 was able to scale - at 100 threads we hit around
106000 IOPS on our Nexenta based Z200 from Pogo Linux but performance on
a single thread is still less than performance to a single "spindle", a
single session. Why is that? Thanks - John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-23-2009, 03:42 AM
Christopher Chen
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Mar 22, 2009, at 8:27, Pasi Kärkkäinen <pasik@iki.fi> wrote:


On Fri, Mar 20, 2009 at 06:01:23AM -0400, John A. Sullivan III wrote:


John:

Thanks for the reply.

I ended up writing a small C program to do the priority
computation for me.


I have two sets of FC-AL shelves attached to two dual-channel Qlogic
cards. That gives me two paths to each disk. I have about 56
spindles

in the current configuration, and am tying them together with md
software raid.

Now, even though each disk says it handles concurrent I/O on each
port, my testing indicates that throughput suffers when using
multibus
by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/
sec

when using multibus).

However, with failover, I am effectively using only one channel on
each card. With my custom priority callout, I more or less match the
disks with even numbers to the even numbered scsi channels with a
higher priority. Same with the odd numbered disks and odd numbered
channels. The odds are 2ndary on even and vice versa. It seems to
work

rather well, and appears to spread the load nicely.

Thanks again for your help!


I'm really glad you brought up the performance problem. I had posted
about it a few days ago but it seems to have gotten lost. We are
really
struggling with performance issues when attempting to combine
multiple

paths (in the case of multipath to one big target) or targets (in the
case of software RAID0 across several targets) rather than using, in
effect, JBODs. In our case, we are using iSCSI.

Like you, we found that using multibus caused almost a linear drop in
performance. Round robin across two paths was half as much as
aggregate

throughput to two separate disks, four paths, one fourth.

We also tried striping across the targets with software RAID0
combined

with failover multipath - roughly the same effect.

We really don't want to be forced to treated SAN attached disks as
JDOBs. Has anyone cracked this problem of using them in either
multibus
or RAID0 so we can present them as a single device to the OS and
still
load balance multiple paths. This is a HUGE problem for us so any
help

is greatly appreciated. Thanks- John


Hello.

Hmm.. just a guess, but could this be related to the fact that if
your paths
to the storage are different iSCSI sessions (open-iscsi _doesn't_
support
multiple connections per session aka MC/s), then there is a separate
SCSI
command queue per path.. and if SCSI requests are split across those
queues

they can get out-of-order and that causes performance drop?

See:
http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html

Especially the reply from Ross (CC). Maybe he has some comments

-- Pasi

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


My problem with dm multipath multibus running at half the speed of
failover is not with iscsi but with some fibre channel disk shelves
I'm treating as a JBOD. I have two loops, and each FC drive is capable
of doing concurrent io through both ports.


I am exporting them via iscsi but the performance dropoff I'm seeing
is on the local host with the HBAs.


cc

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-23-2009, 08:46 AM
"John A. Sullivan III"
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Sun, 2009-03-22 at 17:27 +0200, Pasi Kärkkäinen wrote:
> On Fri, Mar 20, 2009 at 06:01:23AM -0400, John A. Sullivan III wrote:
> > >
> > > John:
> > >
> > > Thanks for the reply.
> > >
> > > I ended up writing a small C program to do the priority computation for me.
> > >
> > > I have two sets of FC-AL shelves attached to two dual-channel Qlogic
> > > cards. That gives me two paths to each disk. I have about 56 spindles
> > > in the current configuration, and am tying them together with md
> > > software raid.
> > >
> > > Now, even though each disk says it handles concurrent I/O on each
> > > port, my testing indicates that throughput suffers when using multibus
> > > by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
> > > when using multibus).
> > >
> > > However, with failover, I am effectively using only one channel on
> > > each card. With my custom priority callout, I more or less match the
> > > disks with even numbers to the even numbered scsi channels with a
> > > higher priority. Same with the odd numbered disks and odd numbered
> > > channels. The odds are 2ndary on even and vice versa. It seems to work
> > > rather well, and appears to spread the load nicely.
> > >
> > > Thanks again for your help!
> > >
> > I'm really glad you brought up the performance problem. I had posted
> > about it a few days ago but it seems to have gotten lost. We are really
> > struggling with performance issues when attempting to combine multiple
> > paths (in the case of multipath to one big target) or targets (in the
> > case of software RAID0 across several targets) rather than using, in
> > effect, JBODs. In our case, we are using iSCSI.
> >
> > Like you, we found that using multibus caused almost a linear drop in
> > performance. Round robin across two paths was half as much as aggregate
> > throughput to two separate disks, four paths, one fourth.
> >
> > We also tried striping across the targets with software RAID0 combined
> > with failover multipath - roughly the same effect.
> >
> > We really don't want to be forced to treated SAN attached disks as
> > JDOBs. Has anyone cracked this problem of using them in either multibus
> > or RAID0 so we can present them as a single device to the OS and still
> > load balance multiple paths. This is a HUGE problem for us so any help
> > is greatly appreciated. Thanks- John
>
> Hello.
>
> Hmm.. just a guess, but could this be related to the fact that if your paths
> to the storage are different iSCSI sessions (open-iscsi _doesn't_ support
> multiple connections per session aka MC/s), then there is a separate SCSI
> command queue per path.. and if SCSI requests are split across those queues
> they can get out-of-order and that causes performance drop?
>
> See:
> http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html
>
> Especially the reply from Ross (CC). Maybe he has some comments
>
> -- Pasi
<snip>
I'm trying to spend a little time on this today and am really feeling my
ignorance on the way iSCSI works It looks like linux-iscsi supports
MC/S but has not been in active development and will not even compile on
my 2.6.27 kernel.

To simplify matters, I did put each SAN interface on a separate network.
Thus, all the different sessions. If I place them all on the same
network and use the iface parameters of open-iscsi, does that eliminate
the out-of-order problem and allow me to achieve the performance
scalability I'm seeking from dm-multipath in multibus mode? Thanks -
John
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-23-2009, 12:07 PM
"John A. Sullivan III"
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Mon, 2009-03-23 at 08:50 -0400, Ross S. W. Walker wrote:
> On Mar 23, 2009, at 5:46 AM, "John A. Sullivan III"
> <jsullivan@opensourcedevel.com> wrote:
>
>
>
>
> > On Sun, 2009-03-22 at 17:27 +0200, Pasi Kärkkäinen wrote:
> > > On Fri, Mar 20, 2009 at 06:01:23AM -0400, John A. Sullivan III
> > wrote:
> > > > >
> > > > > John:
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > I ended up writing a small C program to do the priority
> > computation for me.
> > > > >
> > > > > I have two sets of FC-AL shelves attached to two dual-channel
> > Qlogic
> > > > > cards. That gives me two paths to each disk. I have about 56
> > spindles
> > > > > in the current configuration, and am tying them together with
> > md
> > > > > software raid.
> > > > >
> > > > > Now, even though each disk says it handles concurrent I/O on
> > each
> > > > > port, my testing indicates that throughput suffers when using
> > multibus
> > > > > by about 1/2 (from ~60 MB/sec sustained I/O with failover to
> > 35 MB/sec
> > > > > when using multibus).
> > > > >
> > > > > However, with failover, I am effectively using only one
> > channel on
> > > > > each card. With my custom priority callout, I more or less
> > match the
> > > > > disks with even numbers to the even numbered scsi channels
> > with a
> > > > > higher priority. Same with the odd numbered disks and odd
> > numbered
> > > > > channels. The odds are 2ndary on even and vice versa. It seems
> > to work
> > > > > rather well, and appears to spread the load nicely.
> > > > >
> > > > > Thanks again for your help!
> > > > >
> > > > I'm really glad you brought up the performance problem. I had
> > posted
> > > > about it a few days ago but it seems to have gotten lost. We
> > are really
> > > > struggling with performance issues when attempting to combine
> > multiple
> > > > paths (in the case of multipath to one big target) or targets
> > (in the
> > > > case of software RAID0 across several targets) rather than
> > using, in
> > > > effect, JBODs. In our case, we are using iSCSI.
> > > >
> > > > Like you, we found that using multibus caused almost a linear
> > drop in
> > > > performance. Round robin across two paths was half as much as
> > aggregate
> > > > throughput to two separate disks, four paths, one fourth.
> > > >
> > > > We also tried striping across the targets with software RAID0
> > combined
> > > > with failover multipath - roughly the same effect.
> > > >
> > > > We really don't want to be forced to treated SAN attached disks
> > as
> > > > JDOBs. Has anyone cracked this problem of using them in either
> > multibus
> > > > or RAID0 so we can present them as a single device to the OS and
> > still
> > > > load balance multiple paths. This is a HUGE problem for us so
> > any help
> > > > is greatly appreciated. Thanks- John
> > >
> > > Hello.
> > >
> > > Hmm.. just a guess, but could this be related to the fact that if
> > your paths
> > > to the storage are different iSCSI sessions (open-iscsi _doesn't_
> > support
> > > multiple connections per session aka MC/s), then there is a
> > separate SCSI
> > > command queue per path.. and if SCSI requests are split across
> > those queues
> > > they can get out-of-order and that causes performance drop?
> > >
> > > See:
> > >
> > http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html
> > >
> > > Especially the reply from Ross (CC). Maybe he has some comments
> > >
> > > -- Pasi
> > <snip>
> > I'm trying to spend a little time on this today and am really
> > feeling my
> > ignorance on the way iSCSI works It looks like linux-iscsi
> > supports
> > MC/S but has not been in active development and will not even
> > compile on
> > my 2.6.27 kernel.
> >
> > To simplify matters, I did put each SAN interface on a separate
> > network.
> > Thus, all the different sessions. If I place them all on the same
> > network and use the iface parameters of open-iscsi, does that
> > eliminate
> > the out-of-order problem and allow me to achieve the performance
> > scalability I'm seeking from dm-multipath in multibus mode? Thanks -
> > John
> >
> >
> >
> No, the only way to eliminate the out-of-order problem is MC/s. You
> can mask the issue when using IET by using fileio which caches these
> in page cache which will coalesce these before actually going to disk.
>
>
> The issue here seems it might be dm-multipath though.
>
>
> If your workload is random though, which most is, then sequential
> performance is inconsequential.
>
>
> -Ross
<snip>
Thanks very much, Ross. As I was working through the iface options of
open-iscsi this morning, I began to realize, just as you point out, they
are still separate sessions. We are not using Linux on the target end
but rather ZFS on opensolaris via Nexenta. I'll have to find out from
them what the equivalent to fileio is. Thanks again - John
>
--
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 03-24-2009, 06:39 AM
Pasi Kärkkäinen
 
Default Shell Scripts or Arbitrary Priority Callouts?

On Mon, Mar 23, 2009 at 05:46:36AM -0400, John A. Sullivan III wrote:
> On Sun, 2009-03-22 at 17:27 +0200, Pasi Kärkkäinen wrote:
> > On Fri, Mar 20, 2009 at 06:01:23AM -0400, John A. Sullivan III wrote:
> > > >
> > > > John:
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I ended up writing a small C program to do the priority computation for me.
> > > >
> > > > I have two sets of FC-AL shelves attached to two dual-channel Qlogic
> > > > cards. That gives me two paths to each disk. I have about 56 spindles
> > > > in the current configuration, and am tying them together with md
> > > > software raid.
> > > >
> > > > Now, even though each disk says it handles concurrent I/O on each
> > > > port, my testing indicates that throughput suffers when using multibus
> > > > by about 1/2 (from ~60 MB/sec sustained I/O with failover to 35 MB/sec
> > > > when using multibus).
> > > >
> > > > However, with failover, I am effectively using only one channel on
> > > > each card. With my custom priority callout, I more or less match the
> > > > disks with even numbers to the even numbered scsi channels with a
> > > > higher priority. Same with the odd numbered disks and odd numbered
> > > > channels. The odds are 2ndary on even and vice versa. It seems to work
> > > > rather well, and appears to spread the load nicely.
> > > >
> > > > Thanks again for your help!
> > > >
> > > I'm really glad you brought up the performance problem. I had posted
> > > about it a few days ago but it seems to have gotten lost. We are really
> > > struggling with performance issues when attempting to combine multiple
> > > paths (in the case of multipath to one big target) or targets (in the
> > > case of software RAID0 across several targets) rather than using, in
> > > effect, JBODs. In our case, we are using iSCSI.
> > >
> > > Like you, we found that using multibus caused almost a linear drop in
> > > performance. Round robin across two paths was half as much as aggregate
> > > throughput to two separate disks, four paths, one fourth.
> > >
> > > We also tried striping across the targets with software RAID0 combined
> > > with failover multipath - roughly the same effect.
> > >
> > > We really don't want to be forced to treated SAN attached disks as
> > > JDOBs. Has anyone cracked this problem of using them in either multibus
> > > or RAID0 so we can present them as a single device to the OS and still
> > > load balance multiple paths. This is a HUGE problem for us so any help
> > > is greatly appreciated. Thanks- John
> >
> > Hello.
> >
> > Hmm.. just a guess, but could this be related to the fact that if your paths
> > to the storage are different iSCSI sessions (open-iscsi _doesn't_ support
> > multiple connections per session aka MC/s), then there is a separate SCSI
> > command queue per path.. and if SCSI requests are split across those queues
> > they can get out-of-order and that causes performance drop?
> >
> > See:
> > http://www.nabble.com/round-robin-with-vmware-initiator-and-iscsi-target-td21958346.html
> >
> > Especially the reply from Ross (CC). Maybe he has some comments
> >
> > -- Pasi
> <snip>
> I'm trying to spend a little time on this today and am really feeling my
> ignorance on the way iSCSI works It looks like linux-iscsi supports
> MC/S but has not been in active development and will not even compile on
> my 2.6.27 kernel.
>
> To simplify matters, I did put each SAN interface on a separate network.
> Thus, all the different sessions. If I place them all on the same
> network and use the iface parameters of open-iscsi, does that eliminate
> the out-of-order problem and allow me to achieve the performance
> scalability I'm seeking from dm-multipath in multibus mode? Thanks -
> John

If you use ifaces feature of open-iscsi, you still get separate sessions.

open-iscsi just does not support MC/s

I think core-iscsi does support MC/s..

Then you again you should play with the different multipath settings, and
tweak how often IOs are split to different paths etc.. maybe that helps.

-- Pasi

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 01:06 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org