Are folks in the Centos community succesfully using device-mapper-multipath?
I am looking to deploy it for error handling on our iSCSI setup but there
seems to be little traffic about this package on the Centos forums, as far
as I can tell, and there seems to be a number of small issues based on my
reading the dm-multipath developer lists and related resources.
-geoff
Geoff Galitz
Blankenheim NRW, Deutschland
http://www.galitz.org
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-25-2008, 06:02 PM
Joshua Baker-LePain
dm-multipath use
On Wed, 25 Jun 2008 at 7:49pm, Geoff Galitz wrote
Are folks in the Centos community succesfully using device-mapper-multipath?
I am looking to deploy it for error handling on our iSCSI setup but there
seems to be little traffic about this package on the Centos forums, as far
as I can tell, and there seems to be a number of small issues based on my
reading the dm-multipath developer lists and related resources.
I am in the midst of setting this up on C-5 attached to a MSA1000 running
active/passive. The documentation is... sparse, to say the least.
There's more than a bit of guesswork in my setup, and I have yet to
actually test the failover. I certainly think it'd be worthwhile for you
to document your experience somewhere.
>
>
>
>
> Are folks in the Centos community succesfully using
> device-mapper-multipath?
> I am looking to deploy it for error handling on our iSCSI setup but there
> seems to be little traffic about this package on the Centos forums, as far
> as I can tell, and there seems to be a number of small issues based on my
> reading the dm-multipath developer lists and related resources.
>
> -geoff
>
>
>
> Geoff Galitz
> Blankenheim NRW, Deutschland
> http://www.galitz.org
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
Hi,
I'm running ~45 machines that boot from SAN/or are connected to our SAN
fabrics on CentOS 4.x/5.x (And about 20 more running RHEL5) without
problems at all. The storage is IBM SVC (2145).
Just make sure multipathd is running on your system, otherwise it will not
recover paths correctly.
Thanks,
Finnur
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-25-2008, 06:41 PM
"Raja Subramanian"
dm-multipath use
On Wed, Jun 25, 2008 at 11:19 PM, Geoff Galitz <geoff@galitz.org> wrote:
> Are folks in the Centos community succesfully using device-mapper-multipath?
I have a couple of CentOS 5.1 servers connected to a dual controller
Hitachi SMS 100 array. Both iscsi and multipath are with failover
are all working great.
- Raja
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-25-2008, 06:45 PM
"nate"
dm-multipath use
Geoff Galitz wrote:
>
>
>
>
> Are folks in the Centos community succesfully using device-mapper-multipath?
> I am looking to deploy it for error handling on our iSCSI setup but there
> seems to be little traffic about this package on the Centos forums, as far
> as I can tell, and there seems to be a number of small issues based on my
> reading the dm-multipath developer lists and related resources.
I've used it on CentOS 4.6 and it worked fine in an active-active
configuration connected to a 3PAR E200 storage system(I had both
iSCSI and Fiber channel systems).
I also used it with an OpenFiler running iSCSI as well, though there
was only 1 path(used it for consistency)
nate
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-25-2008, 08:23 PM
"Miskell, Craig"
dm-multipath use
> Are folks in the Centos community succesfully using
> device-mapper-multipath?
> I am looking to deploy it for error handling on our iSCSI
> setup but there
> seems to be little traffic about this package on the Centos
> forums, as far
> as I can tell, and there seems to be a number of small issues
> based on my
> reading the dm-multipath developer lists and related resources.
>
> -geoff
I'm using it on RHEL 5 (close enough for the purposes of your query), connecting to an HP EVA 6000 SAN. The RHEL documentation (http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/DM_Multipath/index.html) certainly covers the basics adequately, and was enough to get me going. I'm using LVM over the top of that, so I found it worthwhile to tweak /etc/lvm/lvm.conf to filter out all the various aliases for the disks that show up in /dev. My filter line is currently:
filter = [ "r/sd.*/", "r:disk/by.*:", "a/.*/" ]
which works well for me, but YMMV, particularly with the filtering out of "sd.*" (That works here because our main OS disks are on /dev/cciss)
You've also got to be a little careful when unpresenting disks (SAN terminology, may not apply to ISCSI). From our internal documentation (some notes I wrote at the time, and with subsequent experience):
*********************
Removing is trickier; you need to ensure no-one is trying to still use the disk. Particularly watch out for lvm. If the disk is part of a volume group, you have to run
#vgchange -an <VGNAME>
first, otherwise LVM still thinks the disk is there, and things like lvmdiskscan/pvdisplay etc start hanging when the disk has gone away.
Once the disk is unused, unpresent the disk from the SAN, rescan to remove no-longer existing disks, then restart multipathd (/etc/init.d/multipathd restart). Running
#multipath -F
may also be sufficient, but I've found restarting multipathd entirely a smidgen more reliable (but I may have been doing things wrong before that).
If things get really stuck, then you might have some luck with dmsetup. If "multipath -ll" shows failed disks (that have been unpresented properly), use dmsetup to remove the failed disk with the command:
#dmsetup remove <device>
where <device> is "mpath<num>". Find the stuck one from the output of multipath -ll; be sure you've got the right mpath device.
Optionally, if you've got stuck lvmdiskscan or pvdisplay type processes (trying to access the missing disk), then the "remove" will fail, claiming the device is in use (which, in some senses, it is). In this case, double check you've got the right mpath device (otherwise you'll fsck your system), and run:
#dmsetup remove --force <device>
This will claim failure (device-mapper: remove ioctl failed: Device or resource busy), but if you now run
#dmsetup info <device>
then you'll see the "Open count" has gone to zero. You can now run the plain remove one more time:
#dmsetup remove <device>
and it will be removed. Your hung processes will finally die the death they deserve, and the unpresented disk will be unknown to the system any longer.
*********************
It has worked well in real life, except for one day when one of our EVA SAN Controllers died; one host survived, another had multipathd itself die with a double free error (which I bugzilla'd upstream). Disks went away, but came back on restarting multipathd. Odd, but survivable, and not indicative of a general problem (probably something I did early on in the setup that hung around).
And one other word of advice: Play with it a lot in a test system first. It should go without saying, but this is really one of those times. There are many things you can learn safely on a production device; this isn't one of them. Get really comfortable with adding/removing/munging before you go live. And you will break it at least once during your preparation, if not more ;-).
Craig Miskell
================================================== =====================
Attention: The information contained in this message and/or attachments
from AgResearch Limited is intended only for the persons or entities
to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipients is prohibited by AgResearch
Limited. If you have received this message in error, please notify the
sender immediately.
================================================== =====================
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-25-2008, 11:31 PM
Tom Lanyon
dm-multipath use
On 26/06/2008, at 3:57 AM, Finnur Örn Guðmundsson wrote:
Hi,
I'm running ~45 machines that boot from SAN/or are connected to our
SAN
fabrics on CentOS 4.x/5.x (And about 20 more running RHEL5) without
problems at all. The storage is IBM SVC (2145).
Just make sure multipathd is running on your system, otherwise it
will not
recover paths correctly.
Thanks,
Finnur
Same here on a similar number of systems connected to two SANs. active-
active over FC, four-paths running multibus, all works fine._____________________________________________ __
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
06-26-2008, 08:01 AM
"Plant, Dean"
dm-multipath use
Geoff Galitz wrote:
> Are folks in the Centos community succesfully using
> device-mapper-multipath? I am looking to deploy it for error handling
> on our iSCSI setup but there seems to be little traffic about this
> package on the Centos forums, as far as I can tell, and there seems
> to be a number of small issues based on my reading the dm-multipath
> developer lists and related resources.
>
> -geoff
Here's my notes from working with our SAN. Please check thoroughly as
these instructions worked for me but I had to learn this from scratch
and there maybe mistakes. One thing I never figured how to achieve was
to rescan an existing ISCSI device for changes if you resize a partition
on the SAN. I have always had to reboot to get the new partition size to
be seen.
# lsscsi
[0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sda
[1:0:0:0] disk COMPELNT Compellent Vol 0306 -
[2:0:0:0] disk COMPELNT Compellent Vol 0306 -
[3:0:0:0] disk COMPELNT Compellent Vol 0306 -
[4:0:0:0] disk COMPELNT Compellent Vol 0306 -
service multipathd start
chkconfig multipathd on
Configuring SAN volumes without reboot
Rescan for ISCSI devices
# iscsiadm -m session -R
Add your partitions For this example I created 2 partitions
Let the running kernel see the new partitions
# partprobe
fdisk /dev/mapper/mpath2
We need to tell the mapper about the new partitions
# ls -l /dev/mapper/mpath2*
brw-rw---- 1 root disk 253, 7 Jan 30 15:42 /dev/mapper/mpath2
# kpartx -l /dev/mapper/mpath2
# ls -l /dev/mapper/mpath2*
brw-rw---- 1 root disk 253, 7 Jan 30 15:42 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 8 Jan 30 15:43 /dev/mapper/mpath2p1
brw-rw---- 1 root disk 253, 9 Jan 30 15:43 /dev/mapper/mpath2p2
# iscsiadm -m discovery
192.168.100.2:3260 via sendtargets
192.168.100.6:3260 via sendtargets
192.168.100.4:3260 via sendtargets
192.168.100.8:3260 via sendtargets
To automatically mount a file system during startup
you must have the partition entry in /etc/fstab marked with the
"_netdev"
option. For example this would mount a iscsi disk sdb:
When you create an LVM logical volume that uses active/passive multipath
arrays as the underlying physical devices, you should include filters in
the lvm.conf to exclude the disks that underlie the multipath devices.
This is because if the array automatically changes the active path to
the passive path when it receives I/O, multipath will failover and
failback whenever LVM scans the passive path if these devices are not
filtered. For active/passive arrays that require a command to make the
passive path active, LVM prints a warning message when this occurs.
To filter all SCSI devices in the multipath configuration file
(lvm.conf), include the following filter in the devices section of the
file.
filter = [ "r/disk/", "r/sd.*/", "a/.*/" ]
A filter to allow sda but disallow all other sd* drives
# pvdisplay /dev/mpath/mpath2p1
--- Physical volume ---
PV Name /dev/mpath/mpath2p1
VG Name VGSAN00
PV Size 1019.72 MB / not usable 3.72 MB
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 254
Free PE 0
Allocated PE 254
PV UUID ZjqvDp-mxMh-xbuV-CFql-QzaB-cC4l-Eo7RNl
vgcreate VGSAN00 /dev/mpath/mpath2p1
# vgdisplay VGSAN00
--- Volume group ---
VG Name VGSAN00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 1016.00 MB
PE Size 4.00 MB
Total PE 254
Alloc PE / Size 0 / 0
Free PE / Size 254 / 1016.00 MB
VG UUID bXv8IW-3eJT-RUXB-MLEo-gnfV-4VJv-j598PE
# lvcreate -l 254 -n data VGSAN00
# lvdisplay /dev/VGSAN00/data
--- Logical volume ---
LV Name /dev/VGSAN00/data
VG Name VGSAN00
LV UUID UorNsT-0NuN-kj86-2YGq-gVvG-aJYu-m04bhj
LV Write Access read/write
LV Status available
# open 0
LV Size 1016.00 MB
Current LE 254
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:9
------------------------------------------------
vi /etc/multipath.conf
dm-multipath use
