FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Debian > Debian User

 
 
LinkBack Thread Tools
 
Old 12-22-2010, 09:22 AM
Ales Kozumplik
 
Default mpath: filter out the slave devices and their partitions.

This change necessitates from two previous commits:
1) 6e6d3df9a78730996e6ac7d463e3b50dbc164b4c, resulting in
DeviceTree.populate() being called twice now, for the second time at the
end of autopart.

2) 5c83bbf01800ca86d1c8e4cd95d14480165b344c, which causes already
coalesced mpath devices (disks and partitions) never to be torn down
during anaconda run.

Given that, we don't want to let identifyMultipath give
DeviceTree.populate() existing mpath devices (dm-0, dm-1,
etc.). populate() would try to add those before the slave mpath members
(sda, sdb) and although the addUdevDevice() logic will eventually
recursively request sda and sdb while working on dm-0, the udev info we
get for sda in sdb at that point is obtained thorugh a call to
udev_get_block_device() which will not give us a correctly filled in
ID_FS_TYPE (unlike the enriched info dictionaries that idenitfyMultipath()
gives us).

Alternative solution would be to embed a layer between DeviceTree and
udev.py. This layer would remember the additional database values that are
not returned by udev itself but rather discovered by us, such as the
"multipath_member" value of ID_FS_TYPE.

For the partitions part, the patch only does what identifyMultipath()'s
documentation promises: "3) removes the individual members of an mpath's
partitions". This step is optional and to make the function better
defined: note that e.g. sda3 will be scanned later in populate() anyway as
it will naturally still appear in udev_get_block_devices().

Resolves: rhbz#636570
---
storage/devicelibs/mpath.py | 70 +++++++++++++++++++++++++++---------------
storage/devicetree.py | 13 ++++----
storage/udev.py | 8 +++++
3 files changed, 59 insertions(+), 32 deletions(-)

diff --git a/storage/devicelibs/mpath.py b/storage/devicelibs/mpath.py
index 0d0efde..113c544 100644
--- a/storage/devicelibs/mpath.py
+++ b/storage/devicelibs/mpath.py
@@ -7,6 +7,31 @@ import logging

log = logging.getLogger("storage")

+def _filter_out_mpath_devices(devices):
+ retval = []
+ for d in devices:
+ if udev_device_is_multipath(d):
+ log.debug("filtering out coalesced mpath device: %s" % d['name'])
+ else:
+ retval.append(d)
+ return retval
+
+def _filter_out_mpath_partitions(devices, multipaths):
+ """
+ Use serial numbers of the multipath members to filter devices from the
+ devices list. The returned list will only partitions that are NOT partitions
+ of the multipath members.
+ """
+ serials = set(udev_device_get_serial(d)
+ for mpath_members in multipaths for d in mpath_members)
+ retval = []
+ for d in devices:
+ if udev_device_get_serial(d) in serials:
+ log.debug("filtering out mpath partition: %s" % d['name'])
+ else:
+ retval.append(d)
+ return retval
+
def parseMultipathOutput(output):
"""
Parse output from "multipath -d" or "multipath -ll" and form a topology.
@@ -84,14 +109,22 @@ def parseMultipathOutput(output):
return mpaths

def identifyMultipaths(devices):
- # this function does a couple of things
- # 1) identifies multipath disks
- # 2) sets their ID_FS_TYPE to multipath_member
- # 3) removes the individual members of an mpath's partitions
- # sample input with multipath pair [sdb,sdc]
- # [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2]
- # sample output:
- # [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]
+ """
+ This function does a couple of things:
+ 1) identifies multipath disks,
+ 2) sets their ID_FS_TYPE to multipath_member,
+ 3) removes the individual members of an mpath's partitions as well as any
+ coalesced multipath devices:
+
+ sample input:
+ [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2, dm-0]
+ where:
+ [sdb, sdc] is a multipath pair
+ dm-0 is a mutliapth device already coalesced from [sdb, sdc]
+
+ sample output:
+ [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]
+ """
log.info("devices to scan for multipath: %s" % [d['name'] for d in devices])

with open("/etc/multipath.conf") as conf:
@@ -165,23 +198,10 @@ def identifyMultipaths(devices):

multipaths.append([devmap[d] for d in disks])

- non_disk_serials = {}
- for name,device in non_disk_devices.items():
- serial = udev_device_get_serial(device)
- non_disk_serials.setdefault(serial, [])
- non_disk_serials[serial].append(device)
-
- for mpath in multipaths:
- for serial in [d.get('ID_SERIAL_SHORT') for d in mpath]:
- if non_disk_serials.has_key(serial):
- log.info("filtering out non disk devices [%s]" % [d['name'] for d in non_disk_serials[serial]])
- for name in [d['name'] for d in non_disk_serials[serial]]:
- if non_disk_devices.has_key(name):
- del non_disk_devices[name]
-
- partition_devices = []
- for device in non_disk_devices.values():
- partition_devices.append(device)
+ # singlepaths and partitions should not contain multipath devices:
+ singlepath_disks = _filter_out_mpath_devices(singlepath_disks)
+ partition_devices = _filter_out_mpath_partitions(
+ non_disk_devices.values(), multipaths)

# this is the list of devices we want to keep from the original
# device list, but we want to maintain its original order.
diff --git a/storage/devicetree.py b/storage/devicetree.py
index 45fc455..f45e656 100644
--- a/storage/devicetree.py
+++ b/storage/devicetree.py
@@ -2026,19 +2026,18 @@ class DeviceTree(object):
% (livetarget,))
self.protectedDevNames.append(livetarget)

- # First iteration - let's just look for disks.
- old_devices = {}
-
- devices = udev_get_block_devices()
- for dev in devices:
- old_devices[dev['name']] = dev
-
cfg = self.__multipathConfigWriter.write()
open("/etc/multipath.conf", "w+").write(cfg)
del cfg

+ devices = udev_get_block_devices()
(singles, mpaths, partitions) = devicelibs.mpath.identifyMultipaths(devices)
devices = singles + reduce(list.__add__, mpaths, []) + partitions
+ # remember all the devices idenitfyMultipaths() gave us at this point
+ old_devices = {}
+ for dev in devices:
+ old_devices[dev['name']] = dev
+
log.info("devices to scan: %s" % [d['name'] for d in devices])
for dev in devices:
self.addUdevDevice(dev)
diff --git a/storage/udev.py b/storage/udev.py
index 59b7519..7ba5a9c 100644
--- a/storage/udev.py
+++ b/storage/udev.py
@@ -472,6 +472,14 @@ def udev_device_is_dmraid_partition(info, devicetree):

return False

+def udev_device_is_multipath(info):
+ """ Return True if the device is a multipath device or a partition of one."""
+ if not udev_device_is_dm(info):
+ return False
+ if udev_device_get_name(info).startswith("mpath"):
+ return True
+ return False
+
def udev_device_is_multipath_partition(info, devicetree):
""" Return True if the device is a partition of a multipath device. """
if not udev_device_is_dm(info):
--
1.7.3.3

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 01-03-2011, 05:22 PM
"Brian C. Lane"
 
Default mpath: filter out the slave devices and their partitions.

On Wed, Dec 22, 2010 at 11:22:41AM +0100, Ales Kozumplik wrote:
> + sample input:
> + [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2, dm-0]
> + where:
> + [sdb, sdc] is a multipath pair
> + dm-0 is a mutliapth device already coalesced from [sdb, sdc]
> +
> + sample output:
> + [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]

Could you explain the return values in more detail? Is this always a 3
element list or does it depend on the number of devices? What does it
look like with no multipath devices?

Thanks,
Brian


--
Brian C. Lane | Anaconda Team | IRC: bcl #anaconda | Port Orchard, WA (PST8PDT)
_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 01-04-2011, 11:52 AM
Ales Kozumplik
 
Default mpath: filter out the slave devices and their partitions.

On 01/03/2011 07:22 PM, Brian C. Lane wrote:

On Wed, Dec 22, 2010 at 11:22:41AM +0100, Ales Kozumplik wrote:

+ sample input:
+ [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2, dm-0]
+ where:
+ [sdb, sdc] is a multipath pair
+ dm-0 is a mutliapth device already coalesced from [sdb, sdc]
+
+ sample output:
+ [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]


Could you explain the return values in more detail? Is this always a 3
element list or does it depend on the number of devices? What does it
look like with no multipath devices?

Thanks,
Brian


Yes,

it is always a tuple of 3 lists, the middle list is empty if there are
no multipath devices found. I'll make sure to document it before pushing.


Ales

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 01-07-2011, 12:38 PM
Ales Kozumplik
 
Default mpath: filter out the slave devices and their partitions.

This change necessitates from two previous commits:
1) 6e6d3df9a78730996e6ac7d463e3b50dbc164b4c, resulting in
DeviceTree.populate() being called twice now, for the second time at the
end of autopart.

2) 5c83bbf01800ca86d1c8e4cd95d14480165b344c, which causes already
coalesced mpath devices (disks and partitions) never to be torn down
during anaconda run.

Given that, we don't want to let identifyMultipath give
DeviceTree.populate() existing mpath devices (dm-0, dm-1,
etc.). populate() would try to add those before the slave mpath members
(sda, sdb) and although the addUdevDevice() logic will eventually
recursively request sda and sdb while working on dm-0, the udev info we
get for sda in sdb at that point is obtained thorugh a call to
udev_get_block_device() which will not give us a correctly filled in
ID_FS_TYPE (unlike the enriched info dictionaries that idenitfyMultipath()
gives us).

Alternative solution would be to embed a layer between DeviceTree and
udev.py. This layer would remember the additional database values that are
not returned by udev itself but rather discovered by us, such as the
"multipath_member" value of ID_FS_TYPE.

For the partitions part, the patch only does what identifyMultipath()'s
documentation promises: "3) removes the individual members of an mpath's
partitions". This step is optional and to make the function better
defined: note that e.g. sda3 will be scanned later in populate() anyway as
it will naturally still appear in udev_get_block_devices().

Resolves: rhbz#636570
---
storage/devicelibs/mpath.py | 75 ++++++++++++++++++++++++++++--------------
storage/devicetree.py | 13 +++----
storage/udev.py | 22 ++++++++++---
3 files changed, 73 insertions(+), 37 deletions(-)

diff --git a/storage/devicelibs/mpath.py b/storage/devicelibs/mpath.py
index 0d0efde..fb7e1aa 100644
--- a/storage/devicelibs/mpath.py
+++ b/storage/devicelibs/mpath.py
@@ -7,6 +7,31 @@ import logging

log = logging.getLogger("storage")

+def _filter_out_mpath_devices(devices):
+ retval = []
+ for d in devices:
+ if udev_device_is_dm_mpath(d):
+ log.debug("filtering out coalesced mpath device: %s" % d['name'])
+ else:
+ retval.append(d)
+ return retval
+
+def _filter_out_mpath_partitions(devices, multipaths):
+ """
+ Use serial numbers of the multipath members to filter devices from the
+ devices list. The returned list will only partitions that are NOT partitions
+ of the multipath members.
+ """
+ serials = set(udev_device_get_serial(d)
+ for mpath_members in multipaths for d in mpath_members)
+ retval = []
+ for d in devices:
+ if udev_device_get_serial(d) in serials:
+ log.debug("filtering out mpath partition: %s" % d['name'])
+ else:
+ retval.append(d)
+ return retval
+
def parseMultipathOutput(output):
"""
Parse output from "multipath -d" or "multipath -ll" and form a topology.
@@ -84,14 +109,27 @@ def parseMultipathOutput(output):
return mpaths

def identifyMultipaths(devices):
- # this function does a couple of things
- # 1) identifies multipath disks
- # 2) sets their ID_FS_TYPE to multipath_member
- # 3) removes the individual members of an mpath's partitions
- # sample input with multipath pair [sdb,sdc]
- # [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2]
- # sample output:
- # [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]
+ """
+ This function does a couple of things:
+ 1) identifies multipath disks,
+ 2) sets their ID_FS_TYPE to multipath_member,
+ 3) removes the individual members of an mpath's partitions as well as any
+ coalesced multipath devices:
+
+ The return value is a tuple of 3 lists, the first list containing all
+ devices except multipath members and partitions, the second list containing
+ only the multipath members and the last list only partitions. Specifically,
+ the second list is empty if there are no multipath devices found.
+
+ sample input:
+ [sr0, sda, sda1, sdb, sdb1, sdb2, sdc, sdc1, sdd, sdd1, sdd2, dm-0]
+ where:
+ [sdb, sdc] is a multipath pair
+ dm-0 is a mutliapth device already coalesced from [sdb, sdc]
+
+ sample output:
+ [sda, sdd], [[sdb, sdc]], [sr0, sda1, sdd1, sdd2]]
+ """
log.info("devices to scan for multipath: %s" % [d['name'] for d in devices])

with open("/etc/multipath.conf") as conf:
@@ -165,23 +203,10 @@ def identifyMultipaths(devices):

multipaths.append([devmap[d] for d in disks])

- non_disk_serials = {}
- for name,device in non_disk_devices.items():
- serial = udev_device_get_serial(device)
- non_disk_serials.setdefault(serial, [])
- non_disk_serials[serial].append(device)
-
- for mpath in multipaths:
- for serial in [d.get('ID_SERIAL_SHORT') for d in mpath]:
- if non_disk_serials.has_key(serial):
- log.info("filtering out non disk devices [%s]" % [d['name'] for d in non_disk_serials[serial]])
- for name in [d['name'] for d in non_disk_serials[serial]]:
- if non_disk_devices.has_key(name):
- del non_disk_devices[name]
-
- partition_devices = []
- for device in non_disk_devices.values():
- partition_devices.append(device)
+ # singlepaths and partitions should not contain multipath devices:
+ singlepath_disks = _filter_out_mpath_devices(singlepath_disks)
+ partition_devices = _filter_out_mpath_partitions(
+ non_disk_devices.values(), multipaths)

# this is the list of devices we want to keep from the original
# device list, but we want to maintain its original order.
diff --git a/storage/devicetree.py b/storage/devicetree.py
index 45fc455..f45e656 100644
--- a/storage/devicetree.py
+++ b/storage/devicetree.py
@@ -2026,19 +2026,18 @@ class DeviceTree(object):
% (livetarget,))
self.protectedDevNames.append(livetarget)

- # First iteration - let's just look for disks.
- old_devices = {}
-
- devices = udev_get_block_devices()
- for dev in devices:
- old_devices[dev['name']] = dev
-
cfg = self.__multipathConfigWriter.write()
open("/etc/multipath.conf", "w+").write(cfg)
del cfg

+ devices = udev_get_block_devices()
(singles, mpaths, partitions) = devicelibs.mpath.identifyMultipaths(devices)
devices = singles + reduce(list.__add__, mpaths, []) + partitions
+ # remember all the devices idenitfyMultipaths() gave us at this point
+ old_devices = {}
+ for dev in devices:
+ old_devices[dev['name']] = dev
+
log.info("devices to scan: %s" % [d['name'] for d in devices])
for dev in devices:
self.addUdevDevice(dev)
diff --git a/storage/udev.py b/storage/udev.py
index 59b7519..c489da5 100644
--- a/storage/udev.py
+++ b/storage/udev.py
@@ -399,10 +399,22 @@ def udev_device_get_lv_attr(info):
attr = [attr]
return attr

-def udev_device_dm_subsystem_match(info, subsystem):
- """ Return True if the device matches a given device-mapper subsystem. """
+def udev_device_dm_subsystem_match(info, subsystem, accept_partitions=False):
+ """ Return True if the device matches a given device-mapper subsystem.
+
+ Examples of DM_UUID values, mpath device:
+ mpath-2003013842bcb000c
+ partition of an mpath device:
+ part1-mpath-2003013842bcb000c
+ """
uuid = info.get("DM_UUID", "")
- _subsystem = uuid.split("-")[0]
+ split = uuid.split("-")
+ subsystem_idx = 0
+ if (accept_partitions and
+ len(split) > 1 and
+ split[0].startswith("part")):
+ subsystem_idx += 1
+ _subsystem = split[subsystem_idx]
if _subsystem == uuid or not _subsystem:
return False

@@ -431,8 +443,8 @@ def udev_device_is_dm_raid(info):
return udev_device_dm_subsystem_match(info, "dmraid")

def udev_device_is_dm_mpath(info):
- """ Return True if the device is an. """
- return udev_device_dm_subsystem_match(info, "mpath")
+ """ Return True if the device is a multipath device. """
+ return udev_device_dm_subsystem_match(info, "mpath", accept_partitions=True)

def udev_device_is_biosraid(info):
# Note that this function does *not* identify raid sets.
--
1.7.3.3

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 01-07-2011, 02:02 PM
David Lehman
 
Default mpath: filter out the slave devices and their partitions.

Check the version of udev_device_dm_subsystem_match on master -- it does
what you want done.

Dave


_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 
Old 01-10-2011, 11:03 AM
Ales Kozumplik
 
Default mpath: filter out the slave devices and their partitions.

On 01/07/2011 04:02 PM, David Lehman wrote:

Check the version of udev_device_dm_subsystem_match on master -- it does
what you want done.

Dave


Thanks,

I picked the patch that did that back to rhel6-branch.

Al_

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/anaconda-devel-list
 

Thread Tools




All times are GMT. The time now is 04:56 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org