FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor


 
 
LinkBack Thread Tools
 
Old 07-22-2008, 11:22 AM
"Vardaris, C - SPLXM"
 
Default Losing paths

To
all,


Â*


Is it
normal DM behavior that we are (temporary) losing paths from ‘unaffected’
LUN’s when we unmap a SAN LUN from the storage side.


We
are using an IBM SVC (4.2.1) and RH AS 4.6. We are using the multipath.conf
file supplied by IBM for the SVC.


Â*


It’s
normal that you loose the paths of the LUN which you unmap (4 paths in our
case), but why we are also losing paths from ‘unaffected’ LUN’s.


Unmapping
a LUN is not a normal storage admin action but could happen accidentally.


We
also see the same behavior when we give a new LUN to the Linux server. The
minute we map the LUN, paths from other LUN’s are temporarily disappearing.


Â*


We
tried several settings/HBA firmware etc. but can not eliminate this behavior of
DM.


Â*


Â*


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:176: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:176 in map
360050768018180a4f8000000000001d2


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:176.


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:192: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:192 in map
360050768018180a4f8000000000001d3


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:192.


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:208: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:208 in map
360050768018180a4f800000000000354


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:208.


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:240: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:240 in map
360050768018180a4f8000000000001d1


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:240.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:0: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:0 in map
360050768018180a4f8000000000001d0


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:0.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:32: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:32 in map
360050768018180a4f8000000000001ce


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:32.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:48.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:48: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:64.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:48 in map
360050768018180a4f8000000000001d2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:80.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:96.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:64: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:112.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:64 in map
360050768018180a4f8000000000001d3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:128.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:160.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:80: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:176.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:80 in map
360050768018180a4f800000000000354


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:192.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:208.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:96: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:224.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:96 in map
360050768018180a4f8000000000001d4


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:240.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 3


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:0.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:112: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:32.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:112 in map
360050768018180a4f8000000000001d1


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:48.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:64.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:128: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:80.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:128 in map
360050768018180a4f8000000000001d0


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:96.


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:112.


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:160: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:128.


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:160 in map
360050768018180a4f8000000000001ce


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:176: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:176 in map
360050768018180a4f8000000000001d2


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:192: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:192 in map
360050768018180a4f8000000000001d3


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:208: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:208 in map
360050768018180a4f800000000000354


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:224: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:224 in map 360050768018180a4f8000000000001d4


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 2


JulÂ*
2 13:36:54 kl1108ju multipathd: 67:240: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 67:240 in map
360050768018180a4f8000000000001d1


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:0: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:0 in map
360050768018180a4f8000000000001d0


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:32: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:32 in map
360050768018180a4f8000000000001ce


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:48: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:48 in map
360050768018180a4f8000000000001d2


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 0


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:64: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:64 in map
360050768018180a4f8000000000001d3


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 0


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:80: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:80 in map
360050768018180a4f800000000000354


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 0


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:96: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:96 in map
360050768018180a4f8000000000001d4


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 1


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:112: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:112 in map
360050768018180a4f8000000000001d1


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 0


JulÂ*
2 13:36:54 kl1108ju multipathd: 65:128: tur checker reports path is down


JulÂ*
2 13:36:54 kl1108ju multipathd: checker failed path 65:128 in map
360050768018180a4f8000000000001d0


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 0


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering
recovery mode: max_retries=5


JulÂ*
2 13:36:55 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering
recovery mode: max_retries=5


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:176: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:176: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2:
queue_if_no_path enabled


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Recovered to
normal mode


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 1


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:192: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:192: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3:
queue_if_no_path enabled


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Recovered to
normal mode


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:208: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:208: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354:
queue_if_no_path enabled


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: Recovered to
normal mode


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:240: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:240: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: queue_if_no_path
enabled


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Recovered to
normal mode


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to
path group #2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:0: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:0: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0:
queue_if_no_path enabled


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Recovered to
normal mode


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 1


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:32: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:32: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:48: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:48: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 2


Jul Â*2
13:37:24 kl1108ju multipathd: 67:64: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:64: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:80: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:80: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:96: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:96: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:112: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:112: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:128: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:128: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 2


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:160: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:160: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:176: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:176: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:192: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:192: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:208: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:208: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:224: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:224: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:240: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 67:240: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 3


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to
path group #1


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:0: tur checker reports path is up


JulÂ*
2 13:37:24 kl1108ju multipathd: 65:0: reinstated


JulÂ*
2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining
active paths: 3


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:32: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:32: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:48: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:48: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:64: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:64: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:80: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:80: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:96: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:96: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:112: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:112: reinstated


JulÂ*
2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining
active paths: 4


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:128: tur checker reports path is up


JulÂ*
2 13:37:25 kl1108ju multipathd: 65:128: reinstated


Â*


Â*


Kind regards,


ChrisÂ*Â*


Â*


Â*


Â*


Â*


Â*








************************************************** ********************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286 ************************************************** ********************
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-22-2008, 02:44 PM
"Romanowski, John (OFT)"
 
Default Losing paths

Hi,
We had similar problem last year using sles9 and SVC 4.2.0.2, as you describe: adding/deleting a LUN causes brief path failures for the host's remaining, unaffected LUNs.

We were using tur checker and it turned out that while adding/deleting a LUN (and some other admin tasks) the SVC does not respond well to test-unit-ready tur requests; but it responds perfectly well to normal read commands. I opened IBM PMR 43118 on that if you want to ask the SVC folks about it.

Workaround for us was to use readsector0 instead of tur as multipath path checker.

Recent post here (see July 8, 2008) said multipath-tools is deprecating readsector0, and to use directio as path checker, but directio was said to be much slower than tur, implying tur was better replacement than directio for readsector0.

Not sure what deprecating readsector0 means for us SVC users.

regards,
John



--------------------------------------------------------
This e-mail, including any attachments, may be confidential, privileged or otherwise legally protected. It is intended only for the addressee. If you received this e-mail in error or from someone who was not authorized to send it to you, do not disseminate, copy or otherwise use this e-mail or its attachments. Please notify the sender immediately by reply e-mail and delete the e-mail from your system.


________________________________________

From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com] On Behalf Of Vardaris, C - SPLXM
Sent: Tuesday, July 22, 2008 7:23 AM
To: dm-devel@redhat.com
Subject: [dm-devel] Losing paths

To all,

Is it normal DM behavior that we are (temporary) losing paths from 'unaffected' LUN's when we unmap a SAN LUN from the storage side.
We are using an IBM SVC (4.2.1) and RH AS 4.6. We are using the multipath.conf file supplied by IBM for the SVC.

It's normal that you loose the paths of the LUN which you unmap (4 paths in our case), but why we are also losing paths from 'unaffected' LUN's.
Unmapping a LUN is not a normal storage admin action but could happen accidentally.
We also see the same behavior when we give a new LUN to the Linux server. The minute we map the LUN, paths from other LUN's are temporarily disappearing.

We tried several settings/HBA firmware etc. but can not eliminate this behavior of DM.


Jul 2 13:36:54 kl1108ju multipathd: 65:176: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:176 in map 360050768018180a4f8000000000001d2
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:176.
Jul 2 13:36:54 kl1108ju multipathd: 65:192: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:192 in map 360050768018180a4f8000000000001d3
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:192.
Jul 2 13:36:54 kl1108ju multipathd: 65:208: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:208 in map 360050768018180a4f800000000000354
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:208.
Jul 2 13:36:54 kl1108ju multipathd: 65:240: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:240 in map 360050768018180a4f8000000000001d1
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:240.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 3
Jul 2 13:36:54 kl1108ju multipathd: 67:0: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:0 in map 360050768018180a4f8000000000001d0
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:0.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 3
Jul 2 13:36:54 kl1108ju multipathd: 67:32: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:32 in map 360050768018180a4f8000000000001ce
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:32.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:48.
Jul 2 13:36:54 kl1108ju multipathd: 67:48: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:64.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:48 in map 360050768018180a4f8000000000001d2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:80.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:96.
Jul 2 13:36:54 kl1108ju multipathd: 67:64: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:112.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:64 in map 360050768018180a4f8000000000001d3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:128.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:160.
Jul 2 13:36:54 kl1108ju multipathd: 67:80: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:176.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:80 in map 360050768018180a4f800000000000354
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:192.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:208.
Jul 2 13:36:54 kl1108ju multipathd: 67:96: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:224.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:96 in map 360050768018180a4f8000000000001d4
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 67:240.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 3
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:0.
Jul 2 13:36:54 kl1108ju multipathd: 67:112: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:32.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:112 in map 360050768018180a4f8000000000001d1
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:48.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:64.
Jul 2 13:36:54 kl1108ju multipathd: 67:128: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:80.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:128 in map 360050768018180a4f8000000000001d0
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:96.
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 2
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:112.
Jul 2 13:36:54 kl1108ju multipathd: 67:160: tur checker reports path is down
Jul 2 13:36:54 kl1108ju kernel: device-mapper: dm-multipath: Failing path 65:128.
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:160 in map 360050768018180a4f8000000000001ce
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 2
Jul 2 13:36:54 kl1108ju multipathd: 67:176: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:176 in map 360050768018180a4f8000000000001d2
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 67:192: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:192 in map 360050768018180a4f8000000000001d3
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 67:208: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:208 in map 360050768018180a4f800000000000354
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 67:224: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:224 in map 360050768018180a4f8000000000001d4
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 2
Jul 2 13:36:54 kl1108ju multipathd: 67:240: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 67:240 in map 360050768018180a4f8000000000001d1
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 65:0: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:0 in map 360050768018180a4f8000000000001d0
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 65:32: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:32 in map 360050768018180a4f8000000000001ce
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 65:48: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:48 in map 360050768018180a4f8000000000001d2
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 0
Jul 2 13:36:54 kl1108ju multipathd: 65:64: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:64 in map 360050768018180a4f8000000000001d3
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 0
Jul 2 13:36:54 kl1108ju multipathd: 65:80: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:80 in map 360050768018180a4f800000000000354
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 0
Jul 2 13:36:54 kl1108ju multipathd: 65:96: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:96 in map 360050768018180a4f8000000000001d4
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 1
Jul 2 13:36:54 kl1108ju multipathd: 65:112: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:112 in map 360050768018180a4f8000000000001d1
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 0
Jul 2 13:36:54 kl1108ju multipathd: 65:128: tur checker reports path is down
Jul 2 13:36:54 kl1108ju multipathd: checker failed path 65:128 in map 360050768018180a4f8000000000001d0
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 0
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f800000000000354: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering recovery mode: max_retries=5
Jul 2 13:36:54 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Entering recovery mode: max_retries=5
Jul 2 13:36:55 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Entering recovery mode: max_retries=5
Jul 2 13:37:24 kl1108ju multipathd: 65:176: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 65:176: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: queue_if_no_path enabled
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: Recovered to normal mode
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 1
Jul 2 13:37:24 kl1108ju multipathd: 65:192: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 65:192: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: queue_if_no_path enabled
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: Recovered to normal mode
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 65:208: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 65:208: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: queue_if_no_path enabled
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: Recovered to normal mode
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 65:240: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 65:240: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: queue_if_no_path enabled
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: Recovered to normal mode
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to path group #2
Jul 2 13:37:24 kl1108ju multipathd: 67:0: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:0: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: queue_if_no_path enabled
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: Recovered to normal mode
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 1
Jul 2 13:37:24 kl1108ju multipathd: 67:32: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:32: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:48: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:48: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:64: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:64: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:80: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:80: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:96: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:96: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:112: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:112: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:128: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:128: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 2
Jul 2 13:37:24 kl1108ju multipathd: 67:160: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:160: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 67:176: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:176: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 67:192: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:192: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d3: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 67:208: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:208: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f800000000000354: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 67:224: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:224: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 67:240: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 67:240: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 3
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d1: switch to path group #1
Jul 2 13:37:24 kl1108ju multipathd: 65:0: tur checker reports path is up
Jul 2 13:37:24 kl1108ju multipathd: 65:0: reinstated
Jul 2 13:37:24 kl1108ju multipathd: 360050768018180a4f8000000000001d0: remaining active paths: 3
Jul 2 13:37:25 kl1108ju multipathd: 65:32: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:32: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001ce: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:48: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:48: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d2: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:64: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:64: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d3: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:80: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:80: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f800000000000354: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:96: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:96: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d4: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:112: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:112: reinstated
Jul 2 13:37:25 kl1108ju multipathd: 360050768018180a4f8000000000001d1: remaining active paths: 4
Jul 2 13:37:25 kl1108ju multipathd: 65:128: tur checker reports path is up
Jul 2 13:37:25 kl1108ju multipathd: 65:128: reinstated


Kind regards,
Chris


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-22-2008, 02:55 PM
Hannes Reinecke
 
Default Losing paths

Hi John,

Romanowski, John (OFT) wrote:

Hi,
We had similar problem last year using sles9 and SVC 4.2.0.2, as you describe: adding/deleting a LUN causes
brief path failures for the host's remaining, unaffected LUNs.

We were using tur checker and it turned out that while adding/deleting a LUN (and some other admin tasks)
the SVC does not respond well to test-unit-ready tur requests; but it responds perfectly well to normal read
commands. I opened IBM PMR 43118 on that if you want to ask the SVC folks about it.

Workaround for us was to use readsector0 instead of tur as multipath path checker.

Recent post here (see July 8, 2008) said multipath-tools is deprecating readsector0, and to use directio as

path checker, but directio was said to be much slower than tur, implying tur was better replacement than
directio for readsector0.


Only for those cases where no actual read-access is necessary.

Not sure what deprecating readsector0 means for us SVC users.


It means you should be using directio there.

Thanks for the information, I'll see to have it included in SLES10.

Cheers,

Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-22-2008, 04:21 PM
"Daniel Keisling"
 
Default Losing paths

FWIW, I have the
same issue under an HP EVA8000.* I'm not really certain if directio is a
solution as I have only seen the tur checker being used with an
EVA.
*
Daniel
*
*
---------------------------------------------------------------------------------------------
Date: Tue, 22 Jul
2008 16:55:27 +0200
From: Hannes Reinecke <hare@suse.de>
Subject: Re: [dm-devel]
Losing paths
To: device-mapper development <dm-devel@redhat.com>
Message-ID:
<4885F4DF.8010000@suse.de>
Content-Type:
text/plain; charset=ISO-8859-1; format=flowed
*
Hi
John,
*
Romanowski, John
(OFT) wrote:
> Hi,
> We had similar problem last year using sles9
and SVC 4.2.0.2,* as you describe: adding/deleting a LUN causes
>
brief path failures for the host's remaining, unaffected LUNs.
>
>
We were using tur checker and it turned out that while adding/deleting a LUN
(and some other admin tasks)
> the SVC does not respond well to
test-unit-ready tur requests; but it responds perfectly well to normal
read
> commands. I opened IBM PMR 43118 on that if you want to ask the SVC
folks about it.*
>
> Workaround for us was to use readsector0
instead of* tur as multipath path checker.
>*
> Recent
post here (see July 8, 2008)* said multipath-tools is deprecating
readsector0, and to use directio as
> path checker, but directio was said
to be much slower than tur, implying tur was better replacement than
>
directio for readsector0.
>
Only for those cases where no actual
read-access is necessary.
*
> Not sure what deprecating
readsector0 means for us SVC users.*
>
It means you should be
using directio there.
*
Thanks for the
information, I'll see to have it included in SLES10.
*
Cheers,
*
Hannes
--
Dr.
Hannes Reinecke******* zSeries &
Storage
hare@suse.de********
+49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409
Nürnberg
GF: Markus Rex, HRB 16746 (AG
Nürnberg)





__________________________________________________ ____________________

This*email*transmission*and*any*documents,*files*o r*previous*email

messages*attached*to*it*may*contain*information*th at*is*confidential*or

legally*privileged.*If*you*are*not*the*intended*re cipient*or*a*person

responsible*for*delivering*this*transmission*to*th e*intended*recipient,

you*are*hereby*notified*that*you*must*not*read*thi s*transmission*and

that*any*disclosure,*copying,*printing,*distributi on*or*use*of*this

transmission*is*strictly*prohibited.*If*you*have*r eceived*this*transmission

in*error,*please*immediately*notify*the*sender*by* telephone*or*return*email

and*delete*the*original*transmission*and*its*attac hments*without*reading

or*saving*in*any*manner.


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-22-2008, 10:10 PM
guy keren
 
Default Losing paths

Does the 'tur' checker take into account unit-attention issues (i.e.
when the configuration of a SCSI device changes, the next command coming
from an afected initiator (not necessarily to the same LU) should be
failed with a 'check condition' error and a 'unit attention' sense code.


if the TUR checker sent a TUR right after the change and before another
read/write command was sent - it will be failed, and then the question
is, whether the lower-level driver retries the TUR command, or returns
an error. any idea?


--guy

Daniel Keisling wrote:
FWIW, I have the same issue under an HP EVA8000. I'm not really certain
if directio is a solution as I have only seen the tur checker being used
with an EVA.

Daniel


---------------------------------------------------------------------------------------------

Date: Tue, 22 Jul 2008 16:55:27 +0200
From: Hannes Reinecke <hare@suse.de <mailto:hare@suse.de>>
Subject: Re: [dm-devel] Losing paths
To: device-mapper development <dm-devel@redhat.com
<mailto:dm-devel@redhat.com>>

Message-ID: <4885F4DF.8010000@suse.de <mailto:4885F4DF.8010000@suse.de>>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi John,

Romanowski, John (OFT) wrote:

Hi,
We had similar problem last year using sles9 and SVC 4.2.0.2, as you

describe: adding/deleting a LUN causes

brief path failures for the host's remaining, unaffected LUNs.

We were using tur checker and it turned out that while adding/deleting

a LUN (and some other admin tasks)
the SVC does not respond well to test-unit-ready tur requests; but it

responds perfectly well to normal read
commands. I opened IBM PMR 43118 on that if you want to ask the SVC
folks about it.


Workaround for us was to use readsector0 instead of tur as multipath

path checker.


Recent post here (see July 8, 2008) said multipath-tools is

deprecating readsector0, and to use directio as
path checker, but directio was said to be much slower than tur,

implying tur was better replacement than

directio for readsector0.


Only for those cases where no actual read-access is necessary.

Not sure what deprecating readsector0 means for us SVC users.


It means you should be using directio there.

Thanks for the information, I'll see to have it included in SLES10.

Cheers,

Hannes

--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de <mailto:hare@suse.de> +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)




__________________________________________________ ____________________
This email transmission and any documents, files or previous email
messages attached to it may contain information that is confidential or
legally privileged. If you are not the intended recipient or a person
responsible for delivering this transmission to the intended recipient,
you are hereby notified that you must not read this transmission and
that any disclosure, copying, printing, distribution or use of this
transmission is strictly prohibited. If you have received this transmission
in error, please immediately notify the sender by telephone or return email
and delete the original transmission and its attachments without reading
or saving in any manner.


------------------------------------------------------------------------

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-23-2008, 06:25 AM
Hannes Reinecke
 
Default Losing paths

Hi Guy,

guy keren wrote:


Does the 'tur' checker take into account unit-attention issues (i.e.
when the configuration of a SCSI device changes, the next command coming
from an afected initiator (not necessarily to the same LU) should be
failed with a 'check condition' error and a 'unit attention' sense code.


if the TUR checker sent a TUR right after the change and before another
read/write command was sent - it will be failed, and then the question
is, whether the lower-level driver retries the TUR command, or returns
an error. any idea?



It does for SLES10 SP2. Upstream not; I'll have to send the
patches one day.

Cheers,

Hannes
--
Dr. Hannes Reinecke zSeries & Storage
hare@suse.de +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-23-2008, 08:44 AM
"Vardaris, C - SPLXM"
 
Default Losing paths

Greetings,

Thanks for all the input so far.

This afternoon we are going to test with directio, but is this an
usable/reliable setting in a Linux cluster environment?


Kind regards,
Chris
************************************************** ********************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286
************************************************** ********************

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-23-2008, 12:33 PM
"Koehler, M - SPLXM"
 
Default Losing paths

Greetings,

We tried testing with path_checker readsector0 end directio but we get
some strange behavior eg the checker used is always tur!!
So if we create a device entry special for IBM 2145 and define as
path_checker directio or readsector0 we still see in the message file
tur messages:

Jul 23 08:09:00 kl11071i multipathd: 65:0: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:16: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:32: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:48: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:80: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:96: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:112: tur checker reports path is
down


Oure multipath.conf file is:

defaults {
user_friendly_names yes
}

devices{
device {
vendor "IBM"
product "2145"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s"
prio_callout "/sbin/mpath_prio_alua %d"
path_checker readsector0
failback immediate
}
}

Did someone see this behavior before? Or are we doing something wrong?
We are using device-mapper-multipath-0.4.5-27.el4_6.3 on a rhel4.6
system.

Kind regards,
Menno
************************************************** ********************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286
************************************************** ********************

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-23-2008, 01:12 PM
"Romanowski, John (OFT)"
 
Default Losing paths

Did you change multipath.conf and reboot the machine?
if not reboot, what steps put the changed multipath.conf into effect
for the 2145's (SVC) LUNs?


--------------------------------------------------------
This e-mail, including any attachments, may be confidential, privileged or otherwise legally protected. It is intended only for the addressee. If you received this e-mail in error or from someone who was not authorized to send it to you, do not disseminate, copy or otherwise use this e-mail or its attachments. Please notify the sender immediately by reply e-mail and delete the e-mail from your system.


-----Original Message-----

From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Koehler, M - SPLXM
Sent: Wednesday, July 23, 2008 8:34 AM
To: device-mapper development
Subject: RE: [dm-devel] Losing paths

Greetings,

We tried testing with path_checker readsector0 end directio but we get
some strange behavior eg the checker used is always tur!!
So if we create a device entry special for IBM 2145 and define as
path_checker directio or readsector0 we still see in the message file
tur messages:

Jul 23 08:09:00 kl11071i multipathd: 65:0: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:16: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:32: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:48: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:80: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:96: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:112: tur checker reports path is
down


Oure multipath.conf file is:

defaults {
user_friendly_names yes
}

devices{
device {
vendor "IBM"
product "2145"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s"
prio_callout "/sbin/mpath_prio_alua %d"
path_checker readsector0
failback immediate
}
}

Did someone see this behavior before? Or are we doing something wrong?
We are using device-mapper-multipath-0.4.5-27.el4_6.3 on a rhel4.6
system.

Kind regards,
Menno
************************************************** ********************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286
************************************************** ********************

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 07-23-2008, 01:22 PM
"Koehler, M - SPLXM"
 
Default Losing paths

Hello,

Thanks for your respond, first I tried the multipathd -k reconfigure
(and get a ok returned) and later when seen that it still was using tur
we tried a reboot but with the same result.

Kind regards,

Menno


-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Romanowski, John (OFT)
Sent: Wednesday, July 23, 2008 3:13 PM
To: device-mapper development
Subject: RE: [dm-devel] Losing paths

Did you change multipath.conf and reboot the machine?
if not reboot, what steps put the changed multipath.conf into effect
for the 2145's (SVC) LUNs?


--------------------------------------------------------
This e-mail, including any attachments, may be confidential, privileged
or otherwise legally protected. It is intended only for the addressee.
If you received this e-mail in error or from someone who was not
authorized to send it to you, do not disseminate, copy or otherwise use
this e-mail or its attachments. Please notify the sender immediately by
reply e-mail and delete the e-mail from your system.


-----Original Message-----

From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Koehler, M - SPLXM
Sent: Wednesday, July 23, 2008 8:34 AM
To: device-mapper development
Subject: RE: [dm-devel] Losing paths

Greetings,

We tried testing with path_checker readsector0 end directio but we get
some strange behavior eg the checker used is always tur!!
So if we create a device entry special for IBM 2145 and define as
path_checker directio or readsector0 we still see in the message file
tur messages:

Jul 23 08:09:00 kl11071i multipathd: 65:0: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:16: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:32: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:48: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:80: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:96: tur checker reports path is
down
Jul 23 08:09:00 kl11071i multipathd: 65:112: tur checker reports path is
down


Oure multipath.conf file is:

defaults {
user_friendly_names yes
}

devices{
device {
vendor "IBM"
product "2145"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s"
prio_callout "/sbin/mpath_prio_alua %d"
path_checker readsector0
failback immediate
}
}

Did someone see this behavior before? Or are we doing something wrong?
We are using device-mapper-multipath-0.4.5-27.el4_6.3 on a rhel4.6
system.

Kind regards,
Menno
************************************************** ********************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286
************************************************** ********************

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
************************************************** ********************
For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee
only. If you are not the addressee, you are notified that no part
of the e-mail or any attachment may be disclosed, copied or
distributed, and that any other action related to this e-mail or
attachment is strictly prohibited, and may be unlawful. If you have
received this e-mail by error, please notify the sender immediately
by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries
and/or its employees shall not be liable for the incorrect or
incomplete transmission of this e-mail or any attachments, nor
responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
Dutch Airlines) is registered in Amstelveen, The Netherlands, with
registered number 33014286
************************************************** ********************

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 05:09 AM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright ©2007 - 2008, www.linux-archive.org