FAQ Search Today's Posts Mark Forums Read
» Video Reviews

» Linux Archive

Linux-archive is a website aiming to archive linux email lists and to make them easily accessible for linux users/developers.


» Sponsor

» Partners

» Sponsor

Go Back   Linux Archive > Redhat > Device-mapper Development

 
 
LinkBack Thread Tools
 
Old 01-20-2012, 01:49 PM
Fil
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

Thanks for the patches Vijay,

I tested the new set of patches against the git repo, and it applied
cleanly compiled and is running as expected. I will do more indepth
testing over the weekend.

thanks
fil

On 01/19/2012 02:20 PM, Chauhan, Vijay wrote:
> Resending this patch with the fix mentioned by Mike Snitzer<snitzer@redhat.com> and Fil<lists@internyc.net>.
> Thanks to all for sending me your review comments. Fil, Thanks for trying out mpathpersist and providing your
> feedback. Also thanking Yanling Qi<Yanling.Qi@netapp.com> for proposing this design for pr management feature.
> These series of patch is made on commit 864162b08067d3e55d52996846d10a046956c6d6.
>
>
> PERSISTENT RESERVE OUT/IN commands are currently not supported on mpath device. Any
> command sent to mpath device is routed to only one of the physical path (selected by
> path selector) from the active path group. PR OUT registration service action is one
> of the such use-case which fails as it expects all the physical path for the given mpath
> device to be registered. Due to these limitations, most of the cluster applications needs
> to manage persistent reservation through underlying physical path.
>
>
> Following are the detailed description for Persistent Reservation management feature:
> ================================================== ====================================
> CONTENT
> ================================================== =====================================
> 1. Feature Introduction
> 2. Steps to configure for persistent reservation management
> 3. mpathpersist Command Usage
> 4. PR IN/OUT service action and use-case handling
> 4.1 Persistent Reserve Out service action
> 4.1.1 [Register]
> 4.1.2 [Reserve]
> 4.1.3 [Release]
> 4.1.4 [CLEAR]
> 4.1.5 [PREEMPT] & [PREEMPT AND ABORT]
> 4.2 Persistent Reserve In service action
> 4.2.1 [READ KEY]
> 4.2.2 [READ RESERVATION]
> 4.2.3 [READ STATUS]
> 4.2.4 [REPORT CAPABILITIES]
> 4.3 Usage case handled by multipath tools for PR management
> 5. mpathpersist library and header
> 5.1 mpathpersist Library APIs.
> 5.1.1 int mpath_lib_init (void )
> 5.1.2 int mpath_lib_exit (void )
> 5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact,
> struct prin_resp *resp, int noisy, int verbose)
> 5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact,
> int rq_scope, unsigned int rq_type, struct rout_param_descriptor *paramp,
> int noisy, int verbose)
> 6. Limitation and restriction.
>
>
>
> 1. Feature Introduction
> ==========================
>
> Persistent reservation management feature allows cluster management software to manage
> persistent reservation through mpath device. It processes management requests from callers
> and hides the management task details. It also handles persistent reservation management of
> data path life cycle and state changes.
>
> The series of patch introduces new utility named 'mpathpersist' for managing pr on multipath device.
> 'mpathpersist' is similar to sg_persist utility with respect to cli usage but
> applicable to mpath devices only.
>
>
> 2. Steps to configure for persistent reservation management
> ================================================== ===========
> This feature adds new parameter 'reservation_key' in default section and multipath section of
> multipath.conf.
>
> 'reservation_key' here represents RESERVATION KEY field of PERSISTENT RESERVE OUT parameter
> list which contains an 8-byte value provided by the application client to the device server
> to identify the I_T nexus.
>
> STEP 1: Set reservation_key parameter in multipath.conf for mpath device. 'reservation_key'
> parameter needs to be set for all the mpath device applicable for persistent management.
> STEP 2: restart multipathd daemon
>
> Note: reservation key in multipath.com must be same as service action reservation key used/to
> be used for mpath device registration reservation.
>
>
> 3. mpathpersist Command Usage
> ===============================
>
> mpathpersist CLI usage is kept close to sg_persist usage except it takes mpath device as device name.
>
> Usage: mpathpersist [OPTIONS] [DEVICE]
> Options:
> --verbose|-v level verbosity level
> 0 Critical messages
> 1 Error messages
> 2 Warning messages
> 3 Informational messages
> 4 Informational messages with trace enabled
> --clear|-C PR Out: Clear
> --device=DEVICE|-d DEVICE query or change DEVICE
> --help|-h output this usage message
> --hex|-H output response in hex
> --in|-i request PR In command
> --out|-o request PR Out command
> --param-aptpl|-Z PR Out parameter 'APTPL'
> --read-keys|-k PR In: Read Keys
> --param-sark=SARK|-S SARK PR Out parameter service action
> reservation key (SARK is in hex)
> --param-rk=RK|-K RK PR Out parameter reservation key
> (RK is in hex)
> --preempt|-P PR Out: Preempt
> --preempt-abort|-A PR Out: Preempt and Abort
> --prout-type=TYPE|-T TYPE PR Out command type
> --read-status|-s PR In: Read Full Status
> --read-keys|-k PR In: Read Keys
> --read-reservation|-r PR In: Read Reservation
> --register|-G PR Out: Register
> --register-ignore|-I PR Out: Register and Ignore
> --release|-L PR Out: Release
> --report-capabilities|-c PR In: Report Capabilities
> --reserve|-R PR Out: Reserve
> --transport-id=TIDS|-X TIDS TransportIDs can be mentioned
> in several forms
>
>
> 4. PR IN/OUT service action and use-case handling
> ================================================== =
>
> 4.1 Persistent Reserve Out service action:
> ----------------------------------
> 4.1.1 [Register]
> The caller provides multipath device name, reservation key, and service action reservation
> key and optionally APTPL bit value. The persistent reservation management command issues PROUT
> Register service action to all active data paths of the multipath device. If reservation key or
> service action reservation key is not provided, fill with all 0s. For each path, a thread is
> created to process PROUT register service action. Consolidated results of all the paths are
> returned to caller. In case if this PROUT command fails with reservation conflict, all the
> successful registration by other threads will be rolled back.
> Example: mpathpersist --out --register --param-sark=123abc --prout-type=8 /dev/mapper/mpath9
> /*Unregister*/
> mpathpersist --out --register --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.2 [Reserve]
> The caller provides multipath device name, reservation type and reservation key. The reservation SCOPE
> value is not required since persistent reservation supports only LU_SCOPE. The persistent reservation
> management command issues PROUT Reserve service action to any active data paths of the multipath
> device. PROUT command is send to one of the active path of multipath device. Result is returned to the
> caller of this module.
> Example: mpathpersist --out --reserve --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.3 [Release]
> The caller provides multipath device name, reservation type and reservation key. The persistent reservation
> management command issues PROUT Release service action to all active data paths of the multipath device. The
> reason why issuing PROUT Release service action is because it is unclear which data path is the reservation
> holder. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or
> reservation type is invalid. Return error status back to the caller. After PROUT Release service action command
> is issued to all active data paths, a PRIN Read Reservation service action should be issued to an active data
> path. If the Read Reservation parameter data indicates that the logical unit is still reserved and the registration
> key matches the reservation key caller provided, this indicates that the reservation holder belongs to a data
> path in failed state or removed data path of the multipath device.
> The following operations should be taken for this error case.
> 1. Issue a PRIN Report Full Status service action to any active data path
> 2. Remember the full status descriptors returned from the step 1.
> 3. Issue a PROUT clear reservation service action, via any active data path, to clear the reservation and registrants
> and
> 4. Restore all registrants by issuing PROUT register service action with transport IDs from the full status
> descriptors saved in the step 2.
> Example: mpathpersist --out --release --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.4 [CLEAR]
> The caller provides multipath device name and reservation key. The persistent reservation management
> command issues PROUT Preempt service action to any active data paths of the multipath device.
> If RESERVATION CONFLICT SCSI status is received, it indicates that the reservation key is invalid or the
> data path is not a registrant. Return error status back to the caller.
> Example: mpathpersist --out --clear --param-rk=123abc --prout-type=8 /dev/mapper/mpath9
>
>
> 4.1.5 [PREEMPT] & [PREEMPT AND ABORT]
> The caller provides multipath device name, reservation type, reservation key and service action reservation key.
> The persistent reservation management command issues PROUT Preempt service action to any active data paths of the
> multipath device. If RESERVATION CONFLICT SCSI status is received, it indicates that either the reservation key or
> service action reservation key is invalid. Return error status back to the caller. PROUT command is send to one of
> the active path of multipath device. Result is returned to the caller of this module.
> Example: mpathpersist --out --preempt-abort --param-rk=0 --paramsark=789def --prout-type=8 /dev/mapper/mpath9
>
>
> PS: This feature does not support ALL_TG_PT bit on. With a storage array having 8 or 16 target ports, the
> total number of registrants will be equal to total-number-of-initiator-ports X total-number-of target-ports.
> If ALL_TG_PT is on, too much storage array resources will be wasted and it also creates implementation
> challenge of managing registration rollback. In a use case where one initiator port (such as a FC port) connects
> to multiple target ports on the same storage array via a switch, ALL_TG_PT=1 will cause registration failed with
> RESERVATION CONFLICT status when issuing PROUT registration to the second data path which shares the same initiator port.
>
> PS: This RFC version does not support REGISTER AND MOVE service action.
>
> 4.2 Persistent Reserve In service action:
> --------------------------------------------
> 4.2.1 [READ KEY]
> The caller provides multipath device. The persistent reservation management command selects any active data path
> and issues a PRIN Read Keys service action to the selected data path of the multipath device. Parameter data is sent
> to the caller with appropriate formatted. The persistent reservation management command will not filter registration
> keys returned from the storage array. In other words, multiple registration keys is reported in general.
> Example: mpathpersist -i -k /dev/mapper/mpath9
>
>
> 4.2.2 [READ RESERVATION]
> The caller provides multipath device. The persistent reservation management command selects any active data path and
> issues a PRIN Read Reservation service action to the selected data path of the multipath device. Parameter data is
> sent to the caller with appropriate formatted.
> Example: mpathpersist -i -r /dev/mapper/mpath9
>
>
> 4.2.3 [READ STATUS]
> The caller provides multipath device. The persistent reservation management command selects any active data path and
> issues a PRIN Read Full Status service action to the selected data path of the multipath device. Parameter data is sent
> to the caller with appropriate formatted. The persistent reservation management command will not filter full status
> descriptor entries returned from the storage array. In other words, full statuses for multiple registrants is reported
> in general.
> Example: mpathpersist -i -s /dev/mapper/mpath9
>
>
> 4.2.4 [REPORT CAPABILITIES]
> The caller provides multipath device. The persistent reservation management command selects any active data path
> and issues a PRIN Report Capabilities service action to the selected data path of the multipath device. Parameter
> data is sent to the caller with appropriate formatted.
> Example: mpathpersist -i -c /dev/mapper/mpath9
>
>
> 4.3 Use-case handled by multipath tools for PR management
> - For any reinstated path, multipath daemon checks for persistent reservation on the path and registers the
> path with reservation_key (from multipath.conf) if required.
> - For any new path discovery, multipath daemon checks for persistent reservation on the path and registers the path if required.
> - In case of device all path failure condition, command is returns with failed status.
> - PR command is retried by pr management for any Unit Attention.
> - While performing persistent reservation registration on mpath device, if any of the path receives reservation conflict
> then successful registration on the other paths will be rolled back.
>
> 5. mpathpersist Library and Header
> ======================================
>
>
> mpathpersist library provides four APIs for managing persistent reservation on device mapper multipath device.
>
> 5.1 mpathpersist Library APIs.
> ---------------------------------
> 5.1.1 int mpath_lib_init (void)
> DESCRIPTION :
> Initialize device mapper multipath configuration. This function must be invoked first before performing reservation
> management functions.
>
> RETURNS:
> 0->Success, 1->Failed
> Note: mpath_lib_init function must be called at the start of the usage of PR management APIs (such as
> mpath_persistent_reserve_in or mpath_persistent_reserve_out). This function is responsible for initializing
> config table. (Any dynamic allocation done by mpath_lib_init function will be de-allocated only by mpath_lib_exit API).
>
>
> 5.1.2 int mpath_lib_exit (void)
> DESCRIPTION:
> Release device mapper multipath configuration. This function must be invoked after performing reservation management functions.
>
> RETURNS:
> 0->Success, 1->Failed.
>
> Note: mpath_lib_exit function must be called at the end of the usage of PR management APIs (such as mpath_persistent_reserve_in
> or mpath_persistent_reserve_out). This function is responsible for freeing up the dynamic allocation made by mpath_lib_init function.
>
>
> 5.1.3 int mpath_persistent_reserve_in (int fd, int rq_servact, struct prin_resp *resp, int noisy, int verbose)
> DESCRIPTION:
> The function in the mpath_persistent_reserve_in() sends PRIN command to the DM device and gets the response.
>
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact - PRIN command service action. Input argument
> resp - The response from PRIN service action. The caller should manage the memory allocation of this structure
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable
> verbose - Set verbosity level. Input argument. value:[0-3]. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
>
> RETURNS
> MPATH_PR_SUCCESS if PR command successful
> MPATH_PR_SYNTAX_ERROR if syntax error or invalid parameter
> MPATH_PR_SENSE_NOT_READY if command fails with [sk,asc,ascq: 0x2,*,*]
> MPATH_PR_SENSE_MEDIUM_ERROR if command fails with [sk,asc,ascq: 0x3,*,*]
> MPATH_PR_SENSE_HARDWARE_ERROR if command fails with [sk,asc,ascq: 0x4,*,*]
> MPATH_PR_SENSE_INVALID_OP if command fails with [sk,asc,ascq: 0x5,0x20,0x0]
> MPATH_PR_ILLEGAL_REQ if command fails with [sk,asc,ascq: 0x5,*,*]
> MPATH_PR_SENSE_UNIT_ATTENTION if command fails with [sk,asc,ascq: 0x6,*,*]
> MPATH_PR_SENSE_ABORTED_COMMAND if command fails with [sk,asc,ascq: 0xb,*,*]
> MPATH_PR_NO_SENSE if command fails with [sk,asc,ascq: 0x0,*,*]
> MPATH_PR_SENSE_MALFORMED if command fails with SCSI command malformed
> MPATH_PR_FILE_ERROR if command fails while accessing file (device node) problems(e.g. not found)
> MPATH_PR_DMMP_ERROR if Device Mapper related error.(e.g Error in getting dm info)
> MPATH_PR_OTHER if other error/warning has occurred(e.g transport or driver error)
>
>
> 5.1.4 int mpath_persistent_reserve_out ( int fd, int rq_servact, int rq_scope, unsigned int rq_type,
> struct rout_param_descriptor *paramp, int noisy, int verbose)
> DESCRIPTION
> The function in the mpath_persistent_reserve_out() sends PR OUT command to the DM device and gets the response.
>
> PARAMETERS:
> fd - The file descriptor of a multipath device. Input argument.
> rq_servact - PROUT command service action. Input argument
> rq_scope - Persistent reservation scope. The value should be always LU_SCOPE (0h).
> rq_type - Persistent reservation type. The valid values of persistent reservation types are
> 5h (Write exclusive - registrants only)
> 8h (Exclusive access - registrants only)
> 7h (Write exclusive - All registrants)
> 8h (Exclusive access - All registrants).
> paramp - PROUT command parameter data. The paramp is a struct which describes PROUT parameter list. Caller should
> manage the memory allocation of this structure.
> noisy - Turn on debugging trace: Input argument. 0->Disable, 1->Enable.
> verbose - Set verbosity level. Input argument. value: 0 to 3. 0->Crits and Errors, 1->Warnings, 2->Info, 3->Debug
>
> 6. Limitation and restriction
> -This feature doesn't support persistent reservation types of Write-Exclusive (1h)
> and Exclusive-Access(3h).
> - This feature will not support ALL_TG_PT bit on.
> - Current implementation is limited to FC SAS and ISCSI transport protocol. Other protocol support could be added in future.
>
>
> Regards,
> Vijay Chauhan
>
>
>
> git status for newly added or changed files in multipath-tools:
>
> # git diff -p --stat
> Makefile | 2 +
> Makefile.inc | 2 +
> libmultipath/config.c | 2 +
> libmultipath/config.h | 2 +
> libmultipath/configure.c | 1 +
> libmultipath/dict.c | 114 ++++++++++++++++++++++++++++++++++++
> libmultipath/discovery.c | 2 +-
> libmultipath/propsel.c | 46 +++++++++++++++
> libmultipath/propsel.h | 1 +
> libmultipath/structs.h | 4 +
> multipathd/Makefile | 4 +-
> multipathd/cli.c | 29 ++++++----
> multipathd/cli.h | 16 ++++--
> multipathd/cli_handlers.c | 72 +++++++++++++++++++++++
> multipathd/cli_handlers.h | 4 +
> multipathd/main.c | 140 +++++++++++++++++++++++++++++++++++++++++++++
> multipathd/main.h | 14 +++++
>
> # git status -s
> M Makefile
> M Makefile.inc
> A libmpathpersist/Makefile
> A libmpathpersist/mpath_persist.c
> A libmpathpersist/mpath_persist.h
> A libmpathpersist/mpath_persistent_reserve_in.3
> A libmpathpersist/mpath_persistent_reserve_out.3
> A libmpathpersist/mpath_pr_ioctl.c
> A libmpathpersist/mpath_pr_ioctl.h
> A libmpathpersist/mpath_updatepr.c
> A libmpathpersist/mpathpr.h
> M libmultipath/config.c
> M libmultipath/config.h
> M libmultipath/configure.c
> M libmultipath/dict.c
> M libmultipath/discovery.c
> M libmultipath/propsel.c
> M libmultipath/propsel.h
> M libmultipath/structs.h
> A mpathpersist/Makefile
> A mpathpersist/main.c
> A mpathpersist/main.h
> A mpathpersist/mpathpersist.8
> M multipathd/Makefile
> M multipathd/cli.c
> M multipathd/cli.h
> M multipathd/cli_handlers.c
> M multipathd/cli_handlers.h
> M multipathd/main.c
> M multipathd/main.h
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-24-2012, 03:09 AM
"Chauhan, Vijay"
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

Thanks for these info Fil. Can you please also share your feedback on your in-depth testing.

I appreciate your effort.

On January 20, 2012, Fil wrote:
>Thanks for the patches Vijay,
>
>I tested the new set of patches against the git repo, and it applied
>cleanly compiled and is running as expected. I will do more indepth
>testing over the weekend.
>
>thanks
>fil
>
>On 01/19/2012 02:20 PM, Chauhan, Vijay wrote:
>> Resending this patch with the fix mentioned by Mike
>Snitzer<snitzer@redhat.com> and Fil<lists@internyc.net>.
>> Thanks to all for sending me your review comments. Fil, Thanks for
>trying out mpathpersist and providing your
>> feedback. Also thanking Yanling Qi<Yanling.Qi@netapp.com> for
>proposing this design for pr management feature.
>> These series of patch is made on commit
>864162b08067d3e55d52996846d10a046956c6d6.


Regards,
Vijay

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-24-2012, 04:54 AM
Fil
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

Hi Vijay,

Sorry for a late report, but I was busy with other stuff and didn't test
as much as I wanted, but anyway here is what I have by now.

1. Patches apply against the 63704387009443bdb37d9deaaafa9ab121d45bfb
without any problems. Everything builds correctly (tested it on centos
6.2 and fedora 16).

2. multipathd runs correctly.

3. mpathpersist is missing --no-inquiry option. It would be nice to be
compatible with sg_persist. (breaks most of my test scripts.)

4. mpathpersist -d /dev/mapper/blah --in --read-status
mpathpersist: unrecognized option '--read-status'
unrecognised switch code 0x3f ??

5. registration works

6. reservation, reserves only a single path. (is this by design?)
mpathpersist --out --reserve --param-rk=123abc --prout-type=8 -d
/dev/mapper/blah

sg_persist -i -s /dev/mapper/blah
QNAP iSCSI Storage 3.1
Peripheral device type: disk
PR generation=0x8
Key=0x123abc
All target ports bit clear
Relative port address: 0x1
not reservation holder
Transport Id of initiator:
iSCSI name and session id: iqn.2009-11.com.adriaticsolutions:blah
Key=0x123abc
All target ports bit clear
Relative port address: 0x1
<< Reservation holder >>
scope: LU_SCOPE, type: Exclusive Access, all registrants
Transport Id of initiator:
iSCSI name and session id: iqn.2009-11.com.adriaticsolutions:blah

multipath -ll
blah (36001405c55fc03cd8193d491eda0d4d7) dm-4 QNAP,iSCSI Storage
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 16:0:0:0 sda 8:0 active ready running
`- 17:0:0:0 sdb 8:16 failed faulty running

7. mpathpersist -d /dev/mapper/blah --in --read-reservation
Persistent Reserve IN command failed

8. release 'fails' because reservation registered only a single path and
the other one is in a failed state.

mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
--prout-type=8
Jan 24 00:43:53 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
reservation command failed.
PR out: command failed

9. de-registration works....

I haven't had time to test path fail over and weird cases when multiple
nodes try to do destructive things to the shared luns...

thanks
fil

On 01/23/2012 11:09 PM, Chauhan, Vijay wrote:
> Thanks for these info Fil. Can you please also share your feedback on your in-depth testing.
>
> I appreciate your effort.
>
> On January 20, 2012, Fil wrote:
>> Thanks for the patches Vijay,
>>
>> I tested the new set of patches against the git repo, and it applied
>> cleanly compiled and is running as expected. I will do more indepth
>> testing over the weekend.
>>
>> thanks
>> fil
>>
>> On 01/19/2012 02:20 PM, Chauhan, Vijay wrote:
>>> Resending this patch with the fix mentioned by Mike
>> Snitzer<snitzer@redhat.com> and Fil<lists@internyc.net>.
>>> Thanks to all for sending me your review comments. Fil, Thanks for
>> trying out mpathpersist and providing your
>>> feedback. Also thanking Yanling Qi<Yanling.Qi@netapp.com> for
>> proposing this design for pr management feature.
>>> These series of patch is made on commit
>> 864162b08067d3e55d52996846d10a046956c6d6.
>
>
> Regards,
> Vijay
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-25-2012, 06:45 AM
"Chauhan, Vijay"
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

On January 24, 2012, Fil Wrote:
>
>1. Patches apply against the 63704387009443bdb37d9deaaafa9ab121d45bfb
>without any problems. Everything builds correctly (tested it on centos
>6.2 and fedora 16).
>
>2. multipathd runs correctly.
>

Fil, Thanks for the update. Good to know.

>3. mpathpersist is missing --no-inquiry option. It would be nice to be
>compatible with sg_persist. (breaks most of my test scripts.)
>
>4. mpathpersist -d /dev/mapper/blah --in --read-status
>mpathpersist: unrecognized option '--read-status'
>unrecognised switch code 0x3f ??
>

There is a typo in usage output. Please use --read-full-status instead. I will correct it.

>5. registration works
>
>6. reservation, reserves only a single path. (is this by design?)

Yes, this is as per design. Reservation is sent to one of the active
path and reservation is applicable to registered I_T nexus with respect to pr type.

I was expecting your device server to apply reservation to all the
registered I_T nexus for pr type "Exclusive Access, all registrants". Can you verify the below output again?

Please see below in SPC4
"For a persistent reservation of the type Write Exclusive – All Registrants or Exclusive Access – All Registrants,
the persistent reservation holder is any registered I_T nexus;"

>mpathpersist --out --reserve --param-rk=123abc --prout-type=8 -d
>/dev/mapper/blah
>
>sg_persist -i -s /dev/mapper/blah
> QNAP iSCSI Storage 3.1
> Peripheral device type: disk
> PR generation=0x8
> Key=0x123abc
> All target ports bit clear
> Relative port address: 0x1
> not reservation holder
> Transport Id of initiator:
> iSCSI name and session id: iqn.2009-
>11.com.adriaticsolutions:blah
> Key=0x123abc
> All target ports bit clear
> Relative port address: 0x1
> << Reservation holder >>
> scope: LU_SCOPE, type: Exclusive Access, all registrants
> Transport Id of initiator:
> iSCSI name and session id: iqn.2009-
>11.com.adriaticsolutions:blah
>
>multipath -ll
>blah (36001405c55fc03cd8193d491eda0d4d7) dm-4 QNAP,iSCSI Storage
>size=10G features='0' hwhandler='0' wp=rw
>`-+- policy='round-robin 0' prio=1 status=active
> |- 16:0:0:0 sda 8:0 active ready running
> `- 17:0:0:0 sdb 8:16 failed faulty running
>
>7. mpathpersist -d /dev/mapper/blah --in --read-reservation
>Persistent Reserve IN command failed
>
Can you please share the output with verbose 3. Append '-v3'.

>8. release 'fails' because reservation registered only a single path and
>the other one is in a failed state.
>
>mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>--prout-type=8
>Jan 24 00:43:53 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
>reservation command failed.
>PR out: command failed
>
As per design, release service action performs following steps:
Step 1) PROUT 'release' SA is sent on the all paths of the multipath device as it is not clear which data path is reservation holder.
STEP 2) PRIN read reservation SA is sent to one of the active path to verify the reservation.
STEP 3) If the Read Reservation parameter data indicates that the logical unit is still reserved then this indicates that the reservation holder belongs to a data path in failed state or removed data path of the multipath device, continue else goto step 7
STEP 4) PRIN Report Full Status service action is issued to any active data path. Full status descriptors are saved.
STEP 5) PROUT clear reservation service action is sent via any active data path to clear the reservation and
Registrants
STEP 6) all registrants are restored by issuing PROUT register service action with transport IDs from the full status
descriptors saved in the step 4
STEP 7) exit with status

It looks like you are getting into some other issue. Can you please share the output with verbose with '-v3'.

>9. de-registration works....
>
>I haven't had time to test path fail over and weird cases when multiple
>nodes try to do destructive things to the shared luns...
>
>thanks
>fil
>
>On 01/23/2012 11:09 PM, Chauhan, Vijay wrote:
>> Thanks for these info Fil. Can you please also share your feedback on
>your in-depth testing.
>>
>> I appreciate your effort.
>>
>> On January 20, 2012, Fil wrote:
>>> Thanks for the patches Vijay,
>>>
>>> I tested the new set of patches against the git repo, and it applied
>>> cleanly compiled and is running as expected. I will do more indepth
>>> testing over the weekend.

Thanks for your detailed testing.

Thanks and regards,
Vijay

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-25-2012, 02:42 PM
Fil
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

Here is the info you requested.

thanks
fil

On 01/25/2012 02:45 AM, Chauhan, Vijay wrote:
> On January 24, 2012, Fil Wrote:
>>
>> 1. Patches apply against the 63704387009443bdb37d9deaaafa9ab121d45bfb
>> without any problems. Everything builds correctly (tested it on centos
>> 6.2 and fedora 16).
>>
>> 2. multipathd runs correctly.
>>
>
> Fil, Thanks for the update. Good to know.
>
>> 3. mpathpersist is missing --no-inquiry option. It would be nice to be
>> compatible with sg_persist. (breaks most of my test scripts.)
>>
>> 4. mpathpersist -d /dev/mapper/blah --in --read-status
>> mpathpersist: unrecognized option '--read-status'
>> unrecognised switch code 0x3f ??
>>
>
> There is a typo in usage output. Please use --read-full-status instead. I will correct it.

still no go...

mpathpersist -d /dev/mapper/blah --in --read-full-status
Persistent Reserve IN command failed
mpathpersist -d /dev/mapper/blah --in --read-full-status -v3
Jan 25 10:32:37 | alias = blah
Jan 25 10:32:37 | ram0: device node name blacklisted
Jan 25 10:32:37 | ram1: device node name blacklisted
Jan 25 10:32:37 | ram2: device node name blacklisted
Jan 25 10:32:37 | ram3: device node name blacklisted
Jan 25 10:32:37 | ram4: device node name blacklisted
Jan 25 10:32:37 | ram5: device node name blacklisted
Jan 25 10:32:37 | ram6: device node name blacklisted
Jan 25 10:32:37 | ram7: device node name blacklisted
Jan 25 10:32:37 | ram8: device node name blacklisted
Jan 25 10:32:37 | ram9: device node name blacklisted
Jan 25 10:32:37 | ram10: device node name blacklisted
Jan 25 10:32:37 | ram11: device node name blacklisted
Jan 25 10:32:37 | ram12: device node name blacklisted
Jan 25 10:32:37 | ram13: device node name blacklisted
Jan 25 10:32:37 | ram14: device node name blacklisted
Jan 25 10:32:37 | ram15: device node name blacklisted
Jan 25 10:32:37 | loop0: device node name blacklisted
Jan 25 10:32:37 | loop1: device node name blacklisted
Jan 25 10:32:37 | loop2: device node name blacklisted
Jan 25 10:32:37 | loop3: device node name blacklisted
Jan 25 10:32:37 | loop4: device node name blacklisted
Jan 25 10:32:37 | loop5: device node name blacklisted
Jan 25 10:32:37 | loop6: device node name blacklisted
Jan 25 10:32:37 | loop7: device node name blacklisted
Jan 25 10:32:37 | vda: device node name blacklisted
Jan 25 10:32:37 | dm-0: device node name blacklisted
Jan 25 10:32:37 | dm-1: device node name blacklisted
Jan 25 10:32:37 | dm-2: device node name blacklisted
Jan 25 10:32:37 | dm-3: device node name blacklisted
Jan 25 10:32:37 | sda: not found in pathvec
Jan 25 10:32:37 | sda: mask = 0x5
Jan 25 10:32:37 | sda: dev_t = 8:0
Jan 25 10:32:37 | sda: size = 20971520
Jan 25 10:32:37 | sda: vendor = QNAP
Jan 25 10:32:37 | sda: product = iSCSI Storage
Jan 25 10:32:37 | sda: rev = 3.1
Jan 25 10:32:37 | sda: h:b:t:l = 16:0:0:0
Jan 25 10:32:37 | sda: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:32:37 | sda: path state = running
Jan 25 10:32:37 | sda: get_state
Jan 25 10:32:37 | loading /lib64/multipath/libcheckdirectio.so checker
Jan 25 10:32:37 | sda: path checker = directio (controller setting)
Jan 25 10:32:37 | sda: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:32:37 | directio: starting new request
Jan 25 10:32:37 | directio: io finished 4096/0
Jan 25 10:32:37 | sda: state = up
Jan 25 10:32:37 | dm-4: device node name blacklisted
Jan 25 10:32:37 | sdb: not found in pathvec
Jan 25 10:32:37 | sdb: mask = 0x5
Jan 25 10:32:37 | sdb: dev_t = 8:16
Jan 25 10:32:37 | sdb: size = 20971520
Jan 25 10:32:37 | sdb: vendor = QNAP
Jan 25 10:32:37 | sdb: product = iSCSI Storage
Jan 25 10:32:37 | sdb: rev = 3.1
Jan 25 10:32:37 | sdb: h:b:t:l = 17:0:0:0
Jan 25 10:32:37 | sdb: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:32:37 | sdb: path state = running
Jan 25 10:32:37 | sdb: get_state
Jan 25 10:32:37 | sdb: path checker = directio (controller setting)
Jan 25 10:32:37 | sdb: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:32:37 | directio: starting new request
Jan 25 10:32:37 | directio: io finished 18446744073709551611/0
Jan 25 10:32:37 | sdb: state = down
Jan 25 10:32:37 | sdb: checker msg is "directio checker reports path is
down"
Jan 25 10:32:37 | params = 0 0 1 1 round-robin 0 2 1 8:0 1 8:16 1
Jan 25 10:32:37 | status = 2 0 0 0 1 1 A 0 2 0 8:0 A 0 8:16 F 2
Jan 25 10:32:37 | blah: disassemble map [0 0 1 1 round-robin 0 2 1 8:0 1
8:16 1 ]
Jan 25 10:32:37 | sda: mask = 0x8
Jan 25 10:32:37 | sda: path state = running
Jan 25 10:32:37 | loading /lib64/multipath/libprioconst.so prioritizer
Jan 25 10:32:37 | sda: prio = const (internal default)
Jan 25 10:32:37 | sda: prio = (internal default)
Jan 25 10:32:37 | sda: const prio = 1
Jan 25 10:32:37 | sdb: mask = 0x8
Jan 25 10:32:37 | sdb: path state = running
Jan 25 10:32:37 | sdb: prio = const (internal default)
Jan 25 10:32:37 | sdb: prio = (internal default)
Jan 25 10:32:37 | sdb: const prio = 1
Jan 25 10:32:37 | blah: disassemble status [2 0 0 0 1 1 A 0 2 0 8:0 A 0
8:16 F 2 ]
Jan 25 10:32:37 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
command to sda
Jan 25 10:32:37 | sda: duration = 2 (ms)
Jan 25 10:32:37 | sda: status driver:00 host:04 scsi:00
Jan 25 10:32:37 | directio checker refcount 2
Jan 25 10:32:37 | directio checker refcount 1
Persistent Reserve IN command failed
Jan 25 10:32:37 | unloading const prioritizer
Jan 25 10:32:37 | unloading directio checker

>
>> 5. registration works
>>
>> 6. reservation, reserves only a single path. (is this by design?)
>
> Yes, this is as per design. Reservation is sent to one of the active
> path and reservation is applicable to registered I_T nexus with respect to pr type.
>
> I was expecting your device server to apply reservation to all the
> registered I_T nexus for pr type "Exclusive Access, all registrants". Can you verify the below output again?
>
> Please see below in SPC4
> "For a persistent reservation of the type Write Exclusive – All Registrants or Exclusive Access – All Registrants,
> the persistent reservation holder is any registered I_T nexus;"
>
>> mpathpersist --out --reserve --param-rk=123abc --prout-type=8 -d
>> /dev/mapper/blah
>>
>> sg_persist -i -s /dev/mapper/blah
>> QNAP iSCSI Storage 3.1
>> Peripheral device type: disk
>> PR generation=0x8
>> Key=0x123abc
>> All target ports bit clear
>> Relative port address: 0x1
>> not reservation holder
>> Transport Id of initiator:
>> iSCSI name and session id: iqn.2009-
>> 11.com.adriaticsolutions:blah
>> Key=0x123abc
>> All target ports bit clear
>> Relative port address: 0x1
>> << Reservation holder >>
>> scope: LU_SCOPE, type: Exclusive Access, all registrants
>> Transport Id of initiator:
>> iSCSI name and session id: iqn.2009-
>> 11.com.adriaticsolutions:blah
>>
>> multipath -ll
>> blah (36001405c55fc03cd8193d491eda0d4d7) dm-4 QNAP,iSCSI Storage
>> size=10G features='0' hwhandler='0' wp=rw
>> `-+- policy='round-robin 0' prio=1 status=active
>> |- 16:0:0:0 sda 8:0 active ready running
>> `- 17:0:0:0 sdb 8:16 failed faulty running
>>
>> 7. mpathpersist -d /dev/mapper/blah --in --read-reservation
>> Persistent Reserve IN command failed
>>
> Can you please share the output with verbose 3. Append '-v3'.


mpathpersist -d /dev/mapper/blah --in --read-reservation
Persistent Reserve IN command failed
mpathpersist -d /dev/mapper/blah --in --read-reservation -v3
Jan 25 10:34:07 | alias = blah
Jan 25 10:34:07 | ram0: device node name blacklisted
Jan 25 10:34:07 | ram1: device node name blacklisted
Jan 25 10:34:07 | ram2: device node name blacklisted
Jan 25 10:34:07 | ram3: device node name blacklisted
Jan 25 10:34:07 | ram4: device node name blacklisted
Jan 25 10:34:07 | ram5: device node name blacklisted
Jan 25 10:34:07 | ram6: device node name blacklisted
Jan 25 10:34:07 | ram7: device node name blacklisted
Jan 25 10:34:07 | ram8: device node name blacklisted
Jan 25 10:34:07 | ram9: device node name blacklisted
Jan 25 10:34:07 | ram10: device node name blacklisted
Jan 25 10:34:07 | ram11: device node name blacklisted
Jan 25 10:34:07 | ram12: device node name blacklisted
Jan 25 10:34:07 | ram13: device node name blacklisted
Jan 25 10:34:07 | ram14: device node name blacklisted
Jan 25 10:34:07 | ram15: device node name blacklisted
Jan 25 10:34:07 | loop0: device node name blacklisted
Jan 25 10:34:07 | loop1: device node name blacklisted
Jan 25 10:34:07 | loop2: device node name blacklisted
Jan 25 10:34:07 | loop3: device node name blacklisted
Jan 25 10:34:07 | loop4: device node name blacklisted
Jan 25 10:34:07 | loop5: device node name blacklisted
Jan 25 10:34:07 | loop6: device node name blacklisted
Jan 25 10:34:07 | loop7: device node name blacklisted
Jan 25 10:34:07 | vda: device node name blacklisted
Jan 25 10:34:07 | dm-0: device node name blacklisted
Jan 25 10:34:07 | dm-1: device node name blacklisted
Jan 25 10:34:07 | dm-2: device node name blacklisted
Jan 25 10:34:07 | dm-3: device node name blacklisted
Jan 25 10:34:07 | sda: not found in pathvec
Jan 25 10:34:07 | sda: mask = 0x5
Jan 25 10:34:07 | sda: dev_t = 8:0
Jan 25 10:34:07 | sda: size = 20971520
Jan 25 10:34:07 | sda: vendor = QNAP
Jan 25 10:34:07 | sda: product = iSCSI Storage
Jan 25 10:34:07 | sda: rev = 3.1
Jan 25 10:34:07 | sda: h:b:t:l = 16:0:0:0
Jan 25 10:34:07 | sda: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:34:07 | sda: path state = running
Jan 25 10:34:07 | sda: get_state
Jan 25 10:34:07 | loading /lib64/multipath/libcheckdirectio.so checker
Jan 25 10:34:07 | sda: path checker = directio (controller setting)
Jan 25 10:34:07 | sda: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:34:07 | directio: starting new request
Jan 25 10:34:07 | directio: io finished 4096/0
Jan 25 10:34:07 | sda: state = up
Jan 25 10:34:07 | dm-4: device node name blacklisted
Jan 25 10:34:07 | sdb: not found in pathvec
Jan 25 10:34:07 | sdb: mask = 0x5
Jan 25 10:34:07 | sdb: dev_t = 8:16
Jan 25 10:34:07 | sdb: size = 20971520
Jan 25 10:34:07 | sdb: vendor = QNAP
Jan 25 10:34:07 | sdb: product = iSCSI Storage
Jan 25 10:34:07 | sdb: rev = 3.1
Jan 25 10:34:07 | sdb: h:b:t:l = 17:0:0:0
Jan 25 10:34:07 | sdb: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:34:07 | sdb: path state = running
Jan 25 10:34:07 | sdb: get_state
Jan 25 10:34:07 | sdb: path checker = directio (controller setting)
Jan 25 10:34:07 | sdb: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:34:07 | directio: starting new request
Jan 25 10:34:07 | directio: io finished 18446744073709551611/0
Jan 25 10:34:07 | sdb: state = down
Jan 25 10:34:07 | sdb: checker msg is "directio checker reports path is
down"
Jan 25 10:34:07 | params = 0 0 1 1 round-robin 0 2 1 8:0 1 8:16 1
Jan 25 10:34:07 | status = 2 0 0 0 1 1 A 0 2 0 8:0 A 0 8:16 F 2
Jan 25 10:34:07 | blah: disassemble map [0 0 1 1 round-robin 0 2 1 8:0 1
8:16 1 ]
Jan 25 10:34:07 | sda: mask = 0x8
Jan 25 10:34:07 | sda: path state = running
Jan 25 10:34:07 | loading /lib64/multipath/libprioconst.so prioritizer
Jan 25 10:34:07 | sda: prio = const (internal default)
Jan 25 10:34:07 | sda: prio = (internal default)
Jan 25 10:34:07 | sda: const prio = 1
Jan 25 10:34:07 | sdb: mask = 0x8
Jan 25 10:34:07 | sdb: path state = running
Jan 25 10:34:07 | sdb: prio = const (internal default)
Jan 25 10:34:07 | sdb: prio = (internal default)
Jan 25 10:34:07 | sdb: const prio = 1
Jan 25 10:34:07 | blah: disassemble status [2 0 0 0 1 1 A 0 2 0 8:0 A 0
8:16 F 2 ]
Jan 25 10:34:07 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
command to sda
Jan 25 10:34:07 | sda: duration = 1 (ms)
Jan 25 10:34:07 | sda: status driver:00 host:04 scsi:00
Jan 25 10:34:07 | directio checker refcount 2
Jan 25 10:34:07 | directio checker refcount 1
Persistent Reserve IN command failed
Jan 25 10:34:07 | unloading const prioritizer
Jan 25 10:34:07 | unloading directio checker

>
>> 8. release 'fails' because reservation registered only a single path and
>> the other one is in a failed state.
>>
>> mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>> --prout-type=8
>> Jan 24 00:43:53 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
>> reservation command failed.
>> PR out: command failed
>>
> As per design, release service action performs following steps:
> Step 1) PROUT 'release' SA is sent on the all paths of the multipath device as it is not clear which data path is reservation holder.
> STEP 2) PRIN read reservation SA is sent to one of the active path to verify the reservation.
> STEP 3) If the Read Reservation parameter data indicates that the logical unit is still reserved then this indicates that the reservation holder belongs to a data path in failed state or removed data path of the multipath device, continue else goto step 7
> STEP 4) PRIN Report Full Status service action is issued to any active data path. Full status descriptors are saved.
> STEP 5) PROUT clear reservation service action is sent via any active data path to clear the reservation and
> Registrants
> STEP 6) all registrants are restored by issuing PROUT register service action with transport IDs from the full status
> descriptors saved in the step 4
> STEP 7) exit with status
>
> It looks like you are getting into some other issue. Can you please share the output with verbose with '-v3'.


mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
--prout-type=8 -v3
Jan 25 10:35:41 | alias = blah
Jan 25 10:35:41 | ram0: device node name blacklisted
Jan 25 10:35:41 | ram1: device node name blacklisted
Jan 25 10:35:41 | ram2: device node name blacklisted
Jan 25 10:35:41 | ram3: device node name blacklisted
Jan 25 10:35:41 | ram4: device node name blacklisted
Jan 25 10:35:41 | ram5: device node name blacklisted
Jan 25 10:35:41 | ram6: device node name blacklisted
Jan 25 10:35:41 | ram7: device node name blacklisted
Jan 25 10:35:41 | ram8: device node name blacklisted
Jan 25 10:35:41 | ram9: device node name blacklisted
Jan 25 10:35:41 | ram10: device node name blacklisted
Jan 25 10:35:41 | ram11: device node name blacklisted
Jan 25 10:35:41 | ram12: device node name blacklisted
Jan 25 10:35:41 | ram13: device node name blacklisted
Jan 25 10:35:41 | ram14: device node name blacklisted
Jan 25 10:35:41 | ram15: device node name blacklisted
Jan 25 10:35:41 | loop0: device node name blacklisted
Jan 25 10:35:41 | loop1: device node name blacklisted
Jan 25 10:35:41 | loop2: device node name blacklisted
Jan 25 10:35:41 | loop3: device node name blacklisted
Jan 25 10:35:41 | loop4: device node name blacklisted
Jan 25 10:35:41 | loop5: device node name blacklisted
Jan 25 10:35:41 | loop6: device node name blacklisted
Jan 25 10:35:41 | loop7: device node name blacklisted
Jan 25 10:35:41 | vda: device node name blacklisted
Jan 25 10:35:41 | dm-0: device node name blacklisted
Jan 25 10:35:41 | dm-1: device node name blacklisted
Jan 25 10:35:41 | dm-2: device node name blacklisted
Jan 25 10:35:41 | dm-3: device node name blacklisted
Jan 25 10:35:41 | sda: not found in pathvec
Jan 25 10:35:41 | sda: mask = 0x5
Jan 25 10:35:41 | sda: dev_t = 8:0
Jan 25 10:35:41 | sda: size = 20971520
Jan 25 10:35:41 | sda: vendor = QNAP
Jan 25 10:35:41 | sda: product = iSCSI Storage
Jan 25 10:35:41 | sda: rev = 3.1
Jan 25 10:35:41 | sda: h:b:t:l = 16:0:0:0
Jan 25 10:35:41 | sda: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:35:41 | sda: path state = running
Jan 25 10:35:41 | sda: get_state
Jan 25 10:35:41 | loading /lib64/multipath/libcheckdirectio.so checker
Jan 25 10:35:41 | sda: path checker = directio (controller setting)
Jan 25 10:35:41 | sda: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:35:41 | directio: starting new request
Jan 25 10:35:41 | directio: io finished 4096/0
Jan 25 10:35:41 | sda: state = up
Jan 25 10:35:41 | dm-4: device node name blacklisted
Jan 25 10:35:41 | sdb: not found in pathvec
Jan 25 10:35:41 | sdb: mask = 0x5
Jan 25 10:35:41 | sdb: dev_t = 8:16
Jan 25 10:35:41 | sdb: size = 20971520
Jan 25 10:35:41 | sdb: vendor = QNAP
Jan 25 10:35:41 | sdb: product = iSCSI Storage
Jan 25 10:35:41 | sdb: rev = 3.1
Jan 25 10:35:41 | sdb: h:b:t:l = 17:0:0:0
Jan 25 10:35:41 | sdb: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:35:41 | sdb: path state = running
Jan 25 10:35:41 | sdb: get_state
Jan 25 10:35:41 | sdb: path checker = directio (controller setting)
Jan 25 10:35:41 | sdb: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:35:41 | directio: starting new request
Jan 25 10:35:41 | directio: io finished 18446744073709551611/0
Jan 25 10:35:41 | sdb: state = down
Jan 25 10:35:41 | sdb: checker msg is "directio checker reports path is
down"
Jan 25 10:35:41 | params = 0 0 1 1 round-robin 0 2 1 8:0 1 8:16 1
Jan 25 10:35:41 | status = 2 0 0 0 1 1 A 0 2 0 8:0 A 0 8:16 F 2
Jan 25 10:35:41 | blah: disassemble map [0 0 1 1 round-robin 0 2 1 8:0 1
8:16 1 ]
Jan 25 10:35:41 | sda: mask = 0x8
Jan 25 10:35:41 | sda: path state = running
Jan 25 10:35:41 | loading /lib64/multipath/libprioconst.so prioritizer
Jan 25 10:35:41 | sda: prio = const (internal default)
Jan 25 10:35:41 | sda: prio = (internal default)
Jan 25 10:35:41 | sda: const prio = 1
Jan 25 10:35:41 | sdb: mask = 0x8
Jan 25 10:35:41 | sdb: path state = running
Jan 25 10:35:41 | sdb: prio = const (internal default)
Jan 25 10:35:41 | sdb: prio = (internal default)
Jan 25 10:35:41 | sdb: const prio = 1
Jan 25 10:35:41 | blah: disassemble status [2 0 0 0 1 1 A 0 2 0 8:0 A 0
8:16 F 2 ]
Jan 25 10:35:41 | blah: reservation_key = 0x123abc (config file default)
Jan 25 10:35:41 | path count = 0
Jan 25 10:35:41 | rq_servact=2
Jan 25 10:35:41 | rq_scope=0
Jan 25 10:35:41 | rq_type=8
Jan 25 10:35:41 | noisy=1
Jan 25 10:35:41 | status=-1
Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr out
command to sda
Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sdb path not up.
Jan 25 10:35:41 | sda: rq_servact = 2
Jan 25 10:35:41 | sda: rq_scope = 0
Jan 25 10:35:41 | sda: rq_type = 8
Jan 25 10:35:41 | sda: paramlen = 24
Jan 25 10:35:41 | sda: Persistent Reservation OUT parameter:
Jan 25 10:35:41 | 00 00 00 00 00 12 3a bc 00 00 00 00 00 00 00
00
Jan 25 10:35:41 | 00 00 00 00 00 00 00 00
Jan 25 10:35:41 | sda: Duration=1 (ms)
Jan 25 10:35:41 | sda: status driver:00 host:00 scsi:00
Jan 25 10:35:41 | sda: status = 0
Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
command to sda
Jan 25 10:35:41 | sda: duration = 1 (ms)
Jan 25 10:35:41 | sda: status driver:08 host:00 scsi:02
Jan 25 10:35:41 | sda: Sense_Key=06, ASC=00 ASCQ=00
Jan 25 10:35:41 | sda: retrying for Unit Attention. Remaining retries = 4
Jan 25 10:35:41 | sda: duration = 0 (ms)
Jan 25 10:35:41 | sda: status driver:00 host:04 scsi:00
Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
reservation command failed.
Jan 25 10:35:41 | directio checker refcount 2
Jan 25 10:35:41 | directio checker refcount 1
PR out: command failed
Jan 25 10:35:41 | unloading const prioritizer
Jan 25 10:35:41 | unloading directio checker

>
>> 9. de-registration works....
>>
>> I haven't had time to test path fail over and weird cases when multiple
>> nodes try to do destructive things to the shared luns...
>>
>> thanks
>> fil
>>
>> On 01/23/2012 11:09 PM, Chauhan, Vijay wrote:
>>> Thanks for these info Fil. Can you please also share your feedback on
>> your in-depth testing.
>>>
>>> I appreciate your effort.
>>>
>>> On January 20, 2012, Fil wrote:
>>>> Thanks for the patches Vijay,
>>>>
>>>> I tested the new set of patches against the git repo, and it applied
>>>> cleanly compiled and is running as expected. I will do more indepth
>>>> testing over the weekend.
>
> Thanks for your detailed testing.
>
> Thanks and regards,
> Vijay
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-25-2012, 06:28 PM
"Chauhan, Vijay"
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

On Wednesday, January 25, 2012 9:12 PM, Fil Wrote:
>Here is the info you requested.

Fil, Thanks for the logs.

>On 01/25/2012 02:45 AM, Chauhan, Vijay wrote:
>> On January 24, 2012, Fil Wrote:
>>>
>>> 1. Patches apply against the 63704387009443bdb37d9deaaafa9ab121d45bfb
>>> without any problems. Everything builds correctly (tested it on
>centos
>>> 6.2 and fedora 16).
>>>
>>> 2. multipathd runs correctly.
>>>
>>
>> Fil, Thanks for the update. Good to know.
>>
>>> 3. mpathpersist is missing --no-inquiry option. It would be nice to
>be
>>> compatible with sg_persist. (breaks most of my test scripts.)
>>>
>>> 4. mpathpersist -d /dev/mapper/blah --in --read-status
>>> mpathpersist: unrecognized option '--read-status'
>>> unrecognised switch code 0x3f ??
>>>
>>
>> There is a typo in usage output. Please use --read-full-status
>instead. I will correct it.
>
>still no go...
>

It seems issue 4 is resolved with option '--read-full-status'

>mpathpersist -d /dev/mapper/blah --in --read-full-status
>Persistent Reserve IN command failed
>mpathpersist -d /dev/mapper/blah --in --read-full-status -v3

>Jan 25 10:32:37 | alias = blah
>Jan 25 10:32:37 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>command to sda
>Jan 25 10:32:37 | sda: duration = 2 (ms)
>Jan 25 10:32:37 | sda: status driver:00 host:04 scsi:00

PRIN command returned with DID_BAD_TARGET host status. iscsi layer returns this when we get some bad output from
target. This does not look like issue related to this feature.

>>
>>> 5. registration works
>>>
>>> 6. reservation, reserves only a single path. (is this by design?)
>>
>> Yes, this is as per design. Reservation is sent to one of the active
>> path and reservation is applicable to registered I_T nexus with
>respect to pr type.
>>
>> I was expecting your device server to apply reservation to all the
>> registered I_T nexus for pr type "Exclusive Access, all registrants".
>Can you verify the below output again?
>>
>> Please see below in SPC4
>> "For a persistent reservation of the type Write Exclusive – All
>Registrants or Exclusive Access – All Registrants,
>> the persistent reservation holder is any registered I_T nexus;"
>>
>>> mpathpersist --out --reserve --param-rk=123abc --prout-type=8 -d
>>> /dev/mapper/blah
>>>
>>> sg_persist -i -s /dev/mapper/blah
>>> QNAP iSCSI Storage 3.1
>>> Peripheral device type: disk
>>> PR generation=0x8
>>> Key=0x123abc
>>> All target ports bit clear
>>> Relative port address: 0x1
>>> not reservation holder
>>> Transport Id of initiator:
>>> iSCSI name and session id: iqn.2009-
>>> 11.com.adriaticsolutions:blah
>>> Key=0x123abc
>>> All target ports bit clear
>>> Relative port address: 0x1
>>> << Reservation holder >>
>>> scope: LU_SCOPE, type: Exclusive Access, all registrants
>>> Transport Id of initiator:
>>> iSCSI name and session id: iqn.2009-
>>> 11.com.adriaticsolutions:blah
>>>
>>> multipath -ll
>>> blah (36001405c55fc03cd8193d491eda0d4d7) dm-4 QNAP,iSCSI Storage
>>> size=10G features='0' hwhandler='0' wp=rw
>>> `-+- policy='round-robin 0' prio=1 status=active
>>> |- 16:0:0:0 sda 8:0 active ready running
>>> `- 17:0:0:0 sdb 8:16 failed faulty running
>>>
>>> 7. mpathpersist -d /dev/mapper/blah --in --read-reservation
>>> Persistent Reserve IN command failed
>>>
>> Can you please share the output with verbose 3. Append '-v3'.
>
>
>mpathpersist -d /dev/mapper/blah --in --read-reservation
>Persistent Reserve IN command failed
>mpathpersist -d /dev/mapper/blah --in --read-reservation -v3
>Jan 25 10:34:07 | alias = blah

>Jan 25 10:34:07 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>command to sda
>Jan 25 10:34:07 | sda: duration = 1 (ms)
>Jan 25 10:34:07 | sda: status driver:00 host:04 scsi:00

Same is with issue 7. PRIN command returned with DID_BAD_TARGET host status.

>Jan 25 10:34:07 | directio checker refcount 2
>Jan 25 10:34:07 | directio checker refcount 1
>Persistent Reserve IN command failed
>Jan 25 10:34:07 | unloading const prioritizer
>Jan 25 10:34:07 | unloading directio checker
>
>>
>>> 8. release 'fails' because reservation registered only a single path
>and
>>> the other one is in a failed state.
>>>
>>> mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>>> --prout-type=8
>>> Jan 24 00:43:53 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
>>> reservation command failed.
>>> PR out: command failed
>>>
>> As per design, release service action performs following steps:
>> Step 1) PROUT 'release' SA is sent on the all paths of the multipath
>device as it is not clear which data path is reservation holder.
>> STEP 2) PRIN read reservation SA is sent to one of the active path to
>verify the reservation.
>> STEP 3) If the Read Reservation parameter data indicates that the
>logical unit is still reserved then this indicates that the reservation
>holder belongs to a data path in failed state or removed data path of
>the multipath device, continue else goto step 7
>> STEP 4) PRIN Report Full Status service action is issued to any
>active data path. Full status descriptors are saved.
>> STEP 5) PROUT clear reservation service action is sent via any active
>data path to clear the reservation and
>> Registrants
>> STEP 6) all registrants are restored by issuing PROUT register service
>action with transport IDs from the full status
>> descriptors saved in the step 4
>> STEP 7) exit with status
>>
>> It looks like you are getting into some other issue. Can you please
>share the output with verbose with '-v3'.
>
>
>mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>--prout-type=8 -v3
>Jan 25 10:35:41 | alias = blah

>Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr out
>command to sda
>Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sdb path not up.
>Jan 25 10:35:41 | sda: rq_servact = 2
>Jan 25 10:35:41 | sda: rq_scope = 0
>Jan 25 10:35:41 | sda: rq_type = 8
>Jan 25 10:35:41 | sda: paramlen = 24
>Jan 25 10:35:41 | sda: Persistent Reservation OUT parameter:
>Jan 25 10:35:41 | 00 00 00 00 00 12 3a bc 00 00 00 00 00 00 00
>00
>Jan 25 10:35:41 | 00 00 00 00 00 00 00 00
>Jan 25 10:35:41 | sda: Duration=1 (ms)
>Jan 25 10:35:41 | sda: status driver:00 host:00 scsi:00
>Jan 25 10:35:41 | sda: status = 0
>Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>command to sda
>Jan 25 10:35:41 | sda: duration = 1 (ms)
>Jan 25 10:35:41 | sda: status driver:08 host:00 scsi:02
>Jan 25 10:35:41 | sda: Sense_Key=06, ASC=00 ASCQ=00
>Jan 25 10:35:41 | sda: retrying for Unit Attention. Remaining retries =
>4
>Jan 25 10:35:41 | sda: duration = 0 (ms)
>Jan 25 10:35:41 | sda: status driver:00 host:04 scsi:00

Same with issue 8. PRIN command returned with DID_BAD_TARGET host status.

All the three failures reported above are due to DID_BAD_TARGET host status returned for
the PRIN command. It does not look like anywhere this feature is related to the above reported errors.

>>> 9. de-registration works....
>>>

Thanks,
Vijay

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-25-2012, 07:55 PM
Fil
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

Vijay,

I guess you are not understanding what I am saying.

1. mpathpersist does not work as expected

here is an output of mpathpersist vs sg_persist

[root@rhc-node3 ~]# mpathpersist -d /dev/mapper/blah --in --read-full-status
Persistent Reserve IN command failed
[root@rhc-node3 ~]# sg_persist -d /dev/mapper/blah --in --read-full-status
QNAP iSCSI Storage 3.1
Peripheral device type: disk
PR generation=0xc
Key=0x123abc
All target ports bit clear
Relative port address: 0x1
not reservation holder
Transport Id of initiator:
iSCSI name and session id: iqn.2009-11.com.adriaticsolutions:blah
Key=0x123abc
All target ports bit clear
Relative port address: 0x1
<< Reservation holder >>
scope: LU_SCOPE, type: Exclusive Access, all registrants
Transport Id of initiator:
iSCSI name and session id: iqn.2009-11.com.adriaticsolutions:blah


2. DID_BAD_TARGET host status is because I have reserved my multipathed
lun and now one of the paths is in a failed state, so mpathpersist is
failing on that path. read 4 examples below....

mpathpersist -d /dev/mapper/blah --in --read-reservation
Persistent Reserve IN command failed

sg_persist -d /dev/mapper/blah --in --read-reservation
QNAP iSCSI Storage 3.1
Peripheral device type: disk
PR generation=0xd, Reservation follows:
Key=0x0
scope: LU_SCOPE, type: Exclusive Access, all registrants

sg_persist -d /dev/sda --in --read-reservation
QNAP iSCSI Storage 3.1
Peripheral device type: disk
PR generation=0xd, Reservation follows:
Key=0x0
scope: LU_SCOPE, type: Exclusive Access, all registrants

sg_persist -d /dev/sdb --in --read-reservation
QNAP iSCSI Storage 3.1
Peripheral device type: disk
persistent reservation in: transport: Host_status=0x11 is invalid
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]

PR in: command failed







On 01/25/2012 02:28 PM, Chauhan, Vijay wrote:
> On Wednesday, January 25, 2012 9:12 PM, Fil Wrote:
>> Here is the info you requested.
>
> Fil, Thanks for the logs.
>
>> On 01/25/2012 02:45 AM, Chauhan, Vijay wrote:
>>> On January 24, 2012, Fil Wrote:
>>>>
>>>> 1. Patches apply against the 63704387009443bdb37d9deaaafa9ab121d45bfb
>>>> without any problems. Everything builds correctly (tested it on
>> centos
>>>> 6.2 and fedora 16).
>>>>
>>>> 2. multipathd runs correctly.
>>>>
>>>
>>> Fil, Thanks for the update. Good to know.
>>>
>>>> 3. mpathpersist is missing --no-inquiry option. It would be nice to
>> be
>>>> compatible with sg_persist. (breaks most of my test scripts.)
>>>>
>>>> 4. mpathpersist -d /dev/mapper/blah --in --read-status
>>>> mpathpersist: unrecognized option '--read-status'
>>>> unrecognised switch code 0x3f ??
>>>>
>>>
>>> There is a typo in usage output. Please use --read-full-status
>> instead. I will correct it.
>>
>> still no go...
>>
>
> It seems issue 4 is resolved with option '--read-full-status'
>
>> mpathpersist -d /dev/mapper/blah --in --read-full-status
>> Persistent Reserve IN command failed
>> mpathpersist -d /dev/mapper/blah --in --read-full-status -v3
>
>> Jan 25 10:32:37 | alias = blah
>> Jan 25 10:32:37 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>> command to sda
>> Jan 25 10:32:37 | sda: duration = 2 (ms)
>> Jan 25 10:32:37 | sda: status driver:00 host:04 scsi:00
>
> PRIN command returned with DID_BAD_TARGET host status. iscsi layer returns this when we get some bad output from
> target. This does not look like issue related to this feature.
>
>>>
>>>> 5. registration works
>>>>
>>>> 6. reservation, reserves only a single path. (is this by design?)
>>>
>>> Yes, this is as per design. Reservation is sent to one of the active
>>> path and reservation is applicable to registered I_T nexus with
>> respect to pr type.
>>>
>>> I was expecting your device server to apply reservation to all the
>>> registered I_T nexus for pr type "Exclusive Access, all registrants".
>> Can you verify the below output again?
>>>
>>> Please see below in SPC4
>>> "For a persistent reservation of the type Write Exclusive – All
>> Registrants or Exclusive Access – All Registrants,
>>> the persistent reservation holder is any registered I_T nexus;"
>>>
>>>> mpathpersist --out --reserve --param-rk=123abc --prout-type=8 -d
>>>> /dev/mapper/blah
>>>>
>>>> sg_persist -i -s /dev/mapper/blah
>>>> QNAP iSCSI Storage 3.1
>>>> Peripheral device type: disk
>>>> PR generation=0x8
>>>> Key=0x123abc
>>>> All target ports bit clear
>>>> Relative port address: 0x1
>>>> not reservation holder
>>>> Transport Id of initiator:
>>>> iSCSI name and session id: iqn.2009-
>>>> 11.com.adriaticsolutions:blah
>>>> Key=0x123abc
>>>> All target ports bit clear
>>>> Relative port address: 0x1
>>>> << Reservation holder >>
>>>> scope: LU_SCOPE, type: Exclusive Access, all registrants
>>>> Transport Id of initiator:
>>>> iSCSI name and session id: iqn.2009-
>>>> 11.com.adriaticsolutions:blah
>>>>
>>>> multipath -ll
>>>> blah (36001405c55fc03cd8193d491eda0d4d7) dm-4 QNAP,iSCSI Storage
>>>> size=10G features='0' hwhandler='0' wp=rw
>>>> `-+- policy='round-robin 0' prio=1 status=active
>>>> |- 16:0:0:0 sda 8:0 active ready running
>>>> `- 17:0:0:0 sdb 8:16 failed faulty running
>>>>
>>>> 7. mpathpersist -d /dev/mapper/blah --in --read-reservation
>>>> Persistent Reserve IN command failed
>>>>
>>> Can you please share the output with verbose 3. Append '-v3'.
>>
>>
>> mpathpersist -d /dev/mapper/blah --in --read-reservation
>> Persistent Reserve IN command failed
>> mpathpersist -d /dev/mapper/blah --in --read-reservation -v3
>> Jan 25 10:34:07 | alias = blah
>
>> Jan 25 10:34:07 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>> command to sda
>> Jan 25 10:34:07 | sda: duration = 1 (ms)
>> Jan 25 10:34:07 | sda: status driver:00 host:04 scsi:00
>
> Same is with issue 7. PRIN command returned with DID_BAD_TARGET host status.
>
>> Jan 25 10:34:07 | directio checker refcount 2
>> Jan 25 10:34:07 | directio checker refcount 1
>> Persistent Reserve IN command failed
>> Jan 25 10:34:07 | unloading const prioritizer
>> Jan 25 10:34:07 | unloading directio checker
>>
>>>
>>>> 8. release 'fails' because reservation registered only a single path
>> and
>>>> the other one is in a failed state.
>>>>
>>>> mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>>>> --prout-type=8
>>>> Jan 24 00:43:53 | 36001405c55fc03cd8193d491eda0d4d7: pr in read
>>>> reservation command failed.
>>>> PR out: command failed
>>>>
>>> As per design, release service action performs following steps:
>>> Step 1) PROUT 'release' SA is sent on the all paths of the multipath
>> device as it is not clear which data path is reservation holder.
>>> STEP 2) PRIN read reservation SA is sent to one of the active path to
>> verify the reservation.
>>> STEP 3) If the Read Reservation parameter data indicates that the
>> logical unit is still reserved then this indicates that the reservation
>> holder belongs to a data path in failed state or removed data path of
>> the multipath device, continue else goto step 7
>>> STEP 4) PRIN Report Full Status service action is issued to any
>> active data path. Full status descriptors are saved.
>>> STEP 5) PROUT clear reservation service action is sent via any active
>> data path to clear the reservation and
>>> Registrants
>>> STEP 6) all registrants are restored by issuing PROUT register service
>> action with transport IDs from the full status
>>> descriptors saved in the step 4
>>> STEP 7) exit with status
>>>
>>> It looks like you are getting into some other issue. Can you please
>> share the output with verbose with '-v3'.
>>
>>
>> mpathpersist -d /dev/mapper/blah --out --release --param-rk=123abc
>> --prout-type=8 -v3
>> Jan 25 10:35:41 | alias = blah
>
>> Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr out
>> command to sda
>> Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sdb path not up.
>> Jan 25 10:35:41 | sda: rq_servact = 2
>> Jan 25 10:35:41 | sda: rq_scope = 0
>> Jan 25 10:35:41 | sda: rq_type = 8
>> Jan 25 10:35:41 | sda: paramlen = 24
>> Jan 25 10:35:41 | sda: Persistent Reservation OUT parameter:
>> Jan 25 10:35:41 | 00 00 00 00 00 12 3a bc 00 00 00 00 00 00 00
>> 00
>> Jan 25 10:35:41 | 00 00 00 00 00 00 00 00
>> Jan 25 10:35:41 | sda: Duration=1 (ms)
>> Jan 25 10:35:41 | sda: status driver:00 host:00 scsi:00
>> Jan 25 10:35:41 | sda: status = 0
>> Jan 25 10:35:41 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
>> command to sda
>> Jan 25 10:35:41 | sda: duration = 1 (ms)
>> Jan 25 10:35:41 | sda: status driver:08 host:00 scsi:02
>> Jan 25 10:35:41 | sda: Sense_Key=06, ASC=00 ASCQ=00
>> Jan 25 10:35:41 | sda: retrying for Unit Attention. Remaining retries =
>> 4
>> Jan 25 10:35:41 | sda: duration = 0 (ms)
>> Jan 25 10:35:41 | sda: status driver:00 host:04 scsi:00
>
> Same with issue 8. PRIN command returned with DID_BAD_TARGET host status.
>
> All the three failures reported above are due to DID_BAD_TARGET host status returned for
> the PRIN command. It does not look like anywhere this feature is related to the above reported errors.
>
>>>> 9. de-registration works....
>>>>
>
> Thanks,
> Vijay
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 01-27-2012, 12:06 PM
"Chauhan, Vijay"
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

On Thursday, January 26, 2012 2:25 AM; Fil Wrote:
>here is an output of mpathpersist vs sg_persist
>
>[root@rhc-node3 ~]# mpathpersist -d /dev/mapper/blah --in --read-full-
>status
>Persistent Reserve IN command failed
>[root@rhc-node3 ~]# sg_persist -d /dev/mapper/blah --in --read-full-
>status
> QNAP iSCSI Storage 3.1
> Peripheral device type: disk
> PR generation=0xc
> Key=0x123abc
> All target ports bit clear
> Relative port address: 0x1
> not reservation holder
> Transport Id of initiator:
> iSCSI name and session id: iqn.2009-
>11.com.adriaticsolutions:blah
> Key=0x123abc
> All target ports bit clear
> Relative port address: 0x1
> << Reservation holder >>
> scope: LU_SCOPE, type: Exclusive Access, all registrants
> Transport Id of initiator:
> iSCSI name and session id: iqn.2009-
>11.com.adriaticsolutions:blah

Can you share the output with verbose:
* mpathpersist -d /dev/mapper/blah --in --read-full-status -v3

> 2. DID_BAD_TARGET host status is because I have reserved my multipathed
>lun and now one of the paths is in a failed state, so mpathpersist is
>failing on that path. read 4 examples below....
>
Again, your assumption is not correct here. DID_BAD_TARGET(hostbyte=0x04)
status is not something dm multipath driver or mpathpersist utility
sets in hostbyte for sense. It is set by lower layer driver.

Output log's recital - 'blah' mpath device had two paths sda and sdb.
sda was in active state where as sdb was in failed state.
mpathpersist selected sda (active path) for sending pr in command.
PR in command failed with hostbyte=DID_BAD_TARGET in lower layer.
mpathpersist reported the command as failed.

Let me know where I missed anything in understanding your log?

Also see same details step by step in the output log provided by you:
mpathpersist -d /dev/mapper/blah --in --read-full-status
Persistent Reserve IN command failed
mpathpersist -d /dev/mapper/blah --in --read-full-status -v3
Jan 25 10:32:37 | alias = blah
Jan 25 10:32:37 | ram0: device node name blacklisted
Jan 25 10:32:37 | ram1: device node name blacklisted
Jan 25 10:32:37 | ram2: device node name blacklisted
Jan 25 10:32:37 | ram3: device node name blacklisted
Jan 25 10:32:37 | ram4: device node name blacklisted
Jan 25 10:32:37 | ram5: device node name blacklisted
Jan 25 10:32:37 | ram6: device node name blacklisted
Jan 25 10:32:37 | ram7: device node name blacklisted
Jan 25 10:32:37 | ram8: device node name blacklisted
Jan 25 10:32:37 | ram9: device node name blacklisted
Jan 25 10:32:37 | ram10: device node name blacklisted
Jan 25 10:32:37 | ram11: device node name blacklisted
Jan 25 10:32:37 | ram12: device node name blacklisted
Jan 25 10:32:37 | ram13: device node name blacklisted
Jan 25 10:32:37 | ram14: device node name blacklisted
Jan 25 10:32:37 | ram15: device node name blacklisted
Jan 25 10:32:37 | loop0: device node name blacklisted
Jan 25 10:32:37 | loop1: device node name blacklisted
Jan 25 10:32:37 | loop2: device node name blacklisted
Jan 25 10:32:37 | loop3: device node name blacklisted
Jan 25 10:32:37 | loop4: device node name blacklisted
Jan 25 10:32:37 | loop5: device node name blacklisted
Jan 25 10:32:37 | loop6: device node name blacklisted
Jan 25 10:32:37 | loop7: device node name blacklisted
Jan 25 10:32:37 | vda: device node name blacklisted
Jan 25 10:32:37 | dm-0: device node name blacklisted
Jan 25 10:32:37 | dm-1: device node name blacklisted
Jan 25 10:32:37 | dm-2: device node name blacklisted
Jan 25 10:32:37 | dm-3: device node name blacklisted
Jan 25 10:32:37 | sda: not found in pathvec
Jan 25 10:32:37 | sda: mask = 0x5
Jan 25 10:32:37 | sda: dev_t = 8:0
Jan 25 10:32:37 | sda: size = 20971520
Jan 25 10:32:37 | sda: vendor = QNAP
Jan 25 10:32:37 | sda: product = iSCSI Storage
Jan 25 10:32:37 | sda: rev = 3.1
Jan 25 10:32:37 | sda: h:b:t:l = 16:0:0:0
Jan 25 10:32:37 | sda: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:32:37 | sda: path state = running
Jan 25 10:32:37 | sda: get_state
Jan 25 10:32:37 | loading /lib64/multipath/libcheckdirectio.so checker
Jan 25 10:32:37 | sda: path checker = directio (controller setting)
Jan 25 10:32:37 | sda: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:32:37 | directio: starting new request
Jan 25 10:32:37 | directio: io finished 4096/0
Jan 25 10:32:37 | sda: state = up

--> You have two physical paths for blah, i.e sda and sdb.
--> sda path is currently in active state.

Jan 25 10:32:37 | dm-4: device node name blacklisted
Jan 25 10:32:37 | sdb: not found in pathvec
Jan 25 10:32:37 | sdb: mask = 0x5
Jan 25 10:32:37 | sdb: dev_t = 8:16
Jan 25 10:32:37 | sdb: size = 20971520
Jan 25 10:32:37 | sdb: vendor = QNAP
Jan 25 10:32:37 | sdb: product = iSCSI Storage
Jan 25 10:32:37 | sdb: rev = 3.1
Jan 25 10:32:37 | sdb: h:b:t:l = 17:0:0:0
Jan 25 10:32:37 | sdb: tgt_node_name =
iqn.2004-04.com.qnap:ts-459proii:iscsi.blah.cb4d16
Jan 25 10:32:37 | sdb: path state = running
Jan 25 10:32:37 | sdb: get_state
Jan 25 10:32:37 | sdb: path checker = directio (controller setting)
Jan 25 10:32:37 | sdb: checker timeout = 30000 ms (sysfs setting)
Jan 25 10:32:37 | directio: starting new request
Jan 25 10:32:37 | directio: io finished 18446744073709551611/0
Jan 25 10:32:37 | sdb: state = down
Jan 25 10:32:37 | sdb: checker msg is "directio checker reports path is
down"

--> sdb is currently in failed state.

Jan 25 10:32:37 | params = 0 0 1 1 round-robin 0 2 1 8:0 1 8:16 1
Jan 25 10:32:37 | status = 2 0 0 0 1 1 A 0 2 0 8:0 A 0 8:16 F 2
Jan 25 10:32:37 | blah: disassemble map [0 0 1 1 round-robin 0 2 1 8:0 1
8:16 1 ]
Jan 25 10:32:37 | sda: mask = 0x8
Jan 25 10:32:37 | sda: path state = running
Jan 25 10:32:37 | loading /lib64/multipath/libprioconst.so prioritizer
Jan 25 10:32:37 | sda: prio = const (internal default)
Jan 25 10:32:37 | sda: prio = (internal default)
Jan 25 10:32:37 | sda: const prio = 1
Jan 25 10:32:37 | sdb: mask = 0x8
Jan 25 10:32:37 | sdb: path state = running
Jan 25 10:32:37 | sdb: prio = const (internal default)
Jan 25 10:32:37 | sdb: prio = (internal default)
Jan 25 10:32:37 | sdb: const prio = 1
Jan 25 10:32:37 | blah: disassemble status [2 0 0 0 1 1 A 0 2 0 8:0 A 0
8:16 F 2 ]
Jan 25 10:32:37 | 36001405c55fc03cd8193d491eda0d4d7: sending pr in
command to sda

--> Among sda and sdb, mpathpersist finds sda as active path and sends pr in command through it.

Jan 25 10:32:37 | sda: duration = 2 (ms)
Jan 25 10:32:37 | sda: status driver:00 host:04 scsi:00

--> Command returned after 2 ms with hostbyte=0x04.

Persistent Reserve IN command failed
--> mpathpersist reports error.

>mpathpersist -d /dev/mapper/blah --in --read-reservation
>Persistent Reserve IN command failed
>
>sg_persist -d /dev/mapper/blah --in --read-reservation
> QNAP iSCSI Storage 3.1
> Peripheral device type: disk
> PR generation=0xd, Reservation follows:
> Key=0x0
> scope: LU_SCOPE, type: Exclusive Access, all registrants
>
>sg_persist -d /dev/sda --in --read-reservation
> QNAP iSCSI Storage 3.1
> Peripheral device type: disk
> PR generation=0xd, Reservation follows:
> Key=0x0
> scope: LU_SCOPE, type: Exclusive Access, all registrants
>
>sg_persist -d /dev/sdb --in --read-reservation
> QNAP iSCSI Storage 3.1
> Peripheral device type: disk
>persistent reservation in: transport: Host_status=0x11 is invalid
>Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
>

This is a new issue. One of your path reports "DID_NEXUS_FAILURE".
For more insight, please share output with verbose:
* mpathpersist -d /dev/mapper/blah --in --read-reservation -v3

Fil, I appreciate your testing effort. I verified all the basic functionality
before posting it and in your opinion if it still needs to handle
few special cases and improving error handling then we can always do it.
I am more interested to know if basic functionality works fine
and does not break any existing dm feature. And also if it is useful to get it upstream.

Note: Currently, PR command is retried only for unit attention and 02/04/07 in mpathpersist error handler.

Thanks,
Vijay


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-11-2012, 07:43 AM
Christophe Varoqui
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

On ven., 2012-02-10 at 18:47 +0000, Chauhan, Vijay wrote:
> git://github.com/vijaychauhan/multipath-tools.git

Ok. This repo is merged.

I had to add a small build fix. Can you verify it's correctness ?

Are you sure the -H (hex) flag is used ? gcc reports the flag as unused.
If it's not used I think you should either remove it from the usage
message and man or add the feature.

On the same topic:
in mpathpersist,
main.c:403:7: attention : variable ‘t_arr_len’ set but not used
[-Wunused-but-set-variable]

and in libmpathpersist,
mpath_persist.c: In function ‘mpath_send_prin_activepath’:
mpath_persist.c:397:20: attention : variable ‘param’ set but not used
[-Wunused-but-set-variable]
mpath_persist.c: In function ‘mpath_prout_common’:
mpath_persist.c:553:15: attention : variable ‘pptemp’ set but not used
[-Wunused-but-set-variable]
mpath_pr_ioctl.c: In function ‘dumpHex’:
mpath_pr_ioctl.c:519:15: attention : variable ‘formatstr’ set but not
used [-Wunused-but-set-variable]


Also I though sg_persist has IN mode set as default, so that simple
commands are simple to write : sg_persist -r /dev/sda. Can we restore
that nice feature ?

Anyway, thanks for the great work.

Regards,
cvaroqui

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 
Old 02-13-2012, 01:07 PM
"Chauhan, Vijay"
 
Default multipath-tools: mpathpersist utility for managing persistent reservation on dm multipath device.

On Saturday, February 11, 2012 2:13 PM, Christophe Varoqui wrote:
>
> On ven., 2012-02-10 at 18:47 +0000, Chauhan, Vijay wrote:
> > git://github.com/vijaychauhan/multipath-tools.git
>
> Ok. This repo is merged.
>
Thanks.

> I had to add a small build fix. Can you verify it's correctness ?
>
Yes, this fix is fine.

> Are you sure the -H (hex) flag is used ? gcc reports the flag as
> unused.
> If it's not used I think you should either remove it from the usage
> message and man or add the feature.
>

I will correct it. Thanks for pointing it out.

> On the same topic:
> in mpathpersist,
> main.c:403:7: attention : variable ‘t_arr_len’ set but not used
> [-Wunused-but-set-variable]
>
> and in libmpathpersist,
> mpath_persist.c: In function ‘mpath_send_prin_activepath’:
> mpath_persist.c:397:20: attention : variable ‘param’ set but not used
> [-Wunused-but-set-variable]
> mpath_persist.c: In function ‘mpath_prout_common’:
> mpath_persist.c:553:15: attention : variable ‘pptemp’ set but not used
> [-Wunused-but-set-variable]
> mpath_pr_ioctl.c: In function ‘dumpHex’:
> mpath_pr_ioctl.c:519:15: attention : variable ‘formatstr’ set but not
> used [-Wunused-but-set-variable]
>
I will clean up these.
>
> Also I though sg_persist has IN mode set as default, so that simple
> commands are simple to write : sg_persist -r /dev/sda. Can we restore
> that nice feature ?
>

Yes, This would be a nice feature. I will submit the patch for all of your above comments. Thanks for your review.

> Anyway, thanks for the great work.

Thanks,
Vijay

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel
 

Thread Tools




All times are GMT. The time now is 07:15 PM.

VBulletin, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content Relevant URLs by vBSEO ©2007, Crawlability, Inc.
Copyright 2007 - 2008, www.linux-archive.org