Status of resizing/growing dm-multipath devices on the fly
 

Pasi K?rkk?inen <pasik@iki.fi> wrote:
> On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > * Pasi Kärkkäinen wrote:
> > >
> > > Is it possible to resize (grow) dm-multipath devices on the
> > > fly nowadays?
> > >
> > > I googled and found some discussions about the subject, but
> > > the conclusion seemed to be it's not possible.. that was a
> > > while ago, so I was wondering if this has been fixed/implemented..
> > >
> > > Using LVM and adding another new LUN/PV is not an option
> > > always.. it's a lot easier to manage the whole thing if it's
> > > possible to resize/grow existing LUNs on the fly.
> >
> > Nowadays all disk-arrays make online LUN extension a breeze but
> > unfortunately it seems that dm-multipath is still not able to see new size
> > without downtime:
> > http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > Look also at this recent thread on RHEL5 mailing-list where you can find
> > also a post from a Red Hat representative:
> > http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> >
> > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > viable solution because it causes LUN proliferation.
> >
> > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > solve this SHAME!
> > It's a feature that Linux really needs.
> >
> > Hope someone pick this cry of pain!
> >
>
> Thanks for the reply.
>
> Does someone know what's the actual problem in this? meaning why it hasn't
> been done yet..

It has to do with the block layer not using the new size value while there
are openers.

Andrew submitted a patch series (url below) a while ago to try and address
this issue, but I do not know the status. I added Andrew to the cc.

http://thread.gmane.org/gmane.linux.scsi/41623

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Status of resizing/growing dm-multipath devices on the fly
 

On Wed, 2008-08-06 at 12:36 -0700, Mike Anderson wrote:
> Pasi K?rkk?inen <pasik@iki.fi> wrote:
> > On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > > * Pasi Kärkkäinen wrote:
> > > >
> > > > Is it possible to resize (grow) dm-multipath devices on the
> > > > fly nowadays?
> > > >
> > > > I googled and found some discussions about the subject, but
> > > > the conclusion seemed to be it's not possible.. that was a
> > > > while ago, so I was wondering if this has been fixed/implemented..
> > > >
> > > > Using LVM and adding another new LUN/PV is not an option
> > > > always.. it's a lot easier to manage the whole thing if it's
> > > > possible to resize/grow existing LUNs on the fly.
> > >
> > > Nowadays all disk-arrays make online LUN extension a breeze but
> > > unfortunately it seems that dm-multipath is still not able to see new size
> > > without downtime:
> > > http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > > Look also at this recent thread on RHEL5 mailing-list where you can find
> > > also a post from a Red Hat representative:
> > > http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> > >
> > > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > > viable solution because it causes LUN proliferation.
> > >
> > > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > > solve this SHAME!
> > > It's a feature that Linux really needs.
> > >
> > > Hope someone pick this cry of pain!
> > >
> >
> > Thanks for the reply.
> >
> > Does someone know what's the actual problem in this? meaning why it hasn't
> > been done yet..
>
> It has to do with the block layer not using the new size value while there
> are openers.
>
> Andrew submitted a patch series (url below) a while ago to try and address
> this issue, but I do not know the status. I added Andrew to the cc.
>
> http://thread.gmane.org/gmane.linux.scsi/41623
>

The above patchset works with SCSI devices. I have tested it with single
SCSI devices under dm/LVM, i.e., a PV is on a single SCSI LUN with only
one path to it. It has not yet been accepted upstream due to not getting
sign-off from Al Viro. I plan on resubmitting soon to add cciss support.
Hopefully Al will getting around to reviewing it sometime this century.
I have heard rumors that Al has some all-encompassing solution for this
problem but hasn't got around to implementing it. I'll test my new
patches with multi-path while I am at it.

Andrew




--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Status of resizing/growing dm-multipath devices on the fly
 

On Wed, 6 Aug 2008, Pasi Kärkkäinen wrote:


Hello!

Is it possible to resize (grow) dm-multipath devices on the fly nowadays?


I googled and found some discussions about the subject, but the conclusion
seemed to be it's not possible.. that was a while ago, so I was wondering if
this has been fixed/implemented..


Using LVM and adding another new LUN/PV is not an option always.. it's a lot
easier to manage the whole thing if it's possible to resize/grow existing
LUNs on the fly.



Sure it is, but it's not pretty. Don't believe the naysayers.
We seem to have a working method for resizing multipath luns from our
NetApp filers. It works on CentOS 4 & 5, for us, and our NetApps.

(2 HBA, times 2 paths on each makes for four SCSI LUN to resize)

The poorly formatted, unpolished, YMMV, no warranty recipe is:

1. Record the state of the dm tables

dmsetup table resize_test | tee orig current new final

2. Split one of the paths

multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=1.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 5:0:0:0 sdb 8:16 [active][undef]
\_ 6:0:0:0 sdc 8:32 [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 6:0:1:0 sdd 8:48 [active][undef]
\_ 5:0:1:0 sde 8:64 [active][undef]

# Choose a pair: sdb+sdc or sdd+sde. For this, let's choose sdd+sde

# Update current
vi current
0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000


# Change this to:
0 2097152 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 4000
8:32 4000


# Note this is what happened:
# Remove the second pair beginning with the second "round-robin". We know
it's sdd+sde based on the major/minor pairs.
# Reduce the number of pairs after "queue_if_no_path" from 2 (0 2 1) to 1
(0 1 1)


multipathd -k

del path sdd
del path sde


# Reload the dm table
dmsetup suspend resize_test current; dmsetup reload resize_test current;
dmsetup resume resize_test


# Verify you now just have one set of paths
multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=1.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
\_ 5:0:0:0 sdb 8:16 [active][undef]
\_ 6:0:0:0 sdc 8:32 [active][undef]

3. Grow the LUN on the SAN (although, this can happen before steps 1 & 2
as well)


(NetApp specific:)
lun resize /vol/resize_test/lun 10g

4. # Update partition table

blockdev --rereadpt /dev/sdd
blockdev --rereadpt /dev/sde
blockdev --getsize /dev/sdd
blockdev --getsize /dev/sde

# The values for getsize in both commands must be the SAME
# Record this value
# The above gave us the value 20971520

# Update the dm table for new
vi new

0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000


# Change this to:
0 20971520 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000
8:64 1000


# Note this is what happened:
# Update the second parameter (num sectors) from 2097152 to 20971520
(obtained from blockdev --getsize)

# Remove the second pair beginning with the second "round-robin"
# Reduce the number of pairs after "queue_if_no_path" from 2 (0 2 1) to 1
(0 1 1)


5. Switch to the new path

multipathd -k

add path sdd
add path sde
del path sdb
del path sdc


# Note: We are adding the INACTIVE Paths first (sdd+sde) and then removing
# the ACTIVE paths (sdb+sdc). This is VERY important. I tried reversing it
by
# deleting ACTIVE first then adding INACTIVE. Fortunately, multipathd kept
one

# of them, refusing to budge.

dmsetup suspend resize_test; dmsetup reload resize_test new; dmsetup
resume resize_test


# Verify we've switched to the new path:
multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=10G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
\_ 6:0:1:0 sdd 8:48 [active][undef]
\_ 5:0:1:0 sde 8:64 [active][undef]

# Note: multipath now shows the new size (10G). If it still shows the old
size, chances are
# that one of the block devices in the pair have not had their partition
tables updated.

# TURN BACK! AND MAKE SURE TO REREAD THE PARTITION TABLES!


6. Update the block devices on the inactive path

blockdev --rereadpt /dev/sdb
blockdev --rereadpt /dev/sdc
blockdev --getsize /dev/sdb
blockdev --getsize /dev/sdc

7. Re-add the final path

# Update the final dm table

vi final
0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000


# Change to:
0 20971520 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000


# Note this is what happened:
# Update the second parameter (num sectors) from 2097152 to 20971520
# (obtained from blockdev --getsize)

multipathd -k
add path sdb
add path sdc

dmsetup suspend resize_test; dmsetup reload resize_test final; dmsetup
resume resize_test


multipathd -k
reconfigure

# Note: This reconfigure is *ESSENTIAL* as it forces multipath to remember
the new size. Without this step, failing one of the paths
# can cause multipath to reuse the previous block device size and cause
filesystem problems.


8. Resize the FS from userland

ext2online /dev/mapper/resize_test
(or resize2fs in newer )


Pfew! Done!
I think this should have been shared by us (myself, Gino Ledesma, Michael Hsu) a long time ago.
We probably wanted to make a pretty pretty script to do it for us.

There is a slightly more convoluted recipe if you were bold and foolish
enough to partition the LUN (like we once were).


Cheers,

-sharif
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Status of resizing/growing dm-multipath devices on the fly
 

On Thu, 7 Aug 2008, Domenico Viggiani wrote:


* Sharif Nassar wrote:


We seem to have a working method for resizing multipath luns
from our NetApp filers. It works on CentOS 4 & 5, for us,
and our NetApps.
(2 HBA, times 2 paths on each makes for four SCSI LUN to resize)

The poorly formatted, unpolished, YMMV, no warranty recipe is:

...

Pfew! Done!


Incredible! I will study this later, thanks.
Any idea if this could work with an EMC Clariion CX 700 array?



While we haven't tested it, I don't see any reason why not. Since the
core concern is freeing the device so the kernel can see the new size, it
shouldn't be an issue.


-sharif

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel