[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

Thanks for your help!

I executed the pvs command and it did find a few partitions. I use iSCSI
to share the LUN's to a XenSource server, and XenSource creates the PV's
and LV's in order to provision VM's. Didn't think of that, but that's on
the disk of course.

Is there any way to configure snapshots for these already created LVM
partitions? Or do I have to whipe the disk clean (rather not do that,
but if I have to it can be done), then create the pv's and lv's and then
share the lv's true iSCSI?

In that case XenSource will again create pv's and lv's in the shared lv.
Ain't that a problem?

Sander



On Mon, 2007-11-19 at 15:06 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
>
> > I have a couple of LUN's that are handled by the multipath driver.
> > This works great, but now I want to use LVM so I can take snapshots
> > of one LUN to another LUN.
> >
> > Can someone tell me how to do this?
> >
> > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > is already part of an volume group. But I sertainly did not configure
> > that.
> >
> > Can it be that the multipath driver of devmapper did that?
>
> If you have a PV signature on the volume, LVM might have used one of the
> paths instead of using the multipath'ed device under /dev/mapper. You
> should be able to check this by running the command "pvs" - it should
> list all the detected PVs on your system.
>
> If this was the problem, you can avoid it by instructing LVM to not scan
> SCSI devices directly, by adding a line in /etc/lvm/lvm.conf like this:
>
> device {
> filter = [ "a|^/dev/mapper/.*|", "r|.*|" ]
> }
>
> This will make LVM only look at devices in /dev/mapper/ as possible PV
> candidates, ignoring all other devices. Beware if you have a PV on the
> internal drives though, you might want to have something like this
> instead in that case (if that PV is on /dev/sda2, for instance):
>
> filter = [ "a|^/dev/mapper/.*|", "a|^/dev/sda2$|", "r|.*|" ]
>
> Make sure lvm.conf makes its way into the initramfs if you start the LVM
> stuff there. I know Ubuntu doesn't include this file by default, at least.
>
> Regards

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

P.S. pvs only scanned the real disks and not the /dev/mapper/diskname
disks. So some of the scan's where rejected because the path to that
path was the redudant path.

But I assume that the /dev/mapper/diskname will have the exact same lvm
info written on it, ass it is in fact the real disk (/dev/sdi for
example)?

Sander


On Mon, 2007-11-19 at 15:06 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
>
> > I have a couple of LUN's that are handled by the multipath driver.
> > This works great, but now I want to use LVM so I can take snapshots
> > of one LUN to another LUN.
> >
> > Can someone tell me how to do this?
> >
> > When I do pvcreate /dev/mapper/diskname I get an error that the disk
> > is already part of an volume group. But I sertainly did not configure
> > that.
> >
> > Can it be that the multipath driver of devmapper did that?
>
> If you have a PV signature on the volume, LVM might have used one of the
> paths instead of using the multipath'ed device under /dev/mapper. You
> should be able to check this by running the command "pvs" - it should
> list all the detected PVs on your system.
>
> If this was the problem, you can avoid it by instructing LVM to not scan
> SCSI devices directly, by adding a line in /etc/lvm/lvm.conf like this:
>
> device {
> filter = [ "a|^/dev/mapper/.*|", "r|.*|" ]
> }
>
> This will make LVM only look at devices in /dev/mapper/ as possible PV
> candidates, ignoring all other devices. Beware if you have a PV on the
> internal drives though, you might want to have something like this
> instead in that case (if that PV is on /dev/sda2, for instance):
>
> filter = [ "a|^/dev/mapper/.*|", "a|^/dev/sda2$|", "r|.*|" ]
>
> Make sure lvm.conf makes its way into the initramfs if you start the LVM
> stuff there. I know Ubuntu doesn't include this file by default, at least.
>
> Regards

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

* S. J. van Harmelen


Thanks for your help!

I executed the pvs command and it did find a few partitions. I use
iSCSI to share the LUN's to a XenSource server, and XenSource creates
the PV's and LV's in order to provision VM's. Didn't think of that,
but that's on the disk of course.

Is there any way to configure snapshots for these already created LVM
partitions? Or do I have to whipe the disk clean (rather not do
that, but if I have to it can be done), then create the pv's and lv's
and then share the lv's true iSCSI?

In that case XenSource will again create pv's and lv's in the shared
lv. Ain't that a problem?


You really need a filter line in /etc/lvm/lvm.conf, to keep LVM from
claiming 1) individual paths to a dm-multipath controlled LUN, and 2)
LUNs that are PVs used by the LVM implementation in your domUs.


You don't want LVM to be claiming the paths directly, because you'd no
longer have any redundancy - it will override dm-multipath. You also
don't want the LVM implementation in your dom0 to be using the same
block device as LVM in your domU as physical volumes, as LVM is designed
to have exclusive access to its PVs (you might get around this with
CLVM, though).


Do I understand correctly if you have one LUN that contains your domO
(having its file systems in LVs), and several LUNs that contain your
domUs, which again are PVs that are supposed to be used by your domU
kernels?


If so you will want to keep your dom0 from claiming the domU PVs. Use a
filter line that accepts only the multipath'ed volume that contains the
dom0 installation, and drop the rest. For instance:


filter = [ "a|^/dev/mapper/domzero$", "r|.*" ]

...assuming your dom0 data volume is called "domzero" in multipath.conf.

Regards
--
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

S. J. van Harmelen [svh@dds.nl] wrote:
> Thanks for your help!
>
> I executed the pvs command and it did find a few partitions. I use iSCSI
> to share the LUN's to a XenSource server, and XenSource creates the PV's
> and LV's in order to provision VM's. Didn't think of that, but that's on
> the disk of course.
>
> Is there any way to configure snapshots for these already created LVM
> partitions?

If I understand, you exported some LUNs to XenSource server. So you
should NOT be using those LUNs at this machine, right? You should be
able to create snapshots at the XenSource server though.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

On Mon, 2007-11-19 at 10:50 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> > Thanks for your help!
> >
> > I executed the pvs command and it did find a few partitions. I use iSCSI
> > to share the LUN's to a XenSource server, and XenSource creates the PV's
> > and LV's in order to provision VM's. Didn't think of that, but that's on
> > the disk of course.
> >
> > Is there any way to configure snapshots for these already created LVM
> > partitions?
>
> If I understand, you exported some LUNs to XenSource server. So you
> should NOT be using those LUNs at this machine, right? You should be
> able to create snapshots at the XenSource server though.

That is correct. Sorry if I did not give enough information to
understand the problem at first, but what your saying now is correct.

I have a storage server with an MD3000 attached to it. This server runs
the multipath-tools to create redundant paths:

root@storage:~# multipath -ll
<snip>
xen (360019b9000d7e11000004485473faa94) dm-0 DELL ,MD3000
[size=200G][features=0][hwhandler=0]
\_ round-robin 0 [prio=3][active]
\_ 1:0:0:3 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:1:3 sdi 8:128 [active][ghost]
<snip>

So the path that I use for sharing the LUN with the XenSource server
is /dev/mapper/xen:

root@storage:~# cat /etc/ietd.conf
<snip>
Target iqn.2007-11.local.xillan:storage.xen.disk1
Lun 0 Path=/dev/mapper/xen,Type=blockio
<snip>

On the XenSource server I use a shared iSCSI storage so multiple servers
can access the same data so I can do live migrations. When I add the
storage XenSource creates a PV and then when I create a VM, it creates
LV's for each virtual harddisk.

So far so good I guess, as the XenSource machine uses the multipathed
drive, right?

Now I would like to take snapshots of the whole PV so I can make
backups. My thoughts where to make a PV on /dev/mapper/xen that spans
the whole disk, and then create a single LV on it.

If I then take the path to that single LV, say /dev/volumegroup/disk1 as
target for iSCSI, then the XenSsource server should only see that LV as
the shared iSCSI repository.

But then XenSource wil create a PV and a couple of LV's in the existing
LV (created on the storage server and shared true iSCSI). Could that
create any problems or can I just do that?

In this way, I can take the snapshots on the storage server and I don't
have to create a second iSCSI target for the XenSource server, And I
don't have to make extensive use of the network (it can then be done on
one machine directly connected to both LUN's).

Sorry for the long post, but hope you give me some insight on this.

Thanks,

Sander

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

Tore,

Please take a look at my reply to Malaha. This is also a reply to your
message...

Hope you have the time to read it.

Thanks,

Sander


On Mon, 2007-11-19 at 19:43 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
>
> > Thanks for your help!
> >
> > I executed the pvs command and it did find a few partitions. I use
> > iSCSI to share the LUN's to a XenSource server, and XenSource creates
> > the PV's and LV's in order to provision VM's. Didn't think of that,
> > but that's on the disk of course.
> >
> > Is there any way to configure snapshots for these already created LVM
> > partitions? Or do I have to whipe the disk clean (rather not do
> > that, but if I have to it can be done), then create the pv's and lv's
> > and then share the lv's true iSCSI?
> >
> > In that case XenSource will again create pv's and lv's in the shared
> > lv. Ain't that a problem?
>
> You really need a filter line in /etc/lvm/lvm.conf, to keep LVM from
> claiming 1) individual paths to a dm-multipath controlled LUN, and 2)
> LUNs that are PVs used by the LVM implementation in your domUs.
>
> You don't want LVM to be claiming the paths directly, because you'd no
> longer have any redundancy - it will override dm-multipath. You also
> don't want the LVM implementation in your dom0 to be using the same
> block device as LVM in your domU as physical volumes, as LVM is designed
> to have exclusive access to its PVs (you might get around this with
> CLVM, though).
>
> Do I understand correctly if you have one LUN that contains your domO
> (having its file systems in LVs), and several LUNs that contain your
> domUs, which again are PVs that are supposed to be used by your domU
> kernels?
>
> If so you will want to keep your dom0 from claiming the domU PVs. Use a
> filter line that accepts only the multipath'ed volume that contains the
> dom0 installation, and drop the rest. For instance:
>
> filter = [ "a|^/dev/mapper/domzero$", "r|.*" ]
>
> ...assuming your dom0 data volume is called "domzero" in multipath.conf.
>
> Regards

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

S. J. van Harmelen [svh@dds.nl] wrote:
>
> On the XenSource server I use a shared iSCSI storage so multiple servers
> can access the same data so I can do live migrations. When I add the
> storage XenSource creates a PV and then when I create a VM, it creates
> LV's for each virtual harddisk.
>
> So far so good I guess, as the XenSource machine uses the multipathed
> drive, right?

Should.

> Now I would like to take snapshots of the whole PV so I can make
> backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> the whole disk, and then create a single LV on it.
>
> If I then take the path to that single LV, say /dev/volumegroup/disk1 as
> target for iSCSI, then the XenSsource server should only see that LV as
> the shared iSCSI repository.
>
> But then XenSource wil create a PV and a couple of LV's in the existing
> LV (created on the storage server and shared true iSCSI). Could that
> create any problems or can I just do that?

I don't see any problem. You should be able to do that, but then I am
not an expert. Give it a try.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

* S. J. van Harmelen

> I have a storage server with an MD3000 attached to it. This server
> runs the multipath-tools to create redundant paths:
>
> root@storage:~# multipath -ll
> <snip>
> xen (360019b9000d7e11000004485473faa94) dm-0 DELL ,MD3000
> [size=200G][features=0][hwhandler=0]
> \_ round-robin 0 [prio=3][active]
> \_ 1:0:0:3 sde 8:64 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ 1:0:1:3 sdi 8:128 [active][ghost]
> <snip>

I think you need hwhandler = "1 rdac" for the MD3000 to be able to fail
over properly. At least a colleague that have played with it told me so.

> On the XenSource server I use a shared iSCSI storage so multiple
> servers can access the same data so I can do live migrations. When I
> add the storage XenSource creates a PV and then when I create a VM,
> it creates LV's for each virtual harddisk.
>
> So far so good I guess, as the XenSource machine uses the multipathed
> drive, right?

You absolutely need to keep LVM (in dom0) from using /dev/sde or
/dev/sdi as the PV (instead of /dev/mapper/xen). You can accomplish
this with a filter line in lvm.conf:

filter = [ "a|^/dev/mapper/xen$|", "r|.*|" ]

Which means that a device node that is exactly «/dev/mapper/xen» is
considered a valid PV candidate, all other devices are rejected.

>From the email you sent with earlier pvscan output you got /dev/dm-0 as
the active PV, with pvs you got /dev/sde (and with both you got I/O
errors to the passive paths). It can't be relied upon that you'll get
dm-0 (which is absolutely necessary if you want dm-multipath to serve
any purpose) in the future unless you force LVM to only look for PVs in
the multipath'ed block devices.

> Now I would like to take snapshots of the whole PV so I can make
> backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> the whole disk, and then create a single LV on it.

You can only take snapshots of LVs, not PVs. And you would have to
create a VG first with /dev/mapper/xen as a PV (you can't add LVs
directly on PVs). But I think you've got the basic idea correct. :-)

> If I then take the path to that single LV, say /dev/volumegroup/disk1
> as target for iSCSI, then the XenSsource server should only see that
> LV as the shared iSCSI repository.
>
> But then XenSource wil create a PV and a couple of LV's in the
> existing LV (created on the storage server and shared true iSCSI).
> Could that create any problems or can I just do that?

I see no reason why that wouldn't work. Try it and see? Inside the
domU the Linux kernel won't know that its available block devices are
really mapped to another LVM implementation that runs in dom0. LVM is
probably clever enough to disregard that LV-containing-a-PV as a valid
PV candidate, but if not the filter line in lvm.conf should handle it.

Regards
--
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

On Tue, 2007-11-20 at 08:36 +0100, Tore Anderson wrote:
> * S. J. van Harmelen
>
> > I have a storage server with an MD3000 attached to it. This server
> > runs the multipath-tools to create redundant paths:
> >
> > root@storage:~# multipath -ll
> > <snip>
> > xen (360019b9000d7e11000004485473faa94) dm-0 DELL ,MD3000
> > [size=200G][features=0][hwhandler=0]
> > \_ round-robin 0 [prio=3][active]
> > \_ 1:0:0:3 sde 8:64 [active][ready]
> > \_ round-robin 0 [prio=0][enabled]
> > \_ 1:0:1:3 sdi 8:128 [active][ghost]
> > <snip>
>
> I think you need hwhandler = "1 rdac" for the MD3000 to be able to fail
> over properly. At least a colleague that have played with it told me so.

You are correct about that, but I did have some issues with the new
2.6.23.x kernels. So for testing I went back to a 2.6.22.x kernel which
does not have the RDAC hardware handler.

When I do use a 2.6.23.x kernel with the RDAC hardware handler I get the
following error: multipathd[6895]: segfault at
000000000000000a rip 00002b251f24394f rsp 00007fff8bfaa730 error 4

Besides that it seems to work fine, but I do not know if this can be
disgarded?

>
> > On the XenSource server I use a shared iSCSI storage so multiple
> > servers can access the same data so I can do live migrations. When I
> > add the storage XenSource creates a PV and then when I create a VM,
> > it creates LV's for each virtual harddisk.
> >
> > So far so good I guess, as the XenSource machine uses the multipathed
> > drive, right?
>
> You absolutely need to keep LVM (in dom0) from using /dev/sde or
> /dev/sdi as the PV (instead of /dev/mapper/xen). You can accomplish
> this with a filter line in lvm.conf:
>
> filter = [ "a|^/dev/mapper/xen$|", "r|.*|" ]
>
> Which means that a device node that is exactly «/dev/mapper/xen» is
> considered a valid PV candidate, all other devices are rejected.

Will add this line to lvm.conf. Thanks!

>
> >From the email you sent with earlier pvscan output you got /dev/dm-0 as
> the active PV, with pvs you got /dev/sde (and with both you got I/O
> errors to the passive paths). It can't be relied upon that you'll get
> dm-0 (which is absolutely necessary if you want dm-multipath to serve
> any purpose) in the future unless you force LVM to only look for PVs in
> the multipath'ed block devices.
>
> > Now I would like to take snapshots of the whole PV so I can make
> > backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> > the whole disk, and then create a single LV on it.
>
> You can only take snapshots of LVs, not PVs. And you would have to
> create a VG first with /dev/mapper/xen as a PV (you can't add LVs
> directly on PVs). But I think you've got the basic idea correct. :-)

Indeed, I understand :)

>
> > If I then take the path to that single LV, say /dev/volumegroup/disk1
> > as target for iSCSI, then the XenSsource server should only see that
> > LV as the shared iSCSI repository.
> >
> > But then XenSource wil create a PV and a couple of LV's in the
> > existing LV (created on the storage server and shared true iSCSI).
> > Could that create any problems or can I just do that?
>
> I see no reason why that wouldn't work. Try it and see? Inside the
> domU the Linux kernel won't know that its available block devices are
> really mapped to another LVM implementation that runs in dom0. LVM is
> probably clever enough to disregard that LV-containing-a-PV as a valid
> PV candidate, but if not the filter line in lvm.conf should handle it.

Oke, I will give it a try then.

Thanks,

Sander


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] Use LVM on /dev/mapper/diskname and iSCSI
 

On Mon, 2007-11-19 at 14:36 -0800, malahal@us.ibm.com wrote:
> S. J. van Harmelen [svh@dds.nl] wrote:
> >
> > On the XenSource server I use a shared iSCSI storage so multiple servers
> > can access the same data so I can do live migrations. When I add the
> > storage XenSource creates a PV and then when I create a VM, it creates
> > LV's for each virtual harddisk.
> >
> > So far so good I guess, as the XenSource machine uses the multipathed
> > drive, right?
>
> Should.
>
> > Now I would like to take snapshots of the whole PV so I can make
> > backups. My thoughts where to make a PV on /dev/mapper/xen that spans
> > the whole disk, and then create a single LV on it.
> >
> > If I then take the path to that single LV, say /dev/volumegroup/disk1 as
> > target for iSCSI, then the XenSsource server should only see that LV as
> > the shared iSCSI repository.
> >
> > But then XenSource wil create a PV and a couple of LV's in the existing
> > LV (created on the storage server and shared true iSCSI). Could that
> > create any problems or can I just do that?
>
> I don't see any problem. You should be able to do that, but then I am
> not an expert. Give it a try.

Thanks, I will give it a try then!

Sander

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel