configure MD3000 episode 2
 

I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
the help of this list I managed to configure the RAID etc. However
there is one remaining problem. udev doesn't seem to recognize this
setup. After every reboot I have to manually create the /dev/dm-*
devices with mknod and run kpartx to make the partitions show up. Can
someone please hint to me where I get the appropriate udev scripts ?
Ubuntu does have dmapper and multipath related scripts in the udev
directory, but they don't seem to be working for me.

Thank you so much,
Thomas Witzel

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
> the help of this list I managed to configure the RAID etc. However
> there is one remaining problem. udev doesn't seem to recognize this
> setup. After every reboot I have to manually create the /dev/dm-*
> devices with mknod and run kpartx to make the partitions show up. Can
> someone please hint to me where I get the appropriate udev scripts ?

You can drop this in your udev/rules.d directory:

> cat /etc/udev/rules.d/95-mpdc.rules
RUN+="socket:/org/kernel/dm/multipath_event"


Obviously you need to have your multipathd to start _before_ your
iSCSI daemon.

> Ubuntu does have dmapper and multipath related scripts in the udev
> directory, but they don't seem to be working for me.
>
> Thank you so much,
> Thomas Witzel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

Thanks, I added that. But it didn't change anything. I'm not sure what
you want me to do with iscsi. I don't think I have an iscsi device. I
have heard about the MD3000i before, but thats not what I have. I
think I have a MD3000 (without i) connected via Symbios Logic SAS1068
PCI-X Fusion-MPT SAS controller.

Thomas

On Mon, Mar 23, 2009 at 10:31 AM, Konrad Rzeszutek
<konrad@virtualiron.com> wrote:
> On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
>> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
>> the help of this list I managed to configure the RAID etc. However
>> there is one remaining problem. udev doesn't seem to recognize this
>> setup. After every reboot I have to manually create the /dev/dm-*
>> devices with mknod and run kpartx to make the partitions show up. Can
>> someone please hint to me where I get the appropriate udev scripts ?
>
> You can drop this in your udev/rules.d directory:
>
>> cat /etc/udev/rules.d/95-mpdc.rules
> RUN+="socket:/org/kernel/dm/multipath_event"
>
>
> Obviously you need to have your multipathd to start _before_ your
> iSCSI daemon.
>
>> Ubuntu does have dmapper and multipath related scripts in the udev
>> directory, but they don't seem to be working for me.
>>
>> Thank you so much,
>> Thomas Witzel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

I'm now looking at the udev monitor and also set the loglevel to debug
and I can see that udev is processing the multipath events, hence
creating the /dev/mapper entries (but not the partition entries). But
there is never anything called that would create the /dev/dm-* devices
and the 95-kpartx rule is also never called it seems.
I have now rigged it to run a script on boot that calls mkdnod and
kpartx, but of course I'd still like udev support. I have not been
able to get any answer from ubuntu guys, so is it a fact that
multipath is basically not supported under ubuntu ? What would be the
next best distribution thats free of annual license fees ?

Thank you very much again,
Thomas


On Mon, Mar 23, 2009 at 11:44 AM, Thomas Witzel <witzel.thomas@gmail.com> wrote:
> Thanks, I added that. But it didn't change anything. I'm not sure what
> you want me to do with iscsi. I don't think I have an iscsi device. I
> have heard about the MD3000i before, but thats not what I have. I
> think I have a MD3000 (without i) connected via Symbios Logic SAS1068
> PCI-X Fusion-MPT SAS controller.
>
> Thomas
>
> On Mon, Mar 23, 2009 at 10:31 AM, Konrad Rzeszutek
> <konrad@virtualiron.com> wrote:
>> On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
>>> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
>>> the help of this list I managed to configure the RAID etc. However
>>> there is one remaining problem. udev doesn't seem to recognize this
>>> setup. After every reboot I have to manually create the /dev/dm-*
>>> devices with mknod and run kpartx to make the partitions show up. Can
>>> someone please hint to me where I get the appropriate udev scripts ?
>>
>> You can drop this in your udev/rules.d directory:
>>
>>> cat /etc/udev/rules.d/95-mpdc.rules
>> RUN+="socket:/org/kernel/dm/multipath_event"
>>
>>
>> Obviously you need to have your multipathd to start _before_ your
>> iSCSI daemon.
>>
>>> Ubuntu does have dmapper and multipath related scripts in the udev
>>> directory, but they don't seem to be working for me.
>>>
>>> Thank you so much,
>>> Thomas Witzel
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

On Mon, Mar 23, 2009 at 05:05:37PM -0400, Thomas Witzel wrote:
> I'm now looking at the udev monitor and also set the loglevel to debug
> and I can see that udev is processing the multipath events, hence
> creating the /dev/mapper entries (but not the partition entries). But

The mechanism that does this is a bit more complex. Let me explain
to you and hopefully that will help you out.

When an HBA inits, it ends up calling a bunch of internal sd.c routines
which end up creating uevents (and of course setup the SysFS structures).
There are usually four sets of them:

1) device_add creates this:

[add@/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[ACTION=add]
[DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[SUBSYSTEM=scsi]
[SEQNUM=8969]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

2) sd_probe creates this guy:

[add@/class/scsi_disk/9:0:0:15]
[ACTION=add]
[DEVPATH=/class/scsi_disk/9:0:0:15]
[SUBSYSTEM=scsi_disk]
[SEQNUM=9088]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

3) add_disk creates this one:

add@/block/sdo]
[ACTION=add]
[DEVPATH=/block/sdo]
[SUBSYSTEM=block]
[SEQNUM=9332]
[MINOR=224]
[MAJOR=8]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

4) and sg_add this one:

[add@/class/scsi_generic/sg15]
[ACTION=add]
[DEVPATH=/class/scsi_generic/sg15]
[SUBSYSTEM=scsi_generic]
[SEQNUM=9110]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]
[MAJOR=21]
[MINOR=15]

the kernel injects them in a netlink (NETLINK_KOBJECT_UEVENT) - I've attached a small C code
that grabs them. At that point the block disk is now usuable and userspace code can
access it by the major:minor numbers.

udev listens to this netlink and based on the udev rules execute them. But there are rules
in there that will short-circuit these uevents. If you see 'RUN=" that means it is done and won't
pass it on to the next one. If you see 'RUN+=' that means process it and pass it on to next rules
that might match the criteria. Udev is the one that actually creates the /dev/sdo.

There might a udev rule in there that eats up the uevent before it is passed to the multipathd
netlink socket (/org/kernel/dm/multipath_event). You can make it less of chance if you
change the name of the rule to a low number, like 01-mpdc.rules.

Multipathd acts on the #3 uevent - it interrogates the block disk and then calls the
device mapper ioctl (/dev/mapper/control). The device mapper (kernel piece) sends its own
uevent, which looks as so:

[add@/block/dm-19]
[ACTION=add]
[DEVPATH=/block/dm-19]
[SUBSYSTEM=block]
[SEQNUM=9412]
[MINOR=19]
[MAJOR=253]

And udev then creates /dev/dm-19. It then would pass this on to multipathd socket. And
multipathd then does it stuff (which is to check the paths, make sure everthing
is right and kick of a checker path if it hasn't already been done).

> there is never anything called that would create the /dev/dm-* devices
> and the 95-kpartx rule is also never called it seems.

> I have now rigged it to run a script on boot that calls mkdnod and
> kpartx, but of course I'd still like udev support. I have not been

During bootup the udev isn't run. I think what gets called is the
HOTPLUG program, which is defined by default is "/sbin/hotplug".
Look in that program on the initrd image. It might not do any udev stuff at all
and just simple block disk creations.

> able to get any answer from ubuntu guys, so is it a fact that
> multipath is basically not supported under ubuntu ? What would be the
> next best distribution thats free of annual license fees ?

Huh? Ubuntu charges license fees on GPL code? I think you are confusing
what you are paying - it isn't license fee but support fee.
#include <sys/socket.h>
#include <linux/netlink.h>
#include <stdio.h>
#include <string.h>
#define MAX_PAYLOAD 1024 /* maximum payload size */
struct sockaddr_nl src_addr;
int sock_fd;
static char buff[MAX_PAYLOAD];
ssize_t buflen;

int
main ()
{
sock_fd = socket (PF_NETLINK, SOCK_DGRAM, NETLINK_KOBJECT_UEVENT);

memset (&src_addr, 0, sizeof (src_addr));
src_addr.nl_family = AF_NETLINK;
src_addr.nl_pid = getpid (); /* self pid */
src_addr.nl_groups = 0xffffffff;

printf ("Listen..
");
bind (sock_fd, (struct sockaddr *) &src_addr, sizeof (src_addr));
printf ("Receiving..
");
while (1)
{
buflen = recv (sock_fd, &buff, sizeof (buff), 0);
printf ("Got data: %d
", buflen);
int i, bufpos;
char *key;
for (i = 0, bufpos = 0; (bufpos < buflen) && i < MAX_PAYLOAD; i++)
{
key = &buff[bufpos];
printf ("[%s]
", key);
bufpos += strlen (key) + 1;
}
memset (&buff, 0, MAX_PAYLOAD);
}
/* Close Netlink Socket */
close (sock_fd);
return 0;
}
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

> Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> what you are paying - it isn't license fee but support fee.

Thats not what I meant. I know that my hardware config does work out
of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
I have not been able to get any of this to work for 3 weeks now and I
have not been able to get help/support on the issue, so I was asking
whether there is a different free distribution that won't put me
through these time consuming google/mailing list/debug orgies to get
some storage mounted.

Thomas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

Okay, I'm looking for a man page:
man -k hotplug finds nothing

I'm looking for the file you mentioned:
ls /sbin/hotplug
ls: cannot access /sbin/hotplug: No such file or directory

I'm looking for a package to install related to hotplug:

apt-cache search hotplug
libxi-dev - X11 Input extension library (development headers)
libxi6 - X11 Input extension library
libxi6-dbg - X11 Input extension library (debug package)
fxload - Firmware download to EZ-USB devices
libaws-dev - Ada Web Server development files
atmel-firmware - Firmware for Atmel at76c50x wireless networking chips.

Seems like there is NOTHING on ubuntu even remotely called hotplug,
except /proc/sys/kernel/hotplug
which is not a configuration file.

What distribution are you using ?

Thomas

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

On Tue, Mar 24, 2009 at 10:29:13AM -0400, Thomas Witzel wrote:
> Okay, I'm looking for a man page:
> man -k hotplug finds nothing
>
> I'm looking for the file you mentioned:
> ls /sbin/hotplug
> ls: cannot access /sbin/hotplug: No such file or directory
>
> I'm looking for a package to install related to hotplug:
>
> apt-cache search hotplug
> libxi-dev - X11 Input extension library (development headers)
> libxi6 - X11 Input extension library
> libxi6-dbg - X11 Input extension library (debug package)
> fxload - Firmware download to EZ-USB devices
> libaws-dev - Ada Web Server development files
> atmel-firmware - Firmware for Atmel at76c50x wireless networking chips.
>
> Seems like there is NOTHING on ubuntu even remotely called hotplug,
> except /proc/sys/kernel/hotplug

I am looking at my Debian box, and it looks as if the initrd
/scripts/init-premount/udev does start the udev daemon. And the
/etc/udev/rules.d has many rules.

Sorry for leading you on this chase.

> which is not a configuration file.

When you create the initrd, does the initrd have your multipathd and
its rules?


>
> What distribution are you using ?

RHEL, FC, Ubuntu, Debian and SLES. But I haven't dug in the initrd of Debian/Ubuntu
so I can't much help there :-(

>
> Thomas
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

Thomas Witzel [witzel.thomas@gmail.com] wrote:
> > Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> > what you are paying - it isn\'t license fee but support fee.
>
> Thats not what I meant. I know that my hardware config does work out
> of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
> I have not been able to get any of this to work for 3 weeks now and I
> have not been able to get help/support on the issue, so I was asking
> whether there is a different free distribution that won\'t put me
> through these time consuming google/mailing list/debug orgies to get
> some storage mounted.
>
> Thomas

Maybe CentOS ?

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

configure MD3000 episode 2
 

On Tue, Mar 24, 2009 at 10:24:34AM -0400, Thomas Witzel wrote:
> > Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> > what you are paying - it isn\'t license fee but support fee.
>
> Thats not what I meant. I know that my hardware config does work out
> of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
> I have not been able to get any of this to work for 3 weeks now and I
> have not been able to get help/support on the issue, so I was asking
> whether there is a different free distribution that won\'t put me
> through these time consuming google/mailing list/debug orgies to get
> some storage mounted.

Since you say that RHEL works, you could use the clone of RHEL - CentOS.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel