Hi all,

I am using fence-virt.
When using a serial listener, fence_virtd does not receive a request
from VM which started after fence_virtd start.

Step 1) Start fence_virtd.
# fence_virtd -d5 -F
Step 2) And then start VM.
# virsh start srv-a1
Step 3) A request sent from the VM,
# fence_virt -D/dev/ttyS1 -oXXX
but fence_virtd not receive it.

Below, x3650f is a host, srv-XX is a VM.

[root@x3650f ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)
[root@x3650f ~]#
[root@x3650f ~]# rpm -qa | egrep '^libvirt|^qemu' | sort
libvirt-0.8.7-18.el6.x86_64
libvirt-client-0.8.7-18.el6.x86_64
libvirt-devel-0.8.7-18.el6.x86_64
libvirt-java-0.4.7-1.el6.noarch
libvirt-java-devel-0.4.7-1.el6.noarch
libvirt-python-0.8.7-18.el6.x86_64
libvirt-qpid-0.2.22-6.el6.x86_64
qemu-img-0.12.1.2-2.160.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64
[root@x3650f ~]#

And I used following fence-virt.
http://fence-virt.git.sourceforge.net/git/gitweb.cgi?p=fence-virt/fence-virt;a=commit;h=53a2b293d12991beb6c2927f78586257bf f5173a

[root@x3650f ~]# tar xfz fence-virt-53a2b29.tar.gz && cd fence-virt-53a2b29 && ./build
[root@x3650f ~]# cd && mkdir plugins && cd plugins
[root@x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/serial.so .
[root@x3650f plugins]# ln -s /root/fence-virt-53a2b29/server/libvirt.so .
[root@x3650f plugins]# ls -l
lrwxrwxrwx 1 root root 42 Sep 13 10:28 libvirt.so -> /root/fence-virt-53a2b29/server/libvirt.so
lrwxrwxrwx 1 root root 41 Sep 13 10:28 serial.so -> /root/fence-virt-53a2b29/server/serial.so
[root@x3650f plugins]#

I attached /etc/fence_virt.conf and /etc/libvirt/qemu/srv-a1.xml,
please refer to it.

Step 1)
Start fence_virtd.

[root@x3650f ~]# ./fence-virt-53a2b29/server/fence_virtd -d5 -F
Background mode disabled
Debugging threshold is now 5
(snip)
Backend plugin: libvirt
Listener plugin: serial
Searching /root/plugins for plugins...
Searching for plugins in /root/plugins
Loading plugin from /root/plugins/serial.so
Failed to map backend_plugin_version
Registered listener plugin serial 0.4
Loading plugin from /root/plugins/libvirt.so
Registered backend plugin libvirt 0.1
2 plugins found
Available backends:
libvirt 0.1
Available listeners:
serial 0.4
Debugging threshold is now 5
Using qemu:///system
Debugging threshold is now 5
Got /var/lib/libvirt/qemu for uri
Got serial for mode
Libvirt event listener starting
* Socket path: /var/lib/libvirt/qemu
* Mode: Serial
myEventAddHandleFunc:128: Add handle 6 1 0x37558aef80 0x7f68cc000920 (nil)
myEventAddTimeoutFunc:160: Adding Timeout -1 0x37558a7f90 0x7f68cc000920
event_thread:448 :: Registering domain event cbs
myEventUpdateHandleFunc:140: Updated Handle 0 0
myEventUpdateHandleFunc:140: Updated Handle 0 1
myEventUpdateHandleFunc:140: Updated Handle 0 0
myEventUpdateHandleFunc:140: Updated Handle 0 1
<- Step 2) I started VM here.
myEventUpdateHandleFunc:140: Updated Handle 0 0
myEventUpdateHandleFunc:140: Updated Handle 0 1
Registered 621d8c61-1070-7aab-6158-3889d68470ab on 9
<- Step 3) I run fence_virt here.
(snip)

Step 3)
A request sent from the VM, but fence_virtd not receive it.

[root@srv-a1 ~]# ./fence-virt-53a2b29/client/fence_virt -D/dev/ttyS1 -olist
Unknown response (255)
[root@srv-a1 ~]#

I attached the patch for this problem.

* Modified to send a SIGHUP to the primary thread from the secondary
thread detects the start of the VM, and
the primary thread is modified so as to select() the requests from
the VM which started.
* Also, when four VM(s) are started simultaneously,
poll() <in event_thread() at server/virt-serial.c> could not handle
all the boot.
Therefore, I revised it to call domainStarted()/domainStopped() in
myDomainEventCallback1().

Regards,
Kazunori INOUE
diff -urN fence-virt-53a2b29/server/daemon_init.c mod/server/daemon_init.c
--- fence-virt-53a2b29/server/daemon_init.c 2011-08-11 09:53:47.000000000 +0900
+++ mod/server/daemon_init.c 2011-09-12 11:12:12.000000000 +0900
@@ -193,6 +193,8 @@
sigdelset(&set, SIGQUIT);
sigdelset(&set, SIGCHLD);

+ sigdelset(&set, SIGHUP);
+
return (sigprocmask(SIG_BLOCK, &set, NULL));
}

diff -urN fence-virt-53a2b29/server/main.c mod/server/main.c
--- fence-virt-53a2b29/server/main.c 2011-08-11 09:53:47.000000000 +0900
+++ mod/server/main.c 2011-09-12 11:14:33.000000000 +0900
@@ -36,7 +36,8 @@
void
exit_handler(int sig)
{
- run = 0;
+ if (sig != SIGHUP)
+ run = 0;
}


@@ -188,6 +189,7 @@
signal(SIGINT, exit_handler);
signal(SIGTERM, exit_handler);
signal(SIGQUIT, exit_handler);
+ signal(SIGHUP, exit_handler);

while (p->init(&backend_ctx, config) < 0) {
if (!wait_for_backend) {
diff -urN fence-virt-53a2b29/server/serial.c mod/server/serial.c
--- fence-virt-53a2b29/server/serial.c 2011-08-11 09:53:47.000000000 +0900
+++ mod/server/serial.c 2011-09-12 12:01:17.000000000 +0900
@@ -266,6 +266,8 @@

n = select(max+1, &rfds, NULL, NULL, timeout);
if (n < 0) {
+ if (errno == EINTR)
+ return 0;
perror("select");
return n;
}
diff -urN fence-virt-53a2b29/server/virt-serial.c mod/server/virt-serial.c
--- fence-virt-53a2b29/server/virt-serial.c 2011-08-11 09:53:47.000000000 +0900
+++ mod/server/virt-serial.c 2011-09-12 11:10:58.000000000 +0900
@@ -69,25 +69,6 @@
};


-int
-myDomainEventCallback1(virConnectPtr conn,
- virDomainPtr dom, int event, int detail, void *opaque)
-{
- struct domain_info *dinfo = (struct domain_info *) opaque;
-
- if (event == VIR_DOMAIN_EVENT_STARTED ||
- event == VIR_DOMAIN_EVENT_STOPPED) {
- virDomainRef(dom);
- dinfo->dom = dom;
- dinfo->event = event;
- } else {
- dinfo->event = VIR_DOMAIN_EVENT_UNDEFINED;
- }
-
- return 0;
-}
-
-
/* EventImpl Functions */
int
myEventHandleTypeToPollEvent(virEventHandleType events)
@@ -413,15 +394,38 @@
char *uri;
char *path;
int mode;
+ pthread_t p_tid;
};


+int
+myDomainEventCallback1(virConnectPtr conn,
+ virDomainPtr dom, int event, int detail, void *opaque)
+{
+ struct event_args *args = (struct event_args *)opaque;
+
+ if (event == VIR_DOMAIN_EVENT_STARTED ||
+ event == VIR_DOMAIN_EVENT_STOPPED) {
+ virDomainRef(dom);
+ if (event == VIR_DOMAIN_EVENT_STARTED) {
+ domainStarted(dom, args->path, args->mode);
+ virDomainFree(dom);
+ pthread_kill(args->p_tid, SIGHUP);
+ } else if (event == VIR_DOMAIN_EVENT_STOPPED) {
+ domainStopped(dom);
+ virDomainFree(dom);
+ }
+ }
+
+ return 0;
+}
+
+
static void *
event_thread(void *arg)
{
struct event_args *args = (struct event_args *)arg;
virConnectPtr dconn = NULL;
- struct domain_info dinfo;
int callback1ret = -1;
int sts;

@@ -450,11 +454,9 @@
registerExisting(dconn, args->path, args->mode);

/* Add 2 callbacks to prove this works with more than just one */
- memset(&dinfo, 0, sizeof (dinfo));
- dinfo.event = VIR_DOMAIN_EVENT_UNDEFINED;
callback1ret =
virConnectDomainEventRegister(dconn, myDomainEventCallback1,
- &dinfo, NULL);
+ arg, NULL);

if ((callback1ret == 0)) {
while (run) {
@@ -469,18 +471,6 @@
t_cb(t_timeout, t_opaque);
}

- if (dinfo.event == VIR_DOMAIN_EVENT_STARTED) {
- domainStarted(dinfo.dom, args->path, args->mode);
- virDomainFree(dinfo.dom);
- dinfo.dom = NULL;
- dinfo.event = VIR_DOMAIN_EVENT_UNDEFINED;
- } else if (dinfo.event == VIR_DOMAIN_EVENT_STOPPED) {
- domainStopped(dinfo.dom);
- virDomainFree(dinfo.dom);
- dinfo.dom = NULL;
- dinfo.event = VIR_DOMAIN_EVENT_UNDEFINED;
- }
-
if (sts == 0) {
/* DEBUG0("Poll timeout"); */
continue;
@@ -546,6 +536,7 @@
}

args->mode = mode;
+ args->p_tid = pthread_self();
run = 1;

return pthread_create(&event_tid, NULL, event_thread, args);
fence_virtd {
listener = "serial";
backend = "libvirt";
module_path = "/root/plugins-53a2b29";
}

listeners {
serial {
mode = "serial";
path = "/var/lib/libvirt/qemu";
}
}

backends {
libvirt {
uri = "qemu:///system";
}
}

groups {
group {
uuid = "srv-a1";
uuid = "srv-a2";
uuid = "621d8c61-1070-7aab-6158-3889d68470ab";
uuid = "4f0ed127-c8a3-4085-a0d0-a93231924202";
}
group {
uuid = "srv-b1";
uuid = "srv-b2";
uuid = "d55103a7-eea7-ad9e-3ee9-8aa660afbe8d";
uuid = "7243da51-2bda-413e-f172-4ed8cfee4297";
}
}

(2011/10/26 8:30), Lon Hohberger wrote:
> On 09/19/2011 04:32 PM, Lon Hohberger wrote:
>> On 09/13/2011 04:44 AM, Kazunori INOUE wrote:
>>
>>> I attached the patch for this problem.
>>>
>>> * Modified to send a SIGHUP to the primary thread from the secondary
>>> thread detects the start of the VM, and
>>> the primary thread is modified so as to select() the requests from
>>> the VM which started.
>>
>> That's certainly one way of doing it. I might have used
>> 'pthread_cond_singal'.
>
> I am sorry for the delay.
>
> I have a fix based on yours that uses a pipe to wake up select() rather
> than pthread_kill; I'll attach it here. This ends up being a bit lower
> impact, as well; it doesn't require changing the main program - only the
> serial/vmchannel plugin.
>
> Apart from that, the patch here is largely the same.
>
> Let me know what you think at your convenience.
>
> -- Lon

Hi, Lon

Thank you for your reply.
I tested this patch, and confirmed that the problem was solved.
(It was good also when four VM (s) was started simultaneously.)

Many thanks!

(11.10.27 16:23), Kazunori INOUE wrote:
> (2011/10/26 8:30), Lon Hohberger wrote:
>> On 09/19/2011 04:32 PM, Lon Hohberger wrote:
>>> On 09/13/2011 04:44 AM, Kazunori INOUE wrote:
>>>
>>>> I attached the patch for this problem.
>>>>
>>>> * Modified to send a SIGHUP to the primary thread from the secondary
>>>> thread detects the start of the VM, and
>>>> the primary thread is modified so as to select() the requests from
>>>> the VM which started.
>>>
>>> That's certainly one way of doing it. I might have used
>>> 'pthread_cond_singal'.
>>
>> I am sorry for the delay.
>>
>> I have a fix based on yours that uses a pipe to wake up select() rather
>> than pthread_kill; I'll attach it here. This ends up being a bit lower
>> impact, as well; it doesn't require changing the main program - only the
>> serial/vmchannel plugin.
>>
>> Apart from that, the patch here is largely the same.
>>
>> Let me know what you think at your convenience.
>>
>> -- Lon
>
> Hi, Lon
>
> Thank you for your reply.
> I tested this patch, and confirmed that the problem was solved.
> (It was good also when four VM (s) was started simultaneously.)
>
> Many thanks!

Hi,

Could you please merge?

Regards,
Kazunori INOUE

On 09/13/2011 04:44 AM, Kazunori INOUE wrote:

[all late and wrong]

Merged in to 0.3.0

-- Lon