<html><head><style type='text/css'>p { margin: 0; }</style><style type='text/css'>body { font-family: 'Times New Roman'; font-size: 12pt; color: #000000}</style></head><body>OK. Thanks for the debug instructions. Will wait for the next event and report back.<br>Regards<br><br>----- Original Message -----<br>From: "Lars Ellenberg" <lars.ellenberg@linbit.com><br>To: drbd-user@lists.linbit.com<br>Sent: Thursday, 7 August, 2008 11:05:51 AM GMT +00:00 GMT Britain, Ireland, Portugal<br>Subject: Re: [DRBD-user] DRBD hangs Xen VMs and won't disconnect without pulling plug<br><br>On Thu, Aug 07, 2008 at 10:21:01AM +0100, simon@onepointltd.com wrote:<br>> Would appreciate some help debugging this problem, and hopefully solving it.<br>> <br>> I am running Paravirutalized 64-bit CentOS 5.x VMs on 64-bit CentOS 5.x Dom0 on<br>> DRBD partitions shared between two Dell 2590s. The DRBD connections are shared<br>> between two dedicated GB network ports using crossover cables. The DRBD<br>> partitions are logical volumes used as virtual disks for the actual VMs and as<br>> mounted pre-formated ext3 partitions for their data partitions.<br>> <br>> Occationally, the VMs will lock up, usually (I think) unable to access their<br>> data partition. In this fault condition "drbdadm disconnect <resourcename>"<br>> times out on both nodes. I can only resolve the situation by breaking the<br>> network connection with an "ifdown ethn" command. The VM is then able to carry<br>> on working and I can reconnect DRBD and carry on.<br>> <br>> Under fault condition I have had a VM where I could still log in via SSH but<br>> not able to access the data partition and another case this morning where SSH<br>> was not working. So I am not 100% sure yet if it is solely the data partitions<br>> of the VMs that is the problem.<br>> <br>> I can't see anything strange in /var/log/messages other than the expected<br>> time-outs that occur when I disconnect the network.<br>> <br>> Running kernel on both Dom0 machines is 2.6.18-92.1.6.el5xen.<br>> DRBD rpms are<br>> kmod-drbd82-xen-8.2.6-1.2.6.18_92.1.6.el5<br>> drbd82-8.2.6-1.el5.centos<br>> <br>> Here is a sample VM and it's drbd.conf entries. Although I am allowing dual<br>> primary, this mode is not normally used. This is for live migrating VMs as a<br>> (currently) manual operation from one machine to the other.<br><br><br>get the cluster into that situation again.<br>log in on both Dom0 where DRBD is running<br>try to figure out what is going on using<br><br> top, netstat, vmstat, free,<br> dmesg<br> watch -n1 cat /proc/drbd<br> cat /proc/meminfo<br> ps -eo pid,state,wchan:30,cmd | grep -e drbd -e D<br> ...<br><br><br>-- <br>: Lars Ellenberg http://www.linbit.com :<br>: DRBD/HA support and consulting sales at linbit.com :<br>: LINBIT Information Technologies GmbH Tel +43-1-8178292-0 :<br>: Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 :<br>__<br>please don't Cc me, but send to list -- I'm subscribed<br>_______________________________________________<br>drbd-user mailing list<br>drbd-user@lists.linbit.com<br>http://lists.linbit.com/mailman/listinfo/drbd-user<br></body></html>