Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi everyone,
This is my first post to drbd-user mailing list so bear with me.
The problem I'm facing is that '$ drbdsetup /dev/drbd33 down' seemed to
deadlocked. It happened while stopping an qemu process and after that
the script should take down the drbd device. There were many drbdXX
devices created, started and syncing during this happened.
Due to huge performance issue during a "bad" reboot I needed to make
some processes wait for sync,
[ ... ]
drbdsetup 34 syncer -a 33
[ ... ]
drbdsetup 15 syncer -a 33
[ ... ]
Both 15 and 34 are fine (and everything else), UpToDate/UpToDate
I'm not sure if that command affect anything related to this issue.
I don't know if i issued this command before or after the process was
deadlocked. But it worked great for all the other drbd devices.
node1 was freshly rebooted before this problem occurred. It looks like
the clocks are off-sync while looking at /var/log/messages, I guess the
ntpd daemon were little slow.
The best solution would be if i could break the deadlock without
rebooting the machine. Any advice would be appreciated.
Here are some information regarding the setup,
Versions:
Gentoo Linux
DRBDADM_VERSION=8.3.13 (compiled from source)
linux-vanilla 3.5.3 (drbd 8.3.13)
Ganeti 2.6.0 is used for drbd and qemu management.
Block device order,
(md)raid6 -> partition -> pv / vg / lv (LVM) -> drbd -> filesystem (ntfs)
one drbd device per LV partition.
The only process accessing drbdX is qemu-kvm (1.1.2)
node2 ~ # ps aux | grep drbd33
root 11610 0.0 0.0 0 0 ? D Oct26 0:02
[drbd33_worker]
root 11625 0.2 0.0 0 0 ? D Oct26 0:59
[drbd33_receiver]
root 18497 0.0 0.0 6768 632 pts/2 S+ 00:15 0:00 grep
--colour=auto drbd33
root 22001 0.0 0.0 4280 572 ? D Oct26 0:00
drbdsetup /dev/drbd33 down
node1 ~ # ps aux | grep drbd33
root 1281 0.0 0.0 6744 616 pts/0 S+ 00:16 0:00 grep
--colour=auto drbd33
node2 ~ # cat /proc/drbd | grep ^33 -A1
33: cs:StandAlone ro:Secondary/Unknown ds:Inconsistent/DUnknown r-----
ns:0 nr:1057140 dw:1057140 dr:0 al:0 bm:188 lo:3 pe:0 ua:0 ap:0 ep:1
wo:f oos:104857600
node1 ~ # cat /proc/drbd | grep ^33 -A1
33: cs:Unconfigured
node2 ~ # cat /var/log/messages | grep drbd33
[ ... ] Was to much output so I posted it on pastebin.com,
http://pastebin.com/8aZZhNqz
node1 ~ # cat /var/log/messages | grep drbd33
[ ... ] Was to much output so I posted it on pastebin.com,
http://pastebin.com/RAuvK6eu
Ganeti log command history,
node2,
#####
node-daemon.log:2012-10-26 17:37:45,612: ganeti-noded pid=27683 INFO
RunCmd drbdsetup /dev/drbd33 down
node-daemon.log:2012-10-26 20:16:53,429: ganeti-noded pid=11604 INFO
RunCmd drbdsetup /dev/drbd33 disk
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_data
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_meta 0 -e detach
--create-device -d 102400m
node-daemon.log:2012-10-26 20:16:54,252: ganeti-noded pid=11604 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 20:16:54,256: ganeti-noded pid=11604 INFO
RunCmd drbdsetup /dev/drbd33 net ipv4:10.25.63.20:11078
ipv4:10.25.63.19:11078 C -A discard-zero-changes -B consensus
--create-device -a md5 -x XXXXXXXXX
node-daemon.log:2012-10-26 20:16:54,262: ganeti-noded pid=11604 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 20:16:54,366: ganeti-noded pid=11604 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 19:12:29,277: ganeti-noded pid=28939 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 19:12:29,384: ganeti-noded pid=28939 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 20:30:55,401: ganeti-noded pid=21936 INFO
RunCmd drbdsetup /dev/drbd33 down
node1,
#####
node-daemon.log:2012-10-26 17:37:47,263: ganeti-noded pid=7280 INFO
RunCmd drbdsetup /dev/drbd33 down
node-daemon.log:2012-10-26 18:19:06,159: ganeti-noded pid=12291 INFO
RunCmd drbdsetup /dev/drbd33 disk
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_data
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_meta 0 -e detach
--create-device -d 102400m
node-daemon.log:2012-10-26 18:19:06,260: ganeti-noded pid=12291 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 18:19:06,266: ganeti-noded pid=12291 INFO
RunCmd drbdsetup /dev/drbd33 net ipv4:10.25.63.19:11078
ipv4:10.25.63.20:11078 C -A discard-zero-changes -B consensus
--create-device -a md5 -x XXXXXXXXX
node-daemon.log:2012-10-26 18:19:06,272: ganeti-noded pid=12291 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 18:19:06,366: ganeti-noded pid=12291 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 18:19:08,959: ganeti-noded pid=12346 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 18:19:09,054: ganeti-noded pid=12346 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 18:19:09,060: ganeti-noded pid=12346 INFO
RunCmd drbdsetup /dev/drbd33 primary
node-daemon.log:2012-10-26 19:12:29,911: ganeti-noded pid=13172 INFO
RunCmd drbdsetup /dev/drbd33 disk
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_data
/dev/raid6vg/51d9cdf9-4612-444d-918a-5cc446c3ce8f.disk0_meta 0 -e detach
--create-device -d 102400m
node-daemon.log:2012-10-26 19:12:30,014: ganeti-noded pid=13172 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 19:12:30,024: ganeti-noded pid=13172 INFO
RunCmd drbdsetup /dev/drbd33 net ipv4:10.25.63.19:11078
ipv4:10.25.63.20:11078 C -A discard-zero-changes -B consensus
--create-device -a md5 -x XXXXXXXXX
node-daemon.log:2012-10-26 19:12:30,032: ganeti-noded pid=13172 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 19:12:30,128: ganeti-noded pid=13172 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 19:12:32,444: ganeti-noded pid=13241 INFO
RunCmd drbdsetup /dev/drbd33 show
node-daemon.log:2012-10-26 19:12:32,544: ganeti-noded pid=13241 INFO
RunCmd drbdsetup /dev/drbd33 syncer -r 61440 --create-device
node-daemon.log:2012-10-26 19:12:33,998: ganeti-noded pid=13241 INFO
RunCmd drbdsetup /dev/drbd33 primary
node-daemon.log:2012-10-26 19:15:45,933: ganeti-noded pid=13770 INFO
RunCmd drbdsetup /dev/drbd33 primary
node-daemon.log:2012-10-26 20:45:44,395: ganeti-noded pid=19321 INFO
RunCmd drbdsetup /dev/drbd33 down
Thanks for taking your time reading this.
Best regards,
Marcus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 262 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20121026/a97adb89/attachment.pgp>