May be a error in drdb you have many sync in drdb :) As I read see that you make this configuration.<br><br><a href="http://www.drbd.org/users-guide-8.3/s-nested-lvm.html">http://www.drbd.org/users-guide-8.3/s-nested-lvm.html</a><br>
<br>Did you make any configuration changes? or kernel changes?...<br><br>the fail comes from kvm did you make any upgrade recently??<br><br><br><div class="gmail_quote">On Mon, Feb 13, 2012 at 11:13 PM, Andreas Bauer <span dir="ltr"><<a href="mailto:ab@voltage.de">ab@voltage.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">> Can you describe to us the configuration.. were is the Software RAID1? if you<br>
> can send the mount points.. What filesystem?<br>
<br>
</div>The RAID:<br>
<br>
root@vm-master:~# cat /proc/mdstat<br>
Personalities : [raid1]<br>
md0 : active raid1 sda1[0] sdb1[1]<br>
<a href="tel:976759672" value="+34976759672">976759672</a> blocks super 1.2 [2/2] [UU]<br>
<br>
md1 : active raid1 sdc1[0] sdd1[12]<br>
<a href="tel:976759672" value="+34976759672">976759672</a> blocks super 1.2 [2/2] [UU]<br>
<br>
The LVM:<br>
<br>
root@vm-master:~# pvs<br>
PV VG Fmt Attr PSize PFree<br>
/dev/md0 vg lvm2 a-- 931,51g 385,30g<br>
/dev/md1 vg lvm2 a-- 931,51g 450,25g<br>
<br>
<br>
On top of LVM sits the DRBD, and on top of DRBD is KVM (no filesystem):<br>
<br>
<disk type='file' device='disk'><br>
<source file='/dev/drbd6'/><br>
<driver name="qemu" type="raw" io="native" cache="none" /><br>
<target dev='vda' bus='virtio' /><br>
</disk><br>
<br>
DRBD:<br>
<br>
root@vm-master:~# cat /proc/drbd<br>
version: 8.3.11 (api:88/proto:86-96)<br>
srcversion: 21CA73FE6D7D9C67B0C6AB2<br>
<br>
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:521268 nr:347036 dw:348112 dr:1007984 al:3 bm:49 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
2: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:53248 nr:472 dw:472 dr:54480 al:0 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
3: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----<br>
ns:1933528 nr:0 dw:1413336 dr:744511 al:132 bm:73 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
4: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----<br>
ns:0 nr:0 dw:0 dr:1104 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
5: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----<br>
ns:0 nr:0 dw:0 dr:1104 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
6: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:520192 nr:19824040 dw:19824040 dr:520192 al:0 bm:82 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
7: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:520192 nr:10933108 dw:10933108 dr:520192 al:0 bm:76 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
8: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:282624 nr:1518364 dw:1518364 dr:282624 al:0 bm:36 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
9: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:0 nr:2262112 dw:2262112 dr:0 al:0 bm:82 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
<br>
20: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:0 nr:89151580 dw:243215196 dr:1987664 al:692 bm:14963 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
21: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r-----<br>
ns:999612 nr:88 dw:479508 dr:2799008 al:330 bm:78 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
22: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:520192 nr:4488702 dw:4488702 dr:520192 al:0 bm:101 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
23: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate A r-----<br>
ns:24576 nr:123638 dw:123638 dr:24576 al:0 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br>
<br>
<br>
Small error below, it is of course DRBD 8.3.11 (not 8.4.11).<br>
<br>
Again, the error occured when a RAID verify and an early morning backup in one of the virtual machines occured at the same time. As I understand, the error itself just says that a task was blocked for more than 2 minutes. The strange thing is that this situation did not recover by itself, but completely blocked *ALL* virtual machines.<br>
<br>
That means, shortly afterwards all virtual machines were completely unresponsive (network, console, ...), and DRBD was unresponsive as well. For example "fdisk /dev/drbd20" would block.<br>
<br>
On shutdown there supposedly was an error that the drbd module could not be unloaded (I was only remote so cannot tell for sure).<br>
<br>
BTW:<br>
<br>
root@vm-master:/sys/class/block# for i in * ;<br>
> do<br>
> echo -n $i scheduler: ; cat $i/queue/scheduler<br>
> done<br>
dm-0 scheduler:none<br>
dm-1 scheduler:none<br>
dm-11 scheduler:none<br>
dm-14 scheduler:none<br>
dm-15 scheduler:none<br>
dm-16 scheduler:none<br>
dm-17 scheduler:none<br>
dm-18 scheduler:none<br>
dm-19 scheduler:none<br>
dm-2 scheduler:none<br>
dm-20 scheduler:none<br>
dm-21 scheduler:none<br>
dm-22 scheduler:none<br>
dm-23 scheduler:none<br>
dm-24 scheduler:none<br>
dm-25 scheduler:none<br>
dm-26 scheduler:none<br>
dm-27 scheduler:none<br>
dm-28 scheduler:none<br>
dm-29 scheduler:none<br>
dm-3 scheduler:none<br>
dm-30 scheduler:none<br>
dm-31 scheduler:none<br>
dm-4 scheduler:none<br>
dm-5 scheduler:none<br>
dm-6 scheduler:none<br>
dm-7 scheduler:none<br>
dm-8 scheduler:none<br>
dm-9 scheduler:none<br>
drbd1 scheduler:none<br>
drbd2 scheduler:none<br>
drbd20 scheduler:none<br>
drbd21 scheduler:none<br>
drbd22 scheduler:none<br>
drbd23 scheduler:none<br>
drbd3 scheduler:none<br>
drbd4 scheduler:none<br>
drbd5 scheduler:none<br>
drbd6 scheduler:none<br>
drbd7 scheduler:none<br>
drbd8 scheduler:none<br>
drbd9 scheduler:none<br>
loop0 scheduler:none<br>
loop1 scheduler:none<br>
loop2 scheduler:none<br>
loop3 scheduler:none<br>
loop4 scheduler:none<br>
loop5 scheduler:none<br>
loop6 scheduler:none<br>
loop7 scheduler:none<br>
md0 scheduler:none<br>
md1 scheduler:none<br>
sda scheduler:noop [deadline] cfq<br>
sdb scheduler:noop [deadline] cfq<br>
sdc scheduler:noop [deadline] cfq<br>
sdd scheduler:noop [deadline] cfq<br>
sr0 scheduler:noop deadline [cfq]<br>
<br>
Any ideas what happened here and how to avoid that in the future?<br>
<br>
Thanks!<br>
<div class="HOEnZb"><div class="h5"><br>
> Kernel 3.1.0 / DRBD 8.4.11<br>
><br>
> KVM virtual machines, running on top of<br>
> DRBD (protocol A, sndbuf-size 10M, data-integrity-alg md5, external meta data,<br>
> meta data on different physical disk), running on top of<br>
> LVM2, running on top of<br>
> Software RAID 1<br>
><br>
> This night there was a MDADM verify run which was still ongoing this morning,<br>
> when this happened:<br>
><br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546758] INFO: task kvm:21152 blocked<br>
> for more than 120 seconds.<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546813] "echo 0 ><br>
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546896] kvm D<br>
> ffff88067fc92f40 0 21152 1 0x00000000<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546901] ffff8806633a4fa0<br>
> 0000000000000082 ffff880600000000 ffff8806668ec0c0<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546904] 0000000000012f40<br>
> ffff88017ae01fd8 ffff88017ae01fd8 ffff8806633a4fa0<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546908] ffff88017ae01fd8<br>
> 0000000181070175 0000000000000046 ffff8806633225c0<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546911] Call Trace:<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546925] [<ffffffffa00939d1>] ?<br>
> wait_barrier+0x87/0xc0 [raid1]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546931] [<ffffffff8103e3ca>] ?<br>
> try_to_wake_up+0x192/0x192<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546935] [<ffffffffa0096831>] ?<br>
> make_request+0x111/0x9c8 [raid1]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546941] [<ffffffffa0102c76>] ?<br>
> clone_bio+0x43/0xcb [dm_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546946] [<ffffffffa010386f>] ?<br>
> __split_and_process_bio+0x4f4/0x506 [dm_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546952] [<ffffffffa00edd93>] ?<br>
> md_make_request+0xce/0x1c3 [md_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546956] [<ffffffff81188919>] ?<br>
> generic_make_request+0x270/0x2ea<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546960] [<ffffffff81188a66>] ?<br>
> submit_bio+0xd3/0xf1<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546964] [<ffffffff81119181>] ?<br>
> __bio_add_page.part.12+0x135/0x1ed<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546967] [<ffffffff8111bbc1>] ?<br>
> dio_bio_submit+0x6c/0x8a<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546970] [<ffffffff8111becd>] ?<br>
> dio_send_cur_page+0x6e/0x91<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546972] [<ffffffff8111bfa3>] ?<br>
> submit_page_section+0xb3/0x11a<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546975] [<ffffffff8111c7d8>] ?<br>
> __blockdev_direct_IO+0x68a/0x995<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546978] [<ffffffff8111a836>] ?<br>
> blkdev_direct_IO+0x4e/0x53<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546981] [<ffffffff8111ab5b>] ?<br>
> blkdev_get_block+0x5b/0x5b<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546985] [<ffffffff810b0fdb>] ?<br>
> generic_file_direct_write+0xdc/0x146<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546987] [<ffffffff810b11d9>] ?<br>
> __generic_file_aio_write+0x194/0x278<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546992] [<ffffffff810131f1>] ?<br>
> paravirt_read_tsc+0x5/0x8<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546995] [<ffffffff810f4501>] ?<br>
> rw_copy_check_uvector+0x48/0xf8<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.546998] [<ffffffff8111ac12>] ?<br>
> bd_may_claim+0x2e/0x2e<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547000] [<ffffffff8111ac31>] ?<br>
> blkdev_aio_write+0x1f/0x61<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547003] [<ffffffff8111ac12>] ?<br>
> bd_may_claim+0x2e/0x2e<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547005] [<ffffffff811249e8>] ?<br>
> aio_rw_vect_retry+0x70/0x18e<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547008] [<ffffffff81124978>] ?<br>
> lookup_ioctx+0x53/0x53<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547010] [<ffffffff811253fe>] ?<br>
> aio_run_iocb+0x70/0x11b<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547013] [<ffffffff8112645b>] ?<br>
> do_io_submit+0x442/0x4d7<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547017] [<ffffffff81332e12>] ?<br>
> system_call_fastpath+0x16/0x1b<br>
><br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547265] INFO: task md1_resync:15384<br>
> blocked for more than 120 seconds.<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547319] "echo 0 ><br>
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547411] md1_resync D<br>
> ffff88067fc12f40 0 15384 2 0x00000000<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547414] ffff880537dc0ee0<br>
> 0000000000000046 0000000000000000 ffffffff8160d020<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547418] 0000000000012f40<br>
> ffff8801212affd8 ffff8801212affd8 ffff880537dc0ee0<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547421] 0000000000011210<br>
> ffffffff81070175 0000000000000046 ffff8806633225c0<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547424] Call Trace:<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547429] [<ffffffff81070175>] ?<br>
> arch_local_irq_save+0x11/0x17<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547433] [<ffffffffa0093917>] ?<br>
> raise_barrier+0x11a/0x14d [raid1]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547436] [<ffffffff8103e3ca>] ?<br>
> try_to_wake_up+0x192/0x192<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547440] [<ffffffffa0095fc7>] ?<br>
> sync_request+0x192/0x70b [raid1]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547446] [<ffffffffa00f147c>] ?<br>
> md_do_sync+0x760/0xb64 [md_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547450] [<ffffffff8103493d>] ?<br>
> set_task_rq+0x23/0x35<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547454] [<ffffffff8105ec6b>] ?<br>
> add_wait_queue+0x3c/0x3c<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547459] [<ffffffffa00ee28a>] ?<br>
> md_thread+0x101/0x11f [md_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547464] [<ffffffffa00ee189>] ?<br>
> md_rdev_init+0xea/0xea [md_mod]<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547467] [<ffffffff8105e625>] ?<br>
> kthread+0x76/0x7e<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547470] [<ffffffff81334f74>] ?<br>
> kernel_thread_helper+0x4/0x10<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547473] [<ffffffff8105e5af>] ?<br>
> kthread_worker_fn+0x139/0x139<br>
> Feb 12 07:03:27 vm-master kernel: [2009644.547475] [<ffffffff81334f70>] ?<br>
> gs_change+0x13/0x13<br>
><br>
> Also, at 06:50 one of the virtual machines did start to generate heavy I/O load<br>
> (backup). So this probably together with the resync run set the scene for this<br>
> to happen.<br>
><br>
> Afterwards all DRBD devices were "blocking", e.g. all virtual machines running<br>
> on these devices were unresponsive and had to be shutdown "cold".<br>
><br>
> I also tried to disconnect & reconnect to the peer the DRBD devices, but to no<br>
> avail. The server had to be restarted.<br>
><br>
> Are there any known problems with DRBD on top of a MDADM raid? I noticed that<br>
> unlike for regular access from userspace, DRBD access to the RAID seems to not<br>
> throttle the verify/rebuild run down to idle speed (1000K/s) but rather keep<br>
> going at nearly full speed ?!<br>
><br>
> Thanks, Andreas<br>
> _______________________________________________<br>
> drbd-user mailing list<br>
> <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
> <a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
><br>
><br>
><br>
</div></div></blockquote></div><br>