<div dir="ltr">there are some error messages in dmesg:<div><br></div><div><div>INFO: task flush-253:5:32096 blocked for more than 120 seconds.</div><div>&quot;echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs&quot; disables this message.</div>
<div>flush-253:5   D 0000000000000002     0 32096      2 0x00000080</div><div> ffff8803b1e339b0 0000000000000046 0000000000000000 ffff88002c292920</div><div> ffff8803b1e33960 ffffffff81127e77 0000000300000001 0000000000000002</div>
<div> ffff8803a0911038 ffff8803b1e33fd8 000000000000f4e8 ffff8803a0911038</div><div>Call Trace:</div><div> [&lt;ffffffff81127e77&gt;] ? ____pagevec_lru_add+0x167/0x180</div><div> [&lt;ffffffffa0168f6a&gt;] start_this_handle+0x25a/0x4d0 [jbd2]</div>
<div> [&lt;ffffffff8115e5db&gt;] ? cache_alloc_refill+0x15b/0x240</div><div> [&lt;ffffffff81090bf0&gt;] ? autoremove_wake_function+0x0/0x40</div><div> [&lt;ffffffffa01693c5&gt;] jbd2_journal_start+0xb5/0x100 [jbd2]</div><div>
 [&lt;ffffffffa018b035&gt;] ? ext4_meta_trans_blocks+0x75/0xf0 [ext4]</div><div> [&lt;ffffffffa01a5218&gt;] ext4_journal_start_sb+0x58/0x90 [ext4]</div><div> [&lt;ffffffffa018eebc&gt;] ext4_da_writepages+0x27c/0x660 [ext4]</div>
<div> [&lt;ffffffff81054754&gt;] ? find_busiest_group+0x244/0xb20</div><div> [&lt;ffffffff81126351&gt;] do_writepages+0x21/0x40</div><div> [&lt;ffffffff811a046d&gt;] writeback_single_inode+0xdd/0x2c0</div><div> [&lt;ffffffff811a08ae&gt;] writeback_sb_inodes+0xce/0x180</div>
<div> [&lt;ffffffff811a0a0b&gt;] writeback_inodes_wb+0xab/0x1b0</div><div> [&lt;ffffffff811a0dab&gt;] wb_writeback+0x29b/0x3f0</div><div> [&lt;ffffffff814eca40&gt;] ? thread_return+0x4e/0x77e</div><div> [&lt;ffffffff8107cc02&gt;] ? del_timer_sync+0x22/0x30</div>
<div> [&lt;ffffffff811a1099&gt;] wb_do_writeback+0x199/0x240</div><div> [&lt;ffffffff811a11a3&gt;] bdi_writeback_task+0x63/0x1b0</div><div> [&lt;ffffffff81090ab7&gt;] ? bit_waitqueue+0x17/0xd0</div><div> [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100</div>
<div> [&lt;ffffffff81134dc6&gt;] bdi_start_fn+0x86/0x100</div><div> [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100</div><div> [&lt;ffffffff81090886&gt;] kthread+0x96/0xa0</div><div> [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20</div>
<div> [&lt;ffffffff810907f0&gt;] ? kthread+0x0/0xa0</div><div> [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20</div><div>INFO: task jbd2/dm-5-8:10717 blocked for more than 120 seconds.</div><div>&quot;echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs&quot; disables this message.</div>
<div>jbd2/dm-5-8   D 0000000000000006     0 10717      2 0x00000080</div><div> ffff880391099d20 0000000000000046 0000000000015fc0 0000000000015fc0</div><div> ffff88002c28fda0 0000000000015fc0 0000000000015fc0 ffff880420cd2b00</div>
<div> ffff880420cd30b8 ffff880391099fd8 000000000000f4e8 ffff880420cd30b8</div><div>Call Trace:</div><div> [&lt;ffffffff81090ede&gt;] ? prepare_to_wait+0x4e/0x80</div><div> [&lt;ffffffffa016a80f&gt;] jbd2_journal_commit_transaction+0x19f/0x14b0 [jbd2]</div>
<div> [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320</div><div> [&lt;ffffffff8107c0ec&gt;] ? lock_timer_base+0x3c/0x70</div><div> [&lt;ffffffff81090bf0&gt;] ? autoremove_wake_function+0x0/0x40</div><div> [&lt;ffffffffa0170928&gt;] kjournald2+0xb8/0x220 [jbd2]</div>
<div> [&lt;ffffffff81090bf0&gt;] ? autoremove_wake_function+0x0/0x40</div><div> [&lt;ffffffffa0170870&gt;] ? kjournald2+0x0/0x220 [jbd2]</div><div> [&lt;ffffffff81090886&gt;] kthread+0x96/0xa0</div><div> [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20</div>
<div> [&lt;ffffffff810907f0&gt;] ? kthread+0x0/0xa0</div><div> [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20</div><div>INFO: task flush-253:5:32096 blocked for more than 120 seconds.</div><div>&quot;echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs&quot; disables this message.</div>
<div>flush-253:5   D 0000000000000000     0 32096      2 0x00000080</div><div> ffff8803b1e338d0 0000000000000046 0000000000000000 ffffffff81012b59</div><div> ffff8803b1e33870 ffffffff8109b809 ffff8803a0911038 0000000000000286</div>
<div> ffff8803a0911038 ffff8803b1e33fd8 000000000000f4e8 ffff8803a0911038</div><div>Call Trace:</div><div> [&lt;ffffffff81012b59&gt;] ? read_tsc+0x9/0x20</div><div> [&lt;ffffffff8109b809&gt;] ? ktime_get_ts+0xa9/0xe0</div>
<div> [&lt;ffffffff81110b10&gt;] ? sync_page+0x0/0x50</div><div> [&lt;ffffffff814ed1e3&gt;] io_schedule+0x73/0xc0</div><div> [&lt;ffffffff81110b4d&gt;] sync_page+0x3d/0x50</div><div> [&lt;ffffffff814eda4a&gt;] __wait_on_bit_lock+0x5a/0xc0</div>
<div> [&lt;ffffffff81110ae7&gt;] __lock_page+0x67/0x70</div><div> [&lt;ffffffff81090c30&gt;] ? wake_bit_function+0x0/0x50</div><div> [&lt;ffffffff811271a5&gt;] ? pagevec_lookup_tag+0x25/0x40</div><div> [&lt;ffffffffa018aa16&gt;] ext4_num_dirty_pages+0x256/0x260 [ext4]</div>
<div> [&lt;ffffffffa01689f6&gt;] ? jbd2_journal_stop+0x1e6/0x2b0 [jbd2]</div><div> [&lt;ffffffffa018f1d4&gt;] ext4_da_writepages+0x594/0x660 [ext4]</div><div> [&lt;ffffffff8126a5e9&gt;] ? cpumask_next_and+0x29/0x50</div><div>
 [&lt;ffffffff81054754&gt;] ? find_busiest_group+0x244/0xb20</div><div> [&lt;ffffffff81126351&gt;] do_writepages+0x21/0x40</div><div> [&lt;ffffffff811a046d&gt;] writeback_single_inode+0xdd/0x2c0</div><div> [&lt;ffffffff811a08ae&gt;] writeback_sb_inodes+0xce/0x180</div>
<div> [&lt;ffffffff811a0a0b&gt;] writeback_inodes_wb+0xab/0x1b0</div><div> [&lt;ffffffff811a0dab&gt;] wb_writeback+0x29b/0x3f0</div><div> [&lt;ffffffff814eca40&gt;] ? thread_return+0x4e/0x77e</div><div> [&lt;ffffffff8107cc02&gt;] ? del_timer_sync+0x22/0x30</div>
<div> [&lt;ffffffff811a1099&gt;] wb_do_writeback+0x199/0x240</div><div> [&lt;ffffffff811a11a3&gt;] bdi_writeback_task+0x63/0x1b0</div><div> [&lt;ffffffff81090ab7&gt;] ? bit_waitqueue+0x17/0xd0</div><div> [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100</div>
<div> [&lt;ffffffff81134dc6&gt;] bdi_start_fn+0x86/0x100</div><div> [&lt;ffffffff81134d40&gt;] ? bdi_start_fn+0x0/0x100</div><div> [&lt;ffffffff81090886&gt;] kthread+0x96/0xa0</div><div> [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20</div>
<div> [&lt;ffffffff810907f0&gt;] ? kthread+0x0/0xa0</div><div> [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20</div><div>INFO: task jbd2/dm-5-8:10717 blocked for more than 120 seconds.</div><div>&quot;echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs&quot; disables this message.</div>
<div>jbd2/dm-5-8   D 0000000000000006     0 10717      2 0x00000080</div><div> ffff880391099d20 0000000000000046 0000000000015fc0 0000000000015fc0</div><div> ffff88002c24fda0 0000000000015fc0 0000000000015fc0 ffff880420cd2b00</div>
<div> ffff880420cd30b8 ffff880391099fd8 000000000000f4e8 ffff880420cd30b8</div><div>Call Trace:</div><div> [&lt;ffffffff81090ede&gt;] ? prepare_to_wait+0x4e/0x80</div><div> [&lt;ffffffffa016a80f&gt;] jbd2_journal_commit_transaction+0x19f/0x14b0 [jbd2]</div>
<div> [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320</div><div> [&lt;ffffffff81059c99&gt;] ? find_busiest_queue+0x69/0x150</div><div> [&lt;ffffffff8107c0ec&gt;] ? lock_timer_base+0x3c/0x70</div><div> [&lt;ffffffff81090bf0&gt;] ? autoremove_wake_function+0x0/0x40</div>
<div> [&lt;ffffffffa0170928&gt;] kjournald2+0xb8/0x220 [jbd2]</div><div> [&lt;ffffffff81090bf0&gt;] ? autoremove_wake_function+0x0/0x40</div><div> [&lt;ffffffffa0170870&gt;] ? kjournald2+0x0/0x220 [jbd2]</div><div> [&lt;ffffffff81090886&gt;] kthread+0x96/0xa0</div>
<div> [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20</div><div> [&lt;ffffffff810907f0&gt;] ? kthread+0x0/0xa0</div><div> [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20</div></div><div><br></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/5/21 Mia Lueng <span dir="ltr">&lt;<a href="mailto:xiaozunvlg@gmail.com" target="_blank">xiaozunvlg@gmail.com</a>&gt;</span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">drbd configure :<div><br></div><div><div>[root@kvm3 ~]# drbdadm dump drbd0</div><div># resource drbd0 on kvm3: not ignored, not stacked</div><div># defined at /etc/drbd.d/drbd0.res:1</div><div>resource drbd0 {</div>

<div>    on kvm3 {</div><div>        device           /dev/drbd0 minor 0;</div><div>        disk             /dev/vg_kvm3/drbd0;</div><div>        meta-disk        internal;</div><div>        address          ipv4 <a href="http://192.168.10.6:7700" target="_blank">192.168.10.6:7700</a>;</div>

<div>    }</div><div>    on kvm4 {</div><div>        device           /dev/drbd0 minor 0;</div><div>        disk             /dev/vg_kvm4/drbd0;</div><div>        meta-disk        internal;</div><div>        address          ipv4 <a href="http://192.168.10.7:7700" target="_blank">192.168.10.7:7700</a>;</div>

<div>    }</div><div>    net {</div><div>        protocol           A;</div><div>        csums-alg        md5;</div><div>        verify-alg       md5;</div><div>        ping-timeout      30;</div><div>        ping-int          30;</div>

<div>        max-epoch-size   8192;</div><div>        max-buffers      8912;</div><div>        unplug-watermark 131072;</div><div>    }</div><div>    disk {</div><div>        on-io-error      pass_on;</div><div>        disk-barrier      no;</div>

<div>        disk-flushes      no;</div><div>        resync-rate      100M;</div><div>        c-plan-ahead      20;</div><div>        c-delay-target   100;</div><div>        c-max-rate       400M;</div><div>        c-min-rate        2M;</div>

<div>        al-extents       601;</div><div>    }</div><div>}</div></div><div><br></div><div><br></div><div>I run iozone on drbd and it blocks for a long time.</div><div><br></div><div>/root/work/iozone/iozone3_414/src/current/iozone -a -i 0 -i 1 -Rb /root/drbd_100M_test.xls<br>

</div><div><br></div><div><div># cat /proc/drbd</div><div>version: 8.4.3 (api:1/proto:86-101)</div><div>GIT-hash: 2acb4389deadd704f072570faf373c835c028bb1 build by root@kvm3, 2013-05-08 23:19:24</div><div> 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate A r---n-</div>

<div>    ns:29990400 nr:0 dw:29995024 dr:1393 al:1441 bm:0 lo:0 pe:1156 ua:0 ap:1156 ep:1 wo:d oos:0</div><div><br></div><div><br></div></div><div><br></div><div><br></div></div>
</blockquote></div><br></div>