Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello list , DRBD: 8.2.6 with patch (to compile on 2.6.26): --- a/drbd/drbd_main.c +++ b/drbd/drbd_main.c @@ -2374,8 +2374,8 @@ STATIC void drbd_cleanup(void) if (minor_table) { if (drbd_proc) - remove_proc_entry("drbd",&proc_root); - i = minor_count; + remove_proc_entry("drbd", NULL); + i = minor_count; while (i--) { drbd_dev *mdev = minor_to_mdev(i); struct gendisk **disk = &mdev->vdisk; @@ -2611,7 +2611,7 @@ int __init drbd_init(void) /* * register with procfs */ - drbd_proc = create_proc_entry("drbd", S_IFREG | S_IRUGO , &proc_root); + drbd_proc = create_proc_entry("drbd", S_IFREG | S_IRUGO , NULL); if (!drbd_proc) { printk(KERN_ERR DEVICE_NAME": unable to register proc file\n"); Kernel: 2.6.26.5 When doing some I/O on the primary node (bonnie, drbd is connected) while drbd verify was running, the primary node crashed. The log shows: Sep 30 12:09:32 myserver kernel: [331031.387072] INFO: task kswapd0:228 blocked for more than 120 seconds. Sep 30 12:09:32 myserver kernel: [331031.402089] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 30 12:09:32 myserver kernel: [331031.418090] kswapd0 D 0000000000000000 0 228 2 Sep 30 12:09:32 myserver kernel: [331031.441606] ffff81012ffdf740 0000000000000046 0000000000000000 ffff81012ffdf8a8 Sep 30 12:09:32 myserver kernel: [331031.459975] ffff81012ffdf710 ffffffff806d3280 ffff81012fdc88c0 ffff81012fca6240 Sep 30 12:09:32 myserver kernel: [331031.473401] ffff81012fdc8c00 0000000104ec7c91 00000000ffffffff 0000000000000286 Sep 30 12:09:32 myserver kernel: [331031.491668] Call Trace: Sep 30 12:09:32 myserver kernel: [331031.497250] [<ffffffffa00fdbad>] :drbd:drbd_al_begin_io+0x20d/0x2e0 Sep 30 12:09:32 myserver kernel: [331031.515982] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331031.527988] [<ffffffff802b3c41>] ? __bio_clone+0x21/0x60 Sep 30 12:09:32 myserver kernel: [331031.538104] [<ffffffffa00f9d46>] :drbd:drbd_make_request_common+0xa86/0xde0 Sep 30 12:09:32 myserver kernel: [331031.552149] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331031.567599] [<ffffffffa00fa2e6>] :drbd:drbd_make_request_26+0x246/0x400 Sep 30 12:09:32 myserver kernel: [331031.579606] [<ffffffff80347834>] generic_make_request+0x1a4/0x260 Sep 30 12:09:32 myserver kernel: [331031.589847] [<ffffffff80349083>] submit_bio+0x73/0xf0 Sep 30 12:09:32 myserver kernel: [331031.601847] [<ffffffff802b3fb4>] ? __bio_add_page+0x124/0x1e0 Sep 30 12:09:32 myserver kernel: [331031.614298] [<ffffffff80332dc6>] xfs_submit_ioend_bio+0x26/0x40 Sep 30 12:09:32 myserver kernel: [331031.626531] [<ffffffff80332e9d>] xfs_submit_ioend+0x8d/0xf0 Sep 30 12:09:32 myserver kernel: [331031.638266] [<ffffffff80333de1>] xfs_page_state_convert+0x3f1/0x6b0 Sep 30 12:09:32 myserver kernel: [331031.651197] [<ffffffff803341c5>] xfs_vm_writepage+0x65/0x100 Sep 30 12:09:32 myserver kernel: [331031.665037] [<ffffffff8026fb57>] shrink_page_list+0x4e7/0x620 Sep 30 12:09:32 myserver kernel: [331031.676770] [<ffffffff802315bd>] ? hrtick_set+0xbd/0x120 Sep 30 12:09:32 myserver kernel: [331031.687719] [<ffffffff8026ec83>] ? isolate_lru_pages+0x83/0x250 Sep 30 12:09:32 myserver kernel: [331031.699723] [<ffffffff8026ec83>] ? isolate_lru_pages+0x83/0x250 Sep 30 12:09:32 myserver kernel: [331031.712229] [<ffffffff8026fde8>] shrink_inactive_list+0x158/0x410 Sep 30 12:09:32 myserver kernel: [331031.724238] [<ffffffff8027015b>] shrink_zone+0xbb/0x130 Sep 30 12:09:32 myserver kernel: [331031.735696] [<ffffffff80270c8c>] kswapd+0x51c/0x540 Sep 30 12:09:32 myserver kernel: [331031.743702] [<ffffffff8026ee50>] ? isolate_pages_global+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331031.758300] [<ffffffff804d3d5f>] ? thread_return+0x5f/0x5a0 Sep 30 12:09:32 myserver kernel: [331031.769841] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331031.782984] [<ffffffff80270770>] ? kswapd+0x0/0x540 Sep 30 12:09:32 myserver kernel: [331031.793140] [<ffffffff8024adad>] kthread+0x4d/0x80 Sep 30 12:09:32 myserver kernel: [331031.803157] [<ffffffff8020d018>] child_rip+0xa/0x12 Sep 30 12:09:32 myserver kernel: [331031.811162] [<ffffffff8024ad60>] ? kthread+0x0/0x80 Sep 30 12:09:32 myserver kernel: [331031.825333] [<ffffffff8020d00e>] ? child_rip+0x0/0x12 Sep 30 12:09:32 myserver kernel: [331031.836571] INFO: task drbd0_receiver:2173 blocked for more than 120 seconds. Sep 30 12:09:32 myserver kernel: [331031.851110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 30 12:09:32 myserver kernel: [331031.867795] drbd0_receive D ffff810115259630 0 2173 2 Sep 30 12:09:32 myserver kernel: [331031.882209] ffff8101152595c0 0000000000000046 0000000000000000 ffff810115259728 Sep 30 12:09:32 myserver kernel: [331031.899955] ffff810115259590 ffffffff806d3280 ffff81011d53a680 ffff81012fd9c740 Sep 30 12:09:32 myserver kernel: [331031.914795] ffff81011d53a9c0 0000000104ec7c15 00000000ffffffff 0000000000000282 Sep 30 12:09:32 myserver kernel: [331031.932823] Call Trace: Sep 30 12:09:32 myserver kernel: [331031.936839] [<ffffffffa00fdbad>] :drbd:drbd_al_begin_io+0x20d/0x2e0 Sep 30 12:09:32 myserver kernel: [331031.951362] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331031.963367] [<ffffffff802b3c41>] ? __bio_clone+0x21/0x60 Sep 30 12:09:32 myserver kernel: [331031.976738] [<ffffffffa00f9d46>] :drbd:drbd_make_request_common+0xa86/0xde0 Sep 30 12:09:32 myserver kernel: [331031.988745] [<ffffffff8033286b>] ? xfs_start_page_writeback+0x2b/0x50 Sep 30 12:09:32 myserver kernel: [331032.003082] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:32 myserver kernel: [331032.016193] [<ffffffffa00fa2e6>] :drbd:drbd_make_request_26+0x246/0x400 Sep 30 12:09:32 myserver kernel: [331032.032721] [<ffffffff80347834>] generic_make_request+0x1a4/0x260 Sep 30 12:09:32 myserver kernel: [331032.044727] [<ffffffff80349083>] submit_bio+0x73/0xf0 Sep 30 12:09:33 myserver kernel: [331032.057312] [<ffffffff802b3fb4>] ? __bio_add_page+0x124/0x1e0 Sep 30 12:09:33 myserver kernel: [331032.069204] [<ffffffff80332dc6>] xfs_submit_ioend_bio+0x26/0x40 Sep 30 12:09:33 myserver kernel: [331032.081478] [<ffffffff80332e9d>] xfs_submit_ioend+0x8d/0xf0 Sep 30 12:09:33 myserver kernel: [331032.090960] [<ffffffff80333de1>] xfs_page_state_convert+0x3f1/0x6b0 Sep 30 12:09:33 myserver kernel: [331032.107464] [<ffffffff803341c5>] xfs_vm_writepage+0x65/0x100 Sep 30 12:09:33 myserver kernel: [331032.119464] [<ffffffff8026fb57>] shrink_page_list+0x4e7/0x620 Sep 30 12:09:33 myserver kernel: [331032.131106] [<ffffffff8024e4a7>] ? hrtimer_start+0xe7/0x170 Sep 30 12:09:33 myserver kernel: [331032.139111] [<ffffffff8026ec83>] ? isolate_lru_pages+0x83/0x250 Sep 30 12:09:33 myserver kernel: [331032.154923] [<ffffffff8026fde8>] shrink_inactive_list+0x158/0x410 Sep 30 12:09:33 myserver kernel: [331032.166931] [<ffffffff80362067>] ? swiotlb_map_sg_attrs+0x67/0x140 Sep 30 12:09:33 myserver kernel: [331032.177271] [<ffffffff8027015b>] shrink_zone+0xbb/0x130 Sep 30 12:09:33 myserver kernel: [331032.191632] [<ffffffff8027055d>] try_to_free_pages+0x20d/0x370 Sep 30 12:09:33 myserver kernel: [331032.203248] [<ffffffff8026ee50>] ? isolate_pages_global+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331032.215255] [<ffffffff8026acc6>] __alloc_pages_internal+0x1a6/0x450 Sep 30 12:09:33 myserver kernel: [331032.226245] [<ffffffff8026af8b>] __alloc_pages+0xb/0x10 Sep 30 12:09:33 myserver kernel: [331032.239782] [<ffffffff80287c2b>] cache_alloc_refill+0x2cb/0x560 Sep 30 12:09:33 myserver kernel: [331032.251747] [<ffffffff80287fb8>] __kmalloc+0xf8/0x100 Sep 30 12:09:33 myserver kernel: [331032.259760] [<ffffffffa00f4f2c>] :drbd:receive_DataRequest+0x3cc/0x770 Sep 30 12:09:33 myserver kernel: [331032.275735] [<ffffffffa00f0e6b>] :drbd:drbdd+0x5b/0x190 Sep 30 12:09:33 myserver kernel: [331032.285325] [<ffffffffa00f4111>] :drbd:drbdd_init+0x121/0x1e0 Sep 30 12:09:33 myserver kernel: [331032.300090] [<ffffffffa0106745>] :drbd:drbd_thread_setup+0xb5/0x180 Sep 30 12:09:33 myserver kernel: [331032.313009] [<ffffffff8020d018>] child_rip+0xa/0x12 Sep 30 12:09:33 myserver kernel: [331032.327124] [<ffffffffa0106690>] ? :drbd:drbd_thread_setup+0x0/0x180 Sep 30 12:09:33 myserver kernel: [331032.339490] [<ffffffff8020d00e>] ? child_rip+0x0/0x12 Sep 30 12:09:33 myserver kernel: [331032.355127] INFO: task bonnie:3956 blocked for more than 120 seconds. Sep 30 12:09:33 myserver kernel: [331032.368245] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 30 12:09:33 myserver kernel: [331032.384182] bonnie D 0000000000000000 0 3956 4038 Sep 30 12:09:33 myserver kernel: [331032.396352] ffff810103ca7328 0000000000000086 0000000000000000 ffff810103ca7490 Sep 30 12:09:33 myserver kernel: [331032.409788] ffff810103ca72f8 ffffffff806d3280 ffff81012c4563c0 ffff81012fce2340 Sep 30 12:09:33 myserver kernel: [331032.427309] ffff81012c456700 0000000104ec7c91 00000000ffffffff 0000000000000286 Sep 30 12:09:33 myserver kernel: [331032.441308] Call Trace: Sep 30 12:09:33 myserver kernel: [331032.447992] [<ffffffffa00fdbad>] :drbd:drbd_al_begin_io+0x20d/0x2e0 Sep 30 12:09:33 myserver kernel: [331032.460133] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331032.474205] [<ffffffff802b3c41>] ? __bio_clone+0x21/0x60 Sep 30 12:09:33 myserver kernel: [331032.485236] [<ffffffffa00f9d46>] :drbd:drbd_make_request_common+0xa86/0xde0 Sep 30 12:09:33 myserver kernel: [331032.497856] [<ffffffff8033286b>] ? xfs_start_page_writeback+0x2b/0x50 Sep 30 12:09:33 myserver kernel: [331032.513513] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331032.528640] [<ffffffffa00fa2e6>] :drbd:drbd_make_request_26+0x246/0x400 Sep 30 12:09:33 myserver kernel: [331032.542252] [<ffffffff80347834>] generic_make_request+0x1a4/0x260 Sep 30 12:09:33 myserver kernel: [331032.554883] [<ffffffff80349083>] submit_bio+0x73/0xf0 Sep 30 12:09:33 myserver kernel: [331032.565384] [<ffffffff802b3fb4>] ? __bio_add_page+0x124/0x1e0 Sep 30 12:09:33 myserver kernel: [331032.577309] [<ffffffff80332dc6>] xfs_submit_ioend_bio+0x26/0x40 Sep 30 12:09:33 myserver kernel: [331032.589314] [<ffffffff80332e9d>] xfs_submit_ioend+0x8d/0xf0 Sep 30 12:09:33 myserver kernel: [331032.601126] [<ffffffff80333de1>] xfs_page_state_convert+0x3f1/0x6b0 Sep 30 12:09:33 myserver kernel: [331032.613135] [<ffffffff803341c5>] xfs_vm_writepage+0x65/0x100 Sep 30 12:09:33 myserver kernel: [331032.625808] [<ffffffff8026fb57>] shrink_page_list+0x4e7/0x620 Sep 30 12:09:33 myserver kernel: [331032.637701] [<ffffffff8026ec83>] ? isolate_lru_pages+0x83/0x250 Sep 30 12:09:33 myserver kernel: [331032.649971] [<ffffffff8026fde8>] shrink_inactive_list+0x158/0x410 Sep 30 12:09:33 myserver kernel: [331032.661980] [<ffffffff80310f3e>] ? xfs_iunlock+0x4e/0xa0 Sep 30 12:09:33 myserver kernel: [331032.673614] [<ffffffff8027015b>] shrink_zone+0xbb/0x130 Sep 30 12:09:33 myserver kernel: [331032.681619] [<ffffffff8027055d>] try_to_free_pages+0x20d/0x370 Sep 30 12:09:33 myserver kernel: [331032.696565] [<ffffffff8026ee50>] ? isolate_pages_global+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331032.708570] [<ffffffff8026acc6>] __alloc_pages_internal+0x1a6/0x450 Sep 30 12:09:33 myserver kernel: [331032.722974] [<ffffffff8026af8b>] __alloc_pages+0xb/0x10 Sep 30 12:09:33 myserver kernel: [331032.731997] [<ffffffff802655e5>] __grab_cache_page+0x65/0xa0 Sep 30 12:09:33 myserver kernel: [331032.746593] [<ffffffff802b14b4>] block_write_begin+0x84/0xe0 Sep 30 12:09:33 myserver kernel: [331032.758310] [<ffffffff803329e5>] xfs_vm_write_begin+0x25/0x30 Sep 30 12:09:33 myserver kernel: [331032.770329] [<ffffffff80333110>] ? xfs_get_blocks+0x0/0x20 Sep 30 12:09:33 myserver kernel: [331032.781694] [<ffffffff80266182>] generic_file_buffered_write+0x142/0x6b0 Sep 30 12:09:33 myserver kernel: [331032.795441] [<ffffffff8035a590>] ? __up_write+0xd0/0x130 Sep 30 12:09:33 myserver kernel: [331032.803994] [<ffffffff8033ae63>] xfs_write+0x613/0x8e0 Sep 30 12:09:33 myserver kernel: [331032.819028] [<ffffffff8033720f>] xfs_file_aio_write+0x4f/0x60 Sep 30 12:09:33 myserver kernel: [331032.829722] [<ffffffff8028ad71>] do_sync_write+0xf1/0x130 Sep 30 12:09:33 myserver kernel: [331032.839237] [<ffffffff80359d8c>] ? rb_insert_color+0xbc/0xf0 Sep 30 12:09:33 myserver kernel: [331032.852846] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331032.865702] [<ffffffff804d4d14>] ? __mutex_lock_slowpath+0x134/0x210 Sep 30 12:09:33 myserver kernel: [331032.877708] [<ffffffff8028b5f7>] vfs_write+0xc7/0x170 Sep 30 12:09:33 myserver kernel: [331032.888217] [<ffffffff8028bc90>] sys_write+0x50/0x90 Sep 30 12:09:33 myserver kernel: [331032.898548] [<ffffffff8020c1fb>] system_call_after_swapgs+0x7b/0x80 Sep 30 12:09:33 myserver kernel: [331032.915990] INFO: task pdflush:5000 blocked for more than 120 seconds. Sep 30 12:09:33 myserver kernel: [331032.926311] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 30 12:09:33 myserver kernel: [331032.945153] pdflush D ffff81010568d820 0 5000 2 Sep 30 12:09:33 myserver kernel: [331032.962223] ffff81010568d7b0 0000000000000046 0000000000000000 ffff81012ebeb080 Sep 30 12:09:33 myserver kernel: [331032.978911] ffff81012ec2c4c0 ffffffff806d3280 ffff81012dac2500 ffff81012c4563c0 Sep 30 12:09:33 myserver kernel: [331032.997128] ffff81012dac2840 0000000104ec7a2c 00000000ffffffff 0000000000000282 Sep 30 12:09:33 myserver kernel: [331033.011897] Call Trace: Sep 30 12:09:33 myserver kernel: [331033.017481] [<ffffffffa00fdbad>] :drbd:drbd_al_begin_io+0x20d/0x2e0 Sep 30 12:09:33 myserver kernel: [331033.030800] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:33 myserver kernel: [331033.043898] [<ffffffff802b3c41>] ? __bio_clone+0x21/0x60 Sep 30 12:09:33 myserver kernel: [331033.056243] [<ffffffffa00f9d46>] :drbd:drbd_make_request_common+0xa86/0xde0 Sep 30 12:09:33 myserver kernel: [331033.070240] [<ffffffff8024b1e0>] ? autoremove_wake_function+0x0/0x40 Sep 30 12:09:34 myserver kernel: [331033.085693] [<ffffffffa00fa2e6>] :drbd:drbd_make_request_26+0x246/0x400 Sep 30 12:09:34 myserver kernel: [331033.097699] [<ffffffff80347834>] generic_make_request+0x1a4/0x260 Sep 30 12:09:34 myserver kernel: [331033.111934] [<ffffffff80349083>] submit_bio+0x73/0xf0 Sep 30 12:09:34 myserver kernel: [331033.123709] [<ffffffff802b3fb4>] ? __bio_add_page+0x124/0x1e0 Sep 30 12:09:34 myserver kernel: [331033.134362] [<ffffffff80332dc6>] xfs_submit_ioend_bio+0x26/0x40 Sep 30 12:09:34 myserver kernel: [331033.146367] [<ffffffff80332e9d>] xfs_submit_ioend+0x8d/0xf0 Sep 30 12:09:34 myserver kernel: [331033.157174] [<ffffffff80333de1>] xfs_page_state_convert+0x3f1/0x6b0 Sep 30 12:09:34 myserver kernel: [331033.170137] [<ffffffff803341c5>] xfs_vm_writepage+0x65/0x100 Sep 30 12:09:34 myserver kernel: [331033.182203] [<ffffffff8026b602>] __writepage+0x12/0x40 Sep 30 12:09:34 myserver kernel: [331033.192878] [<ffffffff8026bc0b>] write_cache_pages+0x25b/0x3a0 Sep 30 12:09:34 myserver kernel: [331033.205637] [<ffffffff8026b5f0>] ? __writepage+0x0/0x40 Sep 30 12:09:34 myserver kernel: [331033.213645] [<ffffffff8026bd6f>] generic_writepages+0x1f/0x30 Sep 30 12:09:34 myserver kernel: [331033.228414] [<ffffffff80332a5a>] xfs_vm_writepages+0x4a/0x60 Sep 30 12:09:34 myserver kernel: [331033.240415] [<ffffffff8026bda8>] do_writepages+0x28/0x40 Sep 30 12:09:34 myserver kernel: [331033.252149] [<ffffffff802ab380>] __writeback_single_inode+0xa0/0x350 Sep 30 12:09:34 myserver kernel: [331033.263194] [<ffffffff802403fa>] ? del_timer_sync+0x1a/0x30 Sep 30 12:09:34 myserver kernel: [331033.275874] [<ffffffff804d445b>] ? schedule_timeout+0x6b/0xd0 Sep 30 12:09:34 myserver kernel: [331033.287761] [<ffffffff802aba7a>] sync_sb_inodes+0x20a/0x310 Sep 30 12:09:34 myserver kernel: [331033.299781] [<ffffffff802abf7e>] writeback_inodes+0x8e/0xd0 Sep 30 12:09:34 myserver kernel: [331033.311318] [<ffffffff8026c7fa>] background_writeout+0xaa/0xe0 Sep 30 12:09:34 myserver kernel: [331033.322986] [<ffffffff8026cdd0>] ? pdflush+0x0/0x1e0 Sep 30 12:09:34 myserver kernel: [331033.333314] [<ffffffff8026ceda>] pdflush+0x10a/0x1e0 Sep 30 12:09:34 myserver kernel: [331033.346182] [<ffffffff8026c750>] ? background_writeout+0x0/0xe0 Sep 30 12:09:34 myserver kernel: [331033.358249] [<ffffffff8026cdd0>] ? pdflush+0x0/0x1e0 Sep 30 12:09:34 myserver kernel: [331033.367645] [<ffffffff8024adad>] kthread+0x4d/0x80 Sep 30 12:09:34 myserver kernel: [331033.376852] [<ffffffff8020d018>] child_rip+0xa/0x12 Sep 30 12:09:34 myserver kernel: [331033.388346] [<ffffffff8024ad60>] ? kthread+0x0/0x80 Sep 30 12:09:34 myserver kernel: [331033.397028] [<ffffffff8020d00e>] ? child_rip+0x0/0x12 Sep 30 12:09:34 myserver kernel: [331033.414183] drbd0: meta connection shut down by peer. Sep 30 12:09:34 myserver kernel: [331033.423789] drbd0: peer( Secondary -> Unknown ) conn( VerifyS -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) Sep 30 12:09:34 myserver kernel: [331033.440278] drbd0: asender terminated Sep 30 12:09:34 myserver kernel: [331033.461756] drbd0: Terminating asender thread Sep 30 12:09:34 myserver kernel: [331033.469793] drbd0: w_resync_inactive: (cancel) in /usr/src/DRBD/drbd-8.2.6/drbd/drbd_worker.c:294 Sep 30 12:09:34 myserver kernel: [331033.486724] drbd0: Creating new current UUID Sep 30 12:09:34 myserver kernel: [331033.497774] drbd0: Writing meta data super block now. Lsat Log entry before reboot: Sep 30 12:10:01 myserver Anyone seen this hung before ? Deadlock ? Regard, Robert