Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi, hope you had a nice Weekend :-). Unfortunately the patch didnt help. I attached the relevant traces of drbd, xfsbufd and 'rm' Thanks, Jens On Fri, Aug 24, 2007 at 08:06:08PM +0200, Lars Ellenberg wrote: > > > > > > problem is understood. > > I'll fix this as soon as I find the time to code it up. > > does attached patch fix it for you? > (untested, I'm supposedly in the weekend already, > no test cluster available :->) > -- Dr. Jens Beyer IT-Systemarchitekt 1&1 Internet AG IT-Portal Brauerstrasse 48 - D-76135 Karlsruhe Tel. +49-721-91374-4245 jens.beyer at 1und1.de - http://1und1.de -------------- next part -------------- Aug 27 09:52:15 boxfe02 kernel: [ 2573.947216] drbd0_worker S ffff8100d97ea058 0 14986 2 (L-TLB) Aug 27 09:52:15 boxfe02 kernel: [ 2573.947233] ffff81011247ddf0 0000000000000046 ffffffff80551100 ffff81011247dd90 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947257] ffff810005748100 ffff81010da13180 ffff81010da13180 ffffffff80226730 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947280] 00000002050b2fc0 ffff81011be819e8 00000000000001ce ffff81010da13180 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947301] Call Trace: Aug 27 09:52:15 boxfe02 kernel: [ 2573.947309] [<ffffffff80226730>] task_rq_lock+0x50/0x90 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947319] [<ffffffff80228e48>] try_to_wake_up+0x3a8/0x4d0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947330] [<ffffffff804055c8>] __down_interruptible+0xa8/0x150 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947340] [<ffffffff80228f70>] default_wake_function+0x0/0x10 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947350] [<ffffffff804053e0>] __down_failed_interruptible+0x35/0x3a Aug 27 09:52:15 boxfe02 kernel: [ 2573.947372] [<ffffffff88134689>] :drbd:drbd_worker+0x289/0x470 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947394] [<ffffffff88148994>] :drbd:drbd_thread_setup+0x84/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947402] [<ffffffff8020ac78>] child_rip+0xa/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947423] [<ffffffff88148910>] :drbd:drbd_thread_setup+0x0/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947432] [<ffffffff8020ac6e>] child_rip+0x0/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947440] Aug 27 09:52:15 boxfe02 kernel: [ 2573.947641] drbd0_receive S 00000255d942c8ca 0 15042 2 (L-TLB) Aug 27 09:52:15 boxfe02 kernel: [ 2573.947657] ffff810105151d70 0000000000000046 0000000000000000 0000000000000000 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947678] 0000000000000000 0000000000000000 0000000000000286 0000000100034fa5 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947700] 0000000205151d80 ffff81010da13358 0000000000000a48 ffff81011bf05750 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947716] Call Trace: Aug 27 09:52:15 boxfe02 kernel: [ 2573.947726] [<ffffffff802378c7>] __mod_timer+0xb7/0xe0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947735] [<ffffffff8040428f>] schedule_timeout+0x5f/0xc0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947742] [<ffffffff80237470>] process_timeout+0x0/0x10 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947751] [<ffffffff803cfead>] inet_csk_accept+0x14d/0x270 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947760] [<ffffffff80242cf0>] autoremove_wake_function+0x0/0x30 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947767] [<ffffffff80242cf0>] autoremove_wake_function+0x0/0x30 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947776] [<ffffffff803efe40>] inet_accept+0x30/0xe0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947795] [<ffffffff88138b06>] :drbd:drbd_accept+0x76/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947815] [<ffffffff88138c4d>] :drbd:drbd_wait_for_connect+0xcd/0x170 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947834] [<ffffffff8813b467>] :drbd:drbd_connect+0xe7/0x500 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947853] [<ffffffff8813c32a>] :drbd:drbdd_init+0x5a/0x1f0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947873] [<ffffffff88148994>] :drbd:drbd_thread_setup+0x84/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947883] [<ffffffff8020ac78>] child_rip+0xa/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947903] [<ffffffff88148910>] :drbd:drbd_thread_setup+0x0/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947911] [<ffffffff8020ac6e>] child_rip+0x0/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.947918] Aug 27 09:52:15 boxfe02 kernel: [ 2573.950004] xfsbufd D 00000214cb08f979 0 15116 2 (L-TLB) Aug 27 09:52:15 boxfe02 kernel: [ 2573.950021] ffff810100b99a50 0000000000000046 0000000000000000 ba8f0fe485451045 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950042] 0f1443b70ffffffd e2c108e8c166d0b7 4514458966d00908 000000010002e271 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950063] 000000008f0f1675 ffff81011b719928 0000000000009eca ffffffff804b6080 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950082] Call Trace: Aug 27 09:52:15 boxfe02 kernel: [ 2573.950092] [<ffffffff804041b1>] wait_for_completion+0xa1/0x100 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950102] [<ffffffff80228f70>] default_wake_function+0x0/0x10 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950111] [<ffffffff80228f70>] default_wake_function+0x0/0x10 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950133] [<ffffffff88143938>] :drbd:_drbd_md_sync_page_io+0xc8/0x130 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950154] [<ffffffff881442fc>] :drbd:drbd_md_sync_page_io+0x29c/0x4f0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950172] [<ffffffff881317a0>] :drbd:drbd_bm_get_lel+0x130/0x220 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950191] [<ffffffff88132a58>] :drbd:drbd_bm_write_sect+0xc8/0x220 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950211] [<ffffffff8814353d>] :drbd:drbd_al_begin_io+0x1cd/0x320 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950221] [<ffffffff80264f99>] mempool_alloc+0x39/0x110 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950230] [<ffffffff80281d99>] cache_alloc_refill+0x199/0x500 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950240] [<ffffffff802ad1ac>] __bio_clone+0x9c/0xc0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950259] [<ffffffff88140803>] :drbd:drbd_make_request_common+0x5b3/0xb90 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950272] [<ffffffff80302130>] elv_rb_add+0x70/0x80 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950280] [<ffffffff80242cf0>] autoremove_wake_function+0x0/0x30 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950289] [<ffffffff80242cf0>] autoremove_wake_function+0x0/0x30 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950299] [<ffffffff80304014>] generic_make_request+0x1c4/0x260 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950308] [<ffffffff8030410e>] submit_bio+0x5e/0xf0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950316] [<ffffffff802ad01b>] __bio_add_page+0x1ab/0x220 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950351] [<ffffffff88296870>] :xfs:_xfs_buf_ioapply+0x230/0x2e0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950385] [<ffffffff88297659>] :xfs:xfs_buf_iorequest+0x29/0x70 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950418] [<ffffffff8829bda5>] :xfs:xfs_bdstrat_cb+0x35/0x50 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950450] [<ffffffff88297902>] :xfs:xfsbufd+0x92/0x150 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950480] [<ffffffff88297870>] :xfs:xfsbufd+0x0/0x150 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950488] [<ffffffff8024294c>] kthread+0x6c/0xa0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950497] [<ffffffff8020ac78>] child_rip+0xa/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950505] [<ffffffff802428e0>] kthread+0x0/0xa0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950513] [<ffffffff8020ac6e>] child_rip+0x0/0x12 Aug 27 09:52:15 boxfe02 kernel: [ 2573.950519] Aug 27 09:52:15 boxfe02 kernel: [ 2573.951615] rm D 00000217a4ea7278 0 15187 13792 (NOTLB) Aug 27 09:52:15 boxfe02 kernel: [ 2573.951633] ffff810109c859a8 0000000000000082 0000000000000000 000000010002e73a Aug 27 09:52:15 boxfe02 kernel: [ 2573.951657] 0000000200000000 ffff81011bd0c318 0000000000002a3f 000000010002e73a Aug 27 09:52:15 boxfe02 kernel: [ 2573.951677] 000000021bd0c140 ffff81011bd0c318 000000000000250f ffff81011bf05750 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951693] Call Trace: Aug 27 09:52:15 boxfe02 kernel: [ 2573.951704] [<ffffffff804056fb>] __down+0x8b/0x106 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951711] [<ffffffff80228f70>] default_wake_function+0x0/0x10 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951720] [<ffffffff804053a6>] __down_failed+0x35/0x3a Aug 27 09:52:15 boxfe02 kernel: [ 2573.951755] [<ffffffff88295a20>] :xfs:xfs_buf_lock+0x40/0x50 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951790] [<ffffffff88297b9c>] :xfs:_xfs_buf_find+0x13c/0x260 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951823] [<ffffffff88297d28>] :xfs:xfs_buf_get_flags+0x68/0x180 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951855] [<ffffffff88297e52>] :xfs:xfs_buf_read_flags+0x12/0x90 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951890] [<ffffffff88288627>] :xfs:xfs_trans_read_buf+0x1f7/0x320 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951923] [<ffffffff8825a8fb>] :xfs:xfs_btree_read_bufs+0x4b/0x60 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951952] [<ffffffff88243b05>] :xfs:xfs_alloc_lookup+0x155/0x3d0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.951982] [<ffffffff88240703>] :xfs:xfs_free_ag_extent+0x1a3/0x720 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952012] [<ffffffff882424f6>] :xfs:xfs_free_extent+0xc6/0x100 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952043] [<ffffffff8825232c>] :xfs:xfs_bmap_finish+0x14c/0x1a0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952079] [<ffffffff8827447c>] :xfs:xfs_itruncate_finish+0x1fc/0x310 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952115] [<ffffffff8828dbeb>] :xfs:xfs_inactive+0x3fb/0x4e0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952148] [<ffffffff8829d97c>] :xfs:xfs_fs_clear_inode+0xec/0x120 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952156] [<ffffffff8029b126>] clear_inode+0x116/0x150 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952164] [<ffffffff8029b76b>] generic_delete_inode+0x11b/0x150 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952173] [<ffffffff8029a557>] iput+0x67/0x80 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952180] [<ffffffff80291391>] do_unlinkat+0x101/0x180 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952190] [<ffffffff802931fb>] sys_getdents+0xbb/0xe0 Aug 27 09:52:15 boxfe02 kernel: [ 2573.952198] [<ffffffff80209e5e>] system_call+0x7e/0x83