Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Lars Ellenberg wrote: >/ 2006-04-11 13:42:55 +0100 > \ Langemeyer, Werner (IBW): >> Lars, >> >> still the same..., the complete /var/log/message could be find below: > > you are very sure that the module in use is the one with the patch? I am also sure that the source code is in the loaded module, as long as there are no other source code issues that I can't see (like that drbd_blk_run_queue is actually resolved to another macro/function...) > so, to get this "NULL pointer dereference" in spinlock, > you have to have no queue defined for the block device, > which due to the macro now would no longer call into blk_run_queue, > thus would not produce the stack trace you have. Here is what I get out of ksymoops ... don't know of that helps since I can't get the module information right. ksymoops 2.4.9 on i686 2.6.16-2.EL4xen0. Options used -V (specified) -K (specified) -l /proc/modules (specified) -o /lib/modules/2.6.16-2.EL4xen0/ (default) -m /boot/System.map-2.6.16-2.EL4xen0 (specified) No modules in ksyms, skipping objects No ksyms, skipping lsmod Unable to handle kernel NULL pointer dereference at virtual address 00000004 c01cd9ab *pde = ma 789f3067 pa 31c5b067 Oops: 0000 [#1] CPU: 0 EIP: 0061:[<c01cd9ab>] Not tainted VLI Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010086 (2.6.16-2.EL4xen0 #1) eax: 00000000 ebx: 00000000 ecx: c73e0cc0 edx: fbc16000 esi: 00000078 edi: 00000078 ebp: c0ec5540 esp: f1c35c9c ds: 007b es: 007b ss: 0069 Stack: <0>00000000 00000078 c02fd78b 00000000 c0942b9c c01be77d f2e10000 00000078 f49f54ce c0942b9c 0005f416 00000000 f2e10000 c02fd8ae f2e104c0 00000001 f4a04859 00000000 00000002 00000101 ffffffff 00000000 ffffffff f1c35da0 Call Trace: [<c02fd78b>] _spin_lock_irqsave+0x22/0x27 [<c01be77d>] blk_run_queue+0x11/0x39 [<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd] [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31 [<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd] [<c01cc330>] sprintf+0x17/0x1b [<f49f5928>] drbd_bm_write+0xd/0x38 [drbd] [<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd] [<c02fd78b>] _spin_lock_irqsave+0x22/0x27 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31 [<c0124eeb>] __mod_timer+0x93/0x9c [<c02fd7ad>] _spin_lock_irq+0x1d/0x1f [<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd] [<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd] [<c01c98c3>] kobject_get+0x12/0x17 [<c01c2106>] get_disk+0x3d/0x75 [<c01617b2>] blkdev_open+0x0/0x4b [<f4926085>] dm_blk_open+0x15/0x19 [dm_mod] [<c016165b>] do_open+0x227/0x30b [<c01cd000>] __copy_to_user_ll+0x56/0x60 [<c01c1062>] blkdev_driver_ioctl+0x58/0x6a [<c01c1232>] blkdev_ioctl+0x1be/0x1cd [<c01619cd>] block_ioctl+0x1a/0x1e [<c016a2a0>] do_ioctl+0x28/0x65 [<c016a568>] vfs_ioctl+0x180/0x18e [<c016a5bc>] sys_ioctl+0x46/0x62 [<c01049d1>] syscall_call+0x7/0xb Code: 31 ff ff b2 9c 00 00 00 51 ff 70 10 68 1f cd 31 c0 e8 4c fe f4 ff e8 1c 75 f3 ff 83 c4 14 eb 8d 5b 5e 5f 5d c3 56 53 8b 5c 24 0c <81> 7b 04 ad 4e ad de 74 0d 68 4d cd 31 c0 53 e8 d9 fe ff ff 59 >>EIP; c01cd9ab <_raw_spin_lock+6/69> <===== Trace; c02fd78b <_spin_lock_irqsave+22/27> Trace; c01be77d <blk_run_queue+11/39> Trace; f49f54ce <END_OF_CODE+3458e4ce/????> Trace; c02fd8ae <_spin_unlock_irqrestore+9/31> Trace; f4a04859 <END_OF_CODE+3459d859/????> Trace; c01cc330 <sprintf+17/1b> Trace; f49f5928 <END_OF_CODE+3458e928/????> Trace; f49f6c80 <END_OF_CODE+3458fc80/????> Trace; c02fd78b <_spin_lock_irqsave+22/27> Trace; c02fd8ae <_spin_unlock_irqrestore+9/31> Trace; c0124eeb <__mod_timer+93/9c> Trace; c02fd7ad <_spin_lock_irq+1d/1f> Trace; f49f7562 <END_OF_CODE+34590562/????> Trace; f49f8fcd <END_OF_CODE+34591fcd/????> Trace; c01c98c3 <kobject_get+12/17> Trace; c01c2106 <get_disk+3d/75> Trace; c01617b2 <blkdev_open+0/4b> Trace; f4926085 <END_OF_CODE+344bf085/????> Trace; c016165b <do_open+227/30b> Trace; c01cd000 <__copy_to_user_ll+56/60> Trace; c01c1062 <blkdev_driver_ioctl+58/6a> Trace; c01c1232 <blkdev_ioctl+1be/1cd> Trace; c01619cd <block_ioctl+1a/1e> Trace; c016a2a0 <do_ioctl+28/65> Trace; c016a568 <vfs_ioctl+180/18e> Trace; c016a5bc <sys_ioctl+46/62> Trace; c01049d1 <syscall_call+7/b> This architecture has variable length instructions, decoding before eip is unreliable, take these instructions with a pinch of salt. Code; c01cd980 <__spin_lock_debug+60/85> 00000000 <_EIP>: Code; c01cd980 <__spin_lock_debug+60/85> 0: 31 ff xor %edi,%edi Code; c01cd982 <__spin_lock_debug+62/85> 2: ff b2 9c 00 00 00 pushl 0x9c(%edx) Code; c01cd988 <__spin_lock_debug+68/85> 8: 51 push %ecx Code; c01cd989 <__spin_lock_debug+69/85> 9: ff 70 10 pushl 0x10(%eax) Code; c01cd98c <__spin_lock_debug+6c/85> c: 68 1f cd 31 c0 push $0xc031cd1f Code; c01cd991 <__spin_lock_debug+71/85> 11: e8 4c fe f4 ff call fff4fe62 <_EIP+0xfff4fe62> Code; c01cd996 <__spin_lock_debug+76/85> 16: e8 1c 75 f3 ff call fff37537 <_EIP+0xfff37537> Code; c01cd99b <__spin_lock_debug+7b/85> 1b: 83 c4 14 add $0x14,%esp Code; c01cd99e <__spin_lock_debug+7e/85> 1e: eb 8d jmp ffffffad <_EIP+0xffffffad> Code; c01cd9a0 <__spin_lock_debug+80/85> 20: 5b pop %ebx Code; c01cd9a1 <__spin_lock_debug+81/85> 21: 5e pop %esi Code; c01cd9a2 <__spin_lock_debug+82/85> 22: 5f pop %edi Code; c01cd9a3 <__spin_lock_debug+83/85> 23: 5d pop %ebp Code; c01cd9a4 <__spin_lock_debug+84/85> 24: c3 ret Code; c01cd9a5 <_raw_spin_lock+0/69> 25: 56 push %esi Code; c01cd9a6 <_raw_spin_lock+1/69> 26: 53 push %ebx Code; c01cd9a7 <_raw_spin_lock+2/69> 27: 8b 5c 24 0c mov 0xc(%esp),%ebx This decode from eip onwards should be reliable Code; c01cd9ab <_raw_spin_lock+6/69> 00000000 <_EIP>: Code; c01cd9ab <_raw_spin_lock+6/69> <===== 0: 81 7b 04 ad 4e ad de cmpl $0xdead4ead,0x4(%ebx) <===== Code; c01cd9b2 <_raw_spin_lock+d/69> 7: 74 0d je 16 <_EIP+0x16> Code; c01cd9b4 <_raw_spin_lock+f/69> 9: 68 4d cd 31 c0 push $0xc031cd4d Code; c01cd9b9 <_raw_spin_lock+14/69> e: 53 push %ebx Code; c01cd9ba <_raw_spin_lock+15/69> f: e8 d9 fe ff ff call fffffeed <_EIP+0xfffffeed> Code; c01cd9bf <_raw_spin_lock+1a/69> 14: 59 pop %ecx [<c011a7b4>] __might_sleep+0xa0/0xa8 [<c011f0a6>] exit_mm+0x2a/0x11e [<c011f829>] do_exit+0x189/0x36e [<c010522b>] do_trap+0x0/0xc1 [<c011d7f0>] printk+0xe/0x11 [<c01141c5>] do_page_fault+0x3cc/0x5d1 [<c01cd9ab>] _raw_spin_lock+0x6/0x69 [<c0113df9>] do_page_fault+0x0/0x5d1 [<c0104b63>] error_code+0x2b/0x30 [<c01cd9ab>] _raw_spin_lock+0x6/0x69 [<c02fd78b>] _spin_lock_irqsave+0x22/0x27 [<c01be77d>] blk_run_queue+0x11/0x39 [<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd] [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31 [<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd] [<c01cc330>] sprintf+0x17/0x1b [<f49f5928>] drbd_bm_write+0xd/0x38 [drbd] [<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd] [<c02fd78b>] _spin_lock_irqsave+0x22/0x27 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31 [<c0124eeb>] __mod_timer+0x93/0x9c [<c02fd7ad>] _spin_lock_irq+0x1d/0x1f [<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd] [<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd] [<c01c98c3>] kobject_get+0x12/0x17 [<c01c2106>] get_disk+0x3d/0x75 [<c01617b2>] blkdev_open+0x0/0x4b [<f4926085>] dm_blk_open+0x15/0x19 [dm_mod] [<c016165b>] do_open+0x227/0x30b [<c01cd000>] __copy_to_user_ll+0x56/0x60 [<c01c1062>] blkdev_driver_ioctl+0x58/0x6a [<c01c1232>] blkdev_ioctl+0x1be/0x1cd [<c01619cd>] block_ioctl+0x1a/0x1e [<c016a2a0>] do_ioctl+0x28/0x65 [<c016a568>] vfs_ioctl+0x180/0x18e [<c016a5bc>] sys_ioctl+0x46/0x62 [<c01049d1>] syscall_call+0x7/0xb Best Regards, Michael Paesold