Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Lars Ellenberg wrote:
>/ 2006-04-11 13:42:55 +0100
> \ Langemeyer, Werner (IBW):
>> Lars,
>>
>> still the same..., the complete /var/log/message could be find below:
>
> you are very sure that the module in use is the one with the patch?
I am also sure that the source code is in the loaded module, as long as
there are no other source code issues that I can't see (like that
drbd_blk_run_queue is actually resolved to another macro/function...)
> so, to get this "NULL pointer dereference" in spinlock,
> you have to have no queue defined for the block device,
> which due to the macro now would no longer call into blk_run_queue,
> thus would not produce the stack trace you have.
Here is what I get out of ksymoops ... don't know of that helps since I
can't get the module information right.
ksymoops 2.4.9 on i686 2.6.16-2.EL4xen0. Options used
-V (specified)
-K (specified)
-l /proc/modules (specified)
-o /lib/modules/2.6.16-2.EL4xen0/ (default)
-m /boot/System.map-2.6.16-2.EL4xen0 (specified)
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Unable to handle kernel NULL pointer dereference at virtual address 00000004
c01cd9ab
*pde = ma 789f3067 pa 31c5b067
Oops: 0000 [#1]
CPU: 0
EIP: 0061:[<c01cd9ab>] Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010086 (2.6.16-2.EL4xen0 #1)
eax: 00000000 ebx: 00000000 ecx: c73e0cc0 edx: fbc16000
esi: 00000078 edi: 00000078 ebp: c0ec5540 esp: f1c35c9c
ds: 007b es: 007b ss: 0069
Stack: <0>00000000 00000078 c02fd78b 00000000 c0942b9c c01be77d f2e10000
00000078
f49f54ce c0942b9c 0005f416 00000000 f2e10000 c02fd8ae f2e104c0
00000001
f4a04859 00000000 00000002 00000101 ffffffff 00000000 ffffffff
f1c35da0
Call Trace:
[<c02fd78b>] _spin_lock_irqsave+0x22/0x27
[<c01be77d>] blk_run_queue+0x11/0x39
[<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd]
[<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
[<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd]
[<c01cc330>] sprintf+0x17/0x1b
[<f49f5928>] drbd_bm_write+0xd/0x38 [drbd]
[<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd]
[<c02fd78b>] _spin_lock_irqsave+0x22/0x27
[<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
[<c0124eeb>] __mod_timer+0x93/0x9c
[<c02fd7ad>] _spin_lock_irq+0x1d/0x1f
[<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd]
[<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd]
[<c01c98c3>] kobject_get+0x12/0x17
[<c01c2106>] get_disk+0x3d/0x75
[<c01617b2>] blkdev_open+0x0/0x4b
[<f4926085>] dm_blk_open+0x15/0x19 [dm_mod]
[<c016165b>] do_open+0x227/0x30b
[<c01cd000>] __copy_to_user_ll+0x56/0x60
[<c01c1062>] blkdev_driver_ioctl+0x58/0x6a
[<c01c1232>] blkdev_ioctl+0x1be/0x1cd
[<c01619cd>] block_ioctl+0x1a/0x1e
[<c016a2a0>] do_ioctl+0x28/0x65
[<c016a568>] vfs_ioctl+0x180/0x18e
[<c016a5bc>] sys_ioctl+0x46/0x62
[<c01049d1>] syscall_call+0x7/0xb
Code: 31 ff ff b2 9c 00 00 00 51 ff 70 10 68 1f cd 31 c0 e8 4c fe f4 ff e8
1c 75 f3 ff 83 c4 14 eb 8d 5b 5e 5f 5d c3 56 53 8b 5c 24 0c <81> 7b 04 ad 4e
ad de 74 0d 68 4d cd 31 c0 53 e8 d9 fe ff ff 59
>>EIP; c01cd9ab <_raw_spin_lock+6/69> <=====
Trace; c02fd78b <_spin_lock_irqsave+22/27>
Trace; c01be77d <blk_run_queue+11/39>
Trace; f49f54ce <END_OF_CODE+3458e4ce/????>
Trace; c02fd8ae <_spin_unlock_irqrestore+9/31>
Trace; f4a04859 <END_OF_CODE+3459d859/????>
Trace; c01cc330 <sprintf+17/1b>
Trace; f49f5928 <END_OF_CODE+3458e928/????>
Trace; f49f6c80 <END_OF_CODE+3458fc80/????>
Trace; c02fd78b <_spin_lock_irqsave+22/27>
Trace; c02fd8ae <_spin_unlock_irqrestore+9/31>
Trace; c0124eeb <__mod_timer+93/9c>
Trace; c02fd7ad <_spin_lock_irq+1d/1f>
Trace; f49f7562 <END_OF_CODE+34590562/????>
Trace; f49f8fcd <END_OF_CODE+34591fcd/????>
Trace; c01c98c3 <kobject_get+12/17>
Trace; c01c2106 <get_disk+3d/75>
Trace; c01617b2 <blkdev_open+0/4b>
Trace; f4926085 <END_OF_CODE+344bf085/????>
Trace; c016165b <do_open+227/30b>
Trace; c01cd000 <__copy_to_user_ll+56/60>
Trace; c01c1062 <blkdev_driver_ioctl+58/6a>
Trace; c01c1232 <blkdev_ioctl+1be/1cd>
Trace; c01619cd <block_ioctl+1a/1e>
Trace; c016a2a0 <do_ioctl+28/65>
Trace; c016a568 <vfs_ioctl+180/18e>
Trace; c016a5bc <sys_ioctl+46/62>
Trace; c01049d1 <syscall_call+7/b>
This architecture has variable length instructions, decoding before eip
is unreliable, take these instructions with a pinch of salt.
Code; c01cd980 <__spin_lock_debug+60/85>
00000000 <_EIP>:
Code; c01cd980 <__spin_lock_debug+60/85>
0: 31 ff xor %edi,%edi
Code; c01cd982 <__spin_lock_debug+62/85>
2: ff b2 9c 00 00 00 pushl 0x9c(%edx)
Code; c01cd988 <__spin_lock_debug+68/85>
8: 51 push %ecx
Code; c01cd989 <__spin_lock_debug+69/85>
9: ff 70 10 pushl 0x10(%eax)
Code; c01cd98c <__spin_lock_debug+6c/85>
c: 68 1f cd 31 c0 push $0xc031cd1f
Code; c01cd991 <__spin_lock_debug+71/85>
11: e8 4c fe f4 ff call fff4fe62 <_EIP+0xfff4fe62>
Code; c01cd996 <__spin_lock_debug+76/85>
16: e8 1c 75 f3 ff call fff37537 <_EIP+0xfff37537>
Code; c01cd99b <__spin_lock_debug+7b/85>
1b: 83 c4 14 add $0x14,%esp
Code; c01cd99e <__spin_lock_debug+7e/85>
1e: eb 8d jmp ffffffad <_EIP+0xffffffad>
Code; c01cd9a0 <__spin_lock_debug+80/85>
20: 5b pop %ebx
Code; c01cd9a1 <__spin_lock_debug+81/85>
21: 5e pop %esi
Code; c01cd9a2 <__spin_lock_debug+82/85>
22: 5f pop %edi
Code; c01cd9a3 <__spin_lock_debug+83/85>
23: 5d pop %ebp
Code; c01cd9a4 <__spin_lock_debug+84/85>
24: c3 ret
Code; c01cd9a5 <_raw_spin_lock+0/69>
25: 56 push %esi
Code; c01cd9a6 <_raw_spin_lock+1/69>
26: 53 push %ebx
Code; c01cd9a7 <_raw_spin_lock+2/69>
27: 8b 5c 24 0c mov 0xc(%esp),%ebx
This decode from eip onwards should be reliable
Code; c01cd9ab <_raw_spin_lock+6/69>
00000000 <_EIP>:
Code; c01cd9ab <_raw_spin_lock+6/69> <=====
0: 81 7b 04 ad 4e ad de cmpl $0xdead4ead,0x4(%ebx) <=====
Code; c01cd9b2 <_raw_spin_lock+d/69>
7: 74 0d je 16 <_EIP+0x16>
Code; c01cd9b4 <_raw_spin_lock+f/69>
9: 68 4d cd 31 c0 push $0xc031cd4d
Code; c01cd9b9 <_raw_spin_lock+14/69>
e: 53 push %ebx
Code; c01cd9ba <_raw_spin_lock+15/69>
f: e8 d9 fe ff ff call fffffeed <_EIP+0xfffffeed>
Code; c01cd9bf <_raw_spin_lock+1a/69>
14: 59 pop %ecx
[<c011a7b4>] __might_sleep+0xa0/0xa8
[<c011f0a6>] exit_mm+0x2a/0x11e
[<c011f829>] do_exit+0x189/0x36e
[<c010522b>] do_trap+0x0/0xc1
[<c011d7f0>] printk+0xe/0x11
[<c01141c5>] do_page_fault+0x3cc/0x5d1
[<c01cd9ab>] _raw_spin_lock+0x6/0x69
[<c0113df9>] do_page_fault+0x0/0x5d1
[<c0104b63>] error_code+0x2b/0x30
[<c01cd9ab>] _raw_spin_lock+0x6/0x69
[<c02fd78b>] _spin_lock_irqsave+0x22/0x27
[<c01be77d>] blk_run_queue+0x11/0x39
[<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd]
[<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
[<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd]
[<c01cc330>] sprintf+0x17/0x1b
[<f49f5928>] drbd_bm_write+0xd/0x38 [drbd]
[<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd]
[<c02fd78b>] _spin_lock_irqsave+0x22/0x27
[<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
[<c0124eeb>] __mod_timer+0x93/0x9c
[<c02fd7ad>] _spin_lock_irq+0x1d/0x1f
[<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd]
[<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd]
[<c01c98c3>] kobject_get+0x12/0x17
[<c01c2106>] get_disk+0x3d/0x75
[<c01617b2>] blkdev_open+0x0/0x4b
[<f4926085>] dm_blk_open+0x15/0x19 [dm_mod]
[<c016165b>] do_open+0x227/0x30b
[<c01cd000>] __copy_to_user_ll+0x56/0x60
[<c01c1062>] blkdev_driver_ioctl+0x58/0x6a
[<c01c1232>] blkdev_ioctl+0x1be/0x1cd
[<c01619cd>] block_ioctl+0x1a/0x1e
[<c016a2a0>] do_ioctl+0x28/0x65
[<c016a568>] vfs_ioctl+0x180/0x18e
[<c016a5bc>] sys_ioctl+0x46/0x62
[<c01049d1>] syscall_call+0x7/0xb
Best Regards,
Michael Paesold