[DRBD-user] kernel oops drbd 8.0_pre2 on Fedora Core 5 and RHEL4

Michael Paesold mpaesold at gmx.at
Tue Apr 11 17:29:16 CEST 2006

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Lars Ellenberg wrote:

>/ 2006-04-11 13:42:55 +0100
> \ Langemeyer, Werner (IBW):
>> Lars,
>>
>> still the same..., the complete /var/log/message could be find below:
>
> you are very sure that the module in use is the one with the patch?

I am also sure that the source code is in the loaded module, as long as 
there are no other source code issues that I can't see (like that 
drbd_blk_run_queue is actually resolved to another macro/function...)

> so, to get this "NULL pointer dereference" in spinlock,
> you have to have no queue defined for the block device,
> which due to the macro now would no longer call into blk_run_queue,
> thus would not produce the stack trace you have.

Here is what I get out of ksymoops ... don't know of that helps since I 
can't get the module information right.

ksymoops 2.4.9 on i686 2.6.16-2.EL4xen0.  Options used
     -V (specified)
     -K (specified)
     -l /proc/modules (specified)
     -o /lib/modules/2.6.16-2.EL4xen0/ (default)
     -m /boot/System.map-2.6.16-2.EL4xen0 (specified)

No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Unable to handle kernel NULL pointer dereference at virtual address 00000004
c01cd9ab
*pde = ma 789f3067 pa 31c5b067
Oops: 0000 [#1]
CPU:    0
EIP:    0061:[<c01cd9ab>]    Not tainted VLI
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010086   (2.6.16-2.EL4xen0 #1)
eax: 00000000   ebx: 00000000   ecx: c73e0cc0   edx: fbc16000
esi: 00000078   edi: 00000078   ebp: c0ec5540   esp: f1c35c9c
ds: 007b   es: 007b   ss: 0069
Stack: <0>00000000 00000078 c02fd78b 00000000 c0942b9c c01be77d f2e10000 
00000078
       f49f54ce c0942b9c 0005f416 00000000 f2e10000 c02fd8ae f2e104c0 
00000001
       f4a04859 00000000 00000002 00000101 ffffffff 00000000 ffffffff 
f1c35da0
Call Trace:
 [<c02fd78b>] _spin_lock_irqsave+0x22/0x27
 [<c01be77d>] blk_run_queue+0x11/0x39
 [<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd]
 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
 [<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd]
 [<c01cc330>] sprintf+0x17/0x1b
 [<f49f5928>] drbd_bm_write+0xd/0x38 [drbd]
 [<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd]
 [<c02fd78b>] _spin_lock_irqsave+0x22/0x27
 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
 [<c0124eeb>] __mod_timer+0x93/0x9c
 [<c02fd7ad>] _spin_lock_irq+0x1d/0x1f
 [<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd]
 [<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd]
 [<c01c98c3>] kobject_get+0x12/0x17
 [<c01c2106>] get_disk+0x3d/0x75
 [<c01617b2>] blkdev_open+0x0/0x4b
 [<f4926085>] dm_blk_open+0x15/0x19 [dm_mod]
 [<c016165b>] do_open+0x227/0x30b
 [<c01cd000>] __copy_to_user_ll+0x56/0x60
 [<c01c1062>] blkdev_driver_ioctl+0x58/0x6a
 [<c01c1232>] blkdev_ioctl+0x1be/0x1cd
 [<c01619cd>] block_ioctl+0x1a/0x1e
 [<c016a2a0>] do_ioctl+0x28/0x65
 [<c016a568>] vfs_ioctl+0x180/0x18e
 [<c016a5bc>] sys_ioctl+0x46/0x62
 [<c01049d1>] syscall_call+0x7/0xb
Code: 31 ff ff b2 9c 00 00 00 51 ff 70 10 68 1f cd 31 c0 e8 4c fe f4 ff e8 
1c 75 f3 ff 83 c4 14 eb 8d 5b 5e 5f 5d c3 56 53 8b 5c 24 0c <81> 7b 04 ad 4e 
ad de 74 0d 68 4d cd 31 c0 53 e8 d9 fe ff ff 59


>>EIP; c01cd9ab <_raw_spin_lock+6/69>   <=====

Trace; c02fd78b <_spin_lock_irqsave+22/27>
Trace; c01be77d <blk_run_queue+11/39>
Trace; f49f54ce <END_OF_CODE+3458e4ce/????>
Trace; c02fd8ae <_spin_unlock_irqrestore+9/31>
Trace; f4a04859 <END_OF_CODE+3459d859/????>
Trace; c01cc330 <sprintf+17/1b>
Trace; f49f5928 <END_OF_CODE+3458e928/????>
Trace; f49f6c80 <END_OF_CODE+3458fc80/????>
Trace; c02fd78b <_spin_lock_irqsave+22/27>
Trace; c02fd8ae <_spin_unlock_irqrestore+9/31>
Trace; c0124eeb <__mod_timer+93/9c>
Trace; c02fd7ad <_spin_lock_irq+1d/1f>
Trace; f49f7562 <END_OF_CODE+34590562/????>
Trace; f49f8fcd <END_OF_CODE+34591fcd/????>
Trace; c01c98c3 <kobject_get+12/17>
Trace; c01c2106 <get_disk+3d/75>
Trace; c01617b2 <blkdev_open+0/4b>
Trace; f4926085 <END_OF_CODE+344bf085/????>
Trace; c016165b <do_open+227/30b>
Trace; c01cd000 <__copy_to_user_ll+56/60>
Trace; c01c1062 <blkdev_driver_ioctl+58/6a>
Trace; c01c1232 <blkdev_ioctl+1be/1cd>
Trace; c01619cd <block_ioctl+1a/1e>
Trace; c016a2a0 <do_ioctl+28/65>
Trace; c016a568 <vfs_ioctl+180/18e>
Trace; c016a5bc <sys_ioctl+46/62>
Trace; c01049d1 <syscall_call+7/b>

This architecture has variable length instructions, decoding before eip
is unreliable, take these instructions with a pinch of salt.

Code;  c01cd980 <__spin_lock_debug+60/85>
00000000 <_EIP>:
Code;  c01cd980 <__spin_lock_debug+60/85>
   0:   31 ff                     xor    %edi,%edi
Code;  c01cd982 <__spin_lock_debug+62/85>
   2:   ff b2 9c 00 00 00         pushl  0x9c(%edx)
Code;  c01cd988 <__spin_lock_debug+68/85>
   8:   51                        push   %ecx
Code;  c01cd989 <__spin_lock_debug+69/85>
   9:   ff 70 10                  pushl  0x10(%eax)
Code;  c01cd98c <__spin_lock_debug+6c/85>
   c:   68 1f cd 31 c0            push   $0xc031cd1f
Code;  c01cd991 <__spin_lock_debug+71/85>
  11:   e8 4c fe f4 ff            call   fff4fe62 <_EIP+0xfff4fe62>
Code;  c01cd996 <__spin_lock_debug+76/85>
  16:   e8 1c 75 f3 ff            call   fff37537 <_EIP+0xfff37537>
Code;  c01cd99b <__spin_lock_debug+7b/85>
  1b:   83 c4 14                  add    $0x14,%esp
Code;  c01cd99e <__spin_lock_debug+7e/85>
  1e:   eb 8d                     jmp    ffffffad <_EIP+0xffffffad>
Code;  c01cd9a0 <__spin_lock_debug+80/85>
  20:   5b                        pop    %ebx
Code;  c01cd9a1 <__spin_lock_debug+81/85>
  21:   5e                        pop    %esi
Code;  c01cd9a2 <__spin_lock_debug+82/85>
  22:   5f                        pop    %edi
Code;  c01cd9a3 <__spin_lock_debug+83/85>
  23:   5d                        pop    %ebp
Code;  c01cd9a4 <__spin_lock_debug+84/85>
  24:   c3                        ret
Code;  c01cd9a5 <_raw_spin_lock+0/69>
  25:   56                        push   %esi
Code;  c01cd9a6 <_raw_spin_lock+1/69>
  26:   53                        push   %ebx
Code;  c01cd9a7 <_raw_spin_lock+2/69>
  27:   8b 5c 24 0c               mov    0xc(%esp),%ebx

This decode from eip onwards should be reliable

Code;  c01cd9ab <_raw_spin_lock+6/69>
00000000 <_EIP>:
Code;  c01cd9ab <_raw_spin_lock+6/69>   <=====
   0:   81 7b 04 ad 4e ad de      cmpl   $0xdead4ead,0x4(%ebx)   <=====
Code;  c01cd9b2 <_raw_spin_lock+d/69>
   7:   74 0d                     je     16 <_EIP+0x16>
Code;  c01cd9b4 <_raw_spin_lock+f/69>
   9:   68 4d cd 31 c0            push   $0xc031cd4d
Code;  c01cd9b9 <_raw_spin_lock+14/69>
   e:   53                        push   %ebx
Code;  c01cd9ba <_raw_spin_lock+15/69>
   f:   e8 d9 fe ff ff            call   fffffeed <_EIP+0xfffffeed>
Code;  c01cd9bf <_raw_spin_lock+1a/69>
  14:   59                        pop    %ecx

 [<c011a7b4>] __might_sleep+0xa0/0xa8
 [<c011f0a6>] exit_mm+0x2a/0x11e
 [<c011f829>] do_exit+0x189/0x36e
 [<c010522b>] do_trap+0x0/0xc1
 [<c011d7f0>] printk+0xe/0x11
 [<c01141c5>] do_page_fault+0x3cc/0x5d1
 [<c01cd9ab>] _raw_spin_lock+0x6/0x69
 [<c0113df9>] do_page_fault+0x0/0x5d1
 [<c0104b63>] error_code+0x2b/0x30
 [<c01cd9ab>] _raw_spin_lock+0x6/0x69
 [<c02fd78b>] _spin_lock_irqsave+0x22/0x27
 [<c01be77d>] blk_run_queue+0x11/0x39
 [<f49f54ce>] drbd_bm_rw+0x65/0x29f [drbd]
 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
 [<f4a04859>] drbd_al_shrink+0x1a8/0x1b0 [drbd]
 [<c01cc330>] sprintf+0x17/0x1b
 [<f49f5928>] drbd_bm_write+0xd/0x38 [drbd]
 [<f49f6c80>] drbd_determin_dev_size+0x2c0/0x349 [drbd]
 [<c02fd78b>] _spin_lock_irqsave+0x22/0x27
 [<c02fd8ae>] _spin_unlock_irqrestore+0x9/0x31
 [<c0124eeb>] __mod_timer+0x93/0x9c
 [<c02fd7ad>] _spin_lock_irq+0x1d/0x1f
 [<f49f7562>] drbd_ioctl_set_disk+0x40d/0x654 [drbd]
 [<f49f8fcd>] drbd_ioctl+0x310/0xb6a [drbd]
 [<c01c98c3>] kobject_get+0x12/0x17
 [<c01c2106>] get_disk+0x3d/0x75
 [<c01617b2>] blkdev_open+0x0/0x4b
 [<f4926085>] dm_blk_open+0x15/0x19 [dm_mod]
 [<c016165b>] do_open+0x227/0x30b
 [<c01cd000>] __copy_to_user_ll+0x56/0x60
 [<c01c1062>] blkdev_driver_ioctl+0x58/0x6a
 [<c01c1232>] blkdev_ioctl+0x1be/0x1cd
 [<c01619cd>] block_ioctl+0x1a/0x1e
 [<c016a2a0>] do_ioctl+0x28/0x65
 [<c016a568>] vfs_ioctl+0x180/0x18e
 [<c016a5bc>] sys_ioctl+0x46/0x62
 [<c01049d1>] syscall_call+0x7/0xb


Best Regards,
Michael Paesold 





More information about the drbd-user mailing list