[DRBD-user] DRBD 8.4 kernel oops when using same resource for 2 minors

Dick Tump (Shock Media B.V.) dick at shockmedia.nl
Mon Jun 16 16:22:16 CEST 2014

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hello Lars,

Thank you for your extensive reply. The tainted 'F' flag happened after 
I manually compiled DRBD 8.4.5. I have the same issue with the built-in 
8.4.3 version (just did a clean install to test it) and it doesn't have 
the 'F' flag:

Jun 16 16:09:11 ubuntu kernel: [ 92.898014] drbd: initialized. Version: 
8.4.3 (api:1/proto:86-101)
Jun 16 16:09:11 ubuntu kernel: [ 92.898032] drbd: srcversion: 
F97798065516C94BE0F27DC 
Jun 16 16:09:11 ubuntu kernel: [ 92.898032] drbd: registered as block 
device major 147
Jun 16 16:09:15 ubuntu kernel: [ 96.964271] BUG: unable to handle 
kernel NULL pointer dereference at 0000000000000028
Jun 16 16:09:15 ubuntu kernel: [ 96.965569] IP: [<ffffffff81351b38>] 
blk_throtl_drain+0x28/0x130
Jun 16 16:09:15 ubuntu kernel: [ 96.966825] PGD 369e8067 PUD 3a90c067 
PMD 0 
Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Oops: 0000 [#1] SMP 
Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Modules linked in: drbd 
lru_cache libcrc32c joydev hid_generic usbhid hid ppdev cirrus psmouse 
serio_raw ttm parport_pc drm_kms_helper drm syscopyarea sysfillrect 
sysimgblt i2c_piix4 mac_hid lp parport floppy
Jun 16 16:09:15 ubuntu kernel: [ 96.968182] CPU: 1 PID: 1318 Comm: 
drbdsetup Not tainted 3.13.0-24-generic #46-Ubuntu
Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Hardware name: Bochs Bochs, 
BIOS Bochs 01/01/2007
Jun 16 16:09:15 ubuntu kernel: [ 96.968182] task: ffff88003a5ddfc0 ti: 
ffff88003a73c000 task.ti: ffff88003a73c000
et cetera

I have tested this issue on several machines, all with different 
hardware (and also KVM virtual machines), but all running Ubuntu 14.04 
with the default (3.13) kernel. It even occurs with a clean Ubuntu 
install (install Ubuntu Server and the first thing I do is install 
drbd8-utils and run the drbdsetup commands). It might be in the Ubuntu 
kernel, I have not tested a 'vanilla' kernel without the Ubuntu 
patches. 

Met vriendelijke groet / Kind regards,
Dick Tump | Shock Media B.V.

Tel: +31 (0)546 - 714360
Fax: +31 (0)546 - 714361
Web: http://www.shockmedia.nl/

Connect to me @ LinkedIn:
http://nl.linkedin.com/in/dicktump 

On Mon, Jun 16, 2014 at 4:19 PM, Dick Tump (Shock Media B.V.) 
<dick at shockmedia.nl> wrote:
> Hello Lars,
> 
> Thank you for your extensive reply. The tainted 'F' flag happened 
> after I manually compiled DRBD 8.4.5. I have the same issue with the 
> built-in 8.4.3 version (just did a clean install to test it) and it 
> doesn't have the 'F' flag:
> 
> Jun 16 16:09:11 ubuntu kernel: [ 92.898014] drbd: initialized. 
> Version: 8.4.3 (api:1/proto:86-101)
> Jun 16 16:09:11 ubuntu kernel: [ 92.898032] drbd: srcversion: 
> F97798065516C94BE0F27DC 
> Jun 16 16:09:11 ubuntu kernel: [ 92.898032] drbd: registered as block 
> device major 147
> Jun 16 16:09:15 ubuntu kernel: [ 96.964271] BUG: unable to handle 
> kernel NULL pointer dereference at 0000000000000028
> Jun 16 16:09:15 ubuntu kernel: [ 96.965569] IP: [<ffffffff81351b38>] 
> blk_throtl_drain+0x28/0x130
> Jun 16 16:09:15 ubuntu kernel: [ 96.966825] PGD 369e8067 PUD 3a90c067 
> PMD 0 
> Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Oops: 0000 [#1] SMP 
> Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Modules linked in: drbd 
> lru_cache libcrc32c joydev hid_generic usbhid hid ppdev cirrus 
> psmouse serio_raw ttm parport_pc drm_kms_helper drm syscopyarea 
> sysfillrect sysimgblt i2c_piix4 mac_hid lp parport floppy
> Jun 16 16:09:15 ubuntu kernel: [ 96.968182] CPU: 1 PID: 1318 Comm: 
> drbdsetup Not tainted 3.13.0-24-generic #46-Ubuntu
> Jun 16 16:09:15 ubuntu kernel: [ 96.968182] Hardware name: Bochs 
> Bochs, BIOS Bochs 01/01/2007
> Jun 16 16:09:15 ubuntu kernel: [ 96.968182] task: ffff88003a5ddfc0 
> ti: ffff88003a73c000 task.ti: ffff88003a73c000
> et cetera
> 
> I have tested this issue on several machines, all with different 
> hardware (and also KVM virtual machines), but all running Ubuntu 
> 14.04 with the default (3.13) kernel. It even occurs with a clean 
> Ubuntu install (install Ubuntu Server and the first thing I do is 
> install drbd8-utils and run the drbdsetup commands). It might be in 
> the Ubuntu kernel, I have not tested a 'vanilla' kernel without the 
> Ubuntu patches. 
> 
> Met vriendelijke groet / Kind regards,
> Dick Tump | Shock Media B.V.
> 
> Tel: +31 (0)546 - 714360
> Fax: +31 (0)546 - 714361
> Web: http://www.shockmedia.nl/
> 
> Connect to me @ LinkedIn:
> http://nl.linkedin.com/in/dicktump 
> 
> On Mon, Jun 16, 2014 at 3:45 PM, Lars Ellenberg 
> <lars.ellenberg at linbit.com> wrote:
>> On Fri, Jun 13, 2014 at 04:48:37PM +0200, Dick Tump (Shock Media 
>> B.V.) wrote:
>>>  Hello,
>>>  
>>>  I have the following issue with DRBD 8.4. I have also reported it 
>>> as
>>>  a bug for the Ubuntu DRBD package, but after I did some more 
>>> testing
>>>  and used a manually compiled drbd kernel module and drbd-utils, the
>>>  issue is still there. So it seems to be a generic DRBD thing.
>>>  
>>>  When you use the same resource for 2 minors, it oops'es the kernel.
>>>  Of course you should not do that, but on the other hand a kernel
>>>  oops should not be the result. The expected output is just an error
>>>  message, I think. This makes DRBD resource/minor management more
>>>  difficult and also kind of dangerous.
>>>  
>>>  DRBD 8.3 does not have this problem, probably because it works a 
>>> bit
>>>  different with resources/minors.
>>>  
>>>  Steps to reproduce:
>>>  * drbdsetup new-resource test1
>>>  * drbdsetup new-minor test1 1 0
>>>  * drbdsetup new-minor test1 3 0
>>>  
>>>  It exits showing the message 'Killed'. The kernel log shows some
>>>  errors and DRBD is not usable anymore. After a short while or when
>>>  executing other drbdsetup commands, the whole system crashes.
>>>  
>>>  I am not sure whether it's the utils package, or the kernel module.
>>>  
>>>  The system is running Ubuntu 14.04 with kernel package
>>>  linux-image-3.13.0-24-generic. I have tested the Ubuntu packages 
>>> for
>>>  DRBD (DRBD 8.4.3 / drbd8-utils 2:8.4.4-1ubuntu1) and the manually
>>>  compiled combination from drbd-8.4.5.tar.gz and
>>>  drbd-utils-8.9.0.tar.gz.
>> 
>> As this is a kernel problem, the user land versions are not relevant.
>> 
>> If I try with my current module 8.4.5, I get
>> 
>> # drbdsetup new-resource s0
>> # drbdsetup new-minor s0 1 0
>> # drbdsetup new-minor s0 3 0
>> s0: Failure: (161) Minor exists already (delete it first)
>> additional info from kernel:
>> requested minor exists already
>> 
>> # dmesg | tail
>> drbd: events: mcg drbd: 2
>> drbd: initialized. Version: 8.4.5 (api:1/proto:86-101)
>> drbd: GIT-hash: 1d360bde0e095d495786eaeb2a1ac76888e4db96 build by 
>> root at alice, 2014-06-16 13:25:11
>> drbd: registered as block device major 147
>> 
>> Admittedly that was on a centos6 kernel.
>> 
>>>  The expected output is just an error message, I think. 
>> 
>> Right. And that's exactly what happens here.
>> 
>> I don't know what's going on there with your setup.
>> I'll try to reproduce with a 3.13 kernel later as well,
>> but I don't see why that would change anything: it's not supposed to
>> reach kernel-version specific code paths here, the drbd configuration
>> interface is catching the double minor already, as seen above.
>> 
>> 
>>>  This is the kernel output:
>>>  
>>>  [ 75.396062] BUG: unable to handle kernel NULL pointer dereference
>>>  at 0000000000000028
>>>  [ 75.397884] IP: [<ffffffff81351b38>] blk_throtl_drain+0x28/0x130
>>>  [ 75.399256] PGD dce0c067 PUD dce0b067 PMD 0 [ 75.400025] Oops: 
>>> 0000
>>>  [#1] SMP [ 75.400025] Modules linked in: drbd(OF) libcrc32c bridge
>> 
>> You should not need to "force" load DRBD.
>> If you have to, you take the blame if something goes wrong :-/
>> 
>>>  stp llc gpio_ich lpc_ich dm_multipath scsi_dh coretemp kvm_intel 
>>> kvm
>>>  serio_raw xgifb(C) lp parport i3200_edac mac_hid edac_core psmouse
>>>  tg3 ptp pps_core 3w_9xxx
>>>  [ 75.400025] CPU: 0 PID: 1290 Comm: drbdsetup-84 Tainted: GF C O
>>>  3.13.0-24-generic #47-Ubuntu
>>>  [ 75.400025] Hardware name: ASUS RS120-E5/PA2/P5BV-R, BIOS 0203
>>>  10/19/2007
>>>  [ 75.400025] task: ffff880035c317f0 ti: ffff8800e7038000 task.ti:
>>>  ffff8800e7038000
>>>  [ 75.400025] RIP: 0010:[<ffffffff81351b38>] [<ffffffff81351b38>]
>>>  blk_throtl_drain+0x28/0x130
>>>  [ 75.400025] RSP: 0018:ffff8800e7039a88 EFLAGS: 00010046
>>>  [ 75.400025] RAX: 0000000000000000 RBX: ffff880105a7b510 RCX:
>>>  000000018010000f
>>>  [ 75.400025] RDX: 000000000000000e RSI: 0000000000000001 RDI:
>>>  0000000000000000
>>>  [ 75.400025] RBP: ffff8800e7039aa0 R08: 0000000000000000 R09:
>>>  0000000000000001
>>>  [ 75.400025] R10: ffffea00039ffb00 R11: ffffffff81340f10 R12:
>>>  ffff880105a7b510
>>>  [ 75.400025] R13: ffff8800e9e88900 R14: ffff880105a7bb60 R15:
>>>  0000000000000003
>>>  [ 75.400025] FS: 00007f6220e82740(0000) GS:ffff88010fc00000(0000)
>>>  knlGS:0000000000000000
>>>  [ 75.400025] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>  [ 75.400025] CR2: 0000000000000028 CR3: 00000000dce09000 CR4:
>>>  00000000000007f0
>>>  [ 75.400025] Stack:
>>>  [ 75.400025] ffff880105a7b510 0000000000000001 ffff880105a7bb70
>>>  ffff8800e7039ab0
>>>  [ 75.400025] ffffffff8134eb7e ffff8800e7039ae0 ffffffff81332c84
>>>  ffff880105a7b510
>>>  [ 75.400025] ffff8801097cbcf0 ffff880105a7bd10 ffff88010592d800
>>>  ffff8800e7039b08
>>>  [ 75.400025] Call Trace:
>>>  [ 75.400025] [<ffffffff8134eb7e>] blkcg_drain_queue+0xe/0x10
>>>  [ 75.400025] [<ffffffff81332c84>] __blk_drain_queue+0x74/0x180
>>>  [ 75.400025] [<ffffffff81332eed>] blk_cleanup_queue+0x8d/0x180
>>>  [ 75.400025] [<ffffffffa01a4d81>] drbd_create_device+0x471/0x780 
>>> [drbd]
>>>  [ 75.400025] [<ffffffff81385164>] ? __nla_reserve_nohdr+0x24/0x30
>>>  [ 75.400025] [<ffffffffa01a3fdb>] ? drbd_find_resource+0x3b/0x90 
>>> [drbd]
>>>  [ 75.400025] [<ffffffffa01adb37>] drbd_adm_new_minor+0x87/0xd0 
>>> [drbd]
>>>  [ 75.400025] [<ffffffff8164978d>] genl_family_rcv_msg+0x18d/0x370
>>>  [ 75.400025] [<ffffffff81649970>] ? genl_family_rcv_msg+0x370/0x370
>>>  [ 75.400025] [<ffffffff81649a01>] genl_rcv_msg+0x91/0xd0
>>>  [ 75.400025] [<ffffffff81647a89>] netlink_rcv_skb+0xa9/0xc0
>>>  [ 75.400025] [<ffffffff81647f88>] genl_rcv+0x28/0x40
>>>  [ 75.400025] [<ffffffff816470b5>] netlink_unicast+0xd5/0x1b0
>>>  [ 75.400025] [<ffffffff8164748f>] netlink_sendmsg+0x2ff/0x740
>>>  [ 75.400025] [<ffffffff816016fe>] sock_aio_write+0xfe/0x130
>>>  [ 75.400025] [<ffffffff8164664d>] ? netlink_insert+0x14d/0x240
>>>  [ 75.400025] [<ffffffff811b8daa>] do_sync_write+0x5a/0x90
>>>  [ 75.400025] [<ffffffff811b962d>] vfs_write+0x1ad/0x1f0
>>>  [ 75.400025] [<ffffffff811b9f69>] SyS_write+0x49/0xa0
>>>  [ 75.400025] [<ffffffff817266bf>] tracesys+0xe1/0xe6
>>>  [ 75.400025] Code: ff 66 90 66 66 66 66 90 55 48 89 e5 41 55 41 54
>>>  49 89 fc 53 4c 8b af 70 08 00 00 49 8b 85 a0 00 00 00 31 ff 48 8b 
>>> 80
>>>  c8 05 00 00 <48> 8b 70 28 e8 9f 92 d9 ff 48 85 c0 48 89 c3 74 61 0f
>>>  1f 80 00 [ 75.400025] RIP [<ffffffff81351b38>]
>>>  blk_throtl_drain+0x28/0x130
>>>  [ 75.400025] RSP <ffff8800e7039a88>
>>>  [ 75.400025] CR2: 0000000000000028
>>>  [ 75.400025] ---[ end trace f76955bc776ce5cd ]---
>>>  
>>>  Met vriendelijke groet / Kind regards,
>>>  Dick Tump | Shock Media B.V.
>>>  
>>>  Tel: +31 (0)546 - 714360
>>>  Fax: +31 (0)546 - 714361
>>>  Web: http://www.shockmedia.nl/
>>>  
>>>  Connect to me @ LinkedIn:
>>>  http://nl.linkedin.com/in/dicktump
>> 
>>>  _______________________________________________
>>>  drbd-user mailing list
>>>  drbd-user at lists.linbit.com
>>>  http://lists.linbit.com/mailman/listinfo/drbd-user
>> 
>> 
>> -- 
>> : Lars Ellenberg
>> : LINBIT | Your Way to High Availability
>> : DRBD/HA support and consulting http://www.linbit.com
>> 
>> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
>> __
>> please don't Cc me, but send to list   --   I'm subscribed
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/drbd-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20140616/ae7ee3c2/attachment.htm>


More information about the drbd-user mailing list