Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello all, I'm experiencing some kernel oops when using DRBD. I'm not sure if the root cause of the problem is DRBD, but it appears that DRBD resources are always affected by the oops and I've seen other DRBD related messages on the web that look similar to mine. The system is a Xen dom0 Gentoo system, running kernel 4.4.6, Xen 4.6, ZFS 0.6.5.7 and DRBD 8.4.5. The system runs very well while it is running, but at some unpredictable point in time, it has this problem. The DRBD resource appears to stop working all together when this happens. It is a multi-volume DRBD resource on top of ZFS zvols. * Here is the relevant output from dmesg when the event happens: [260175.411938] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028 [260175.411992] IP: [<ffffffff813fb602>] __memcpy+0x12/0x20 [260175.412026] PGD 40145f067 PUD 404ede067 PMD 0 [260175.412054] Oops: 0000 [#1] SMP [260175.412074] Modules linked in: xt_physdev br_netfilter iptable_filter ip_tables x_tables xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn xenfs xen_privcmd binfmt_misc drbd lru_cache openvswitch nf_defrag_ipv6 nf_conntrack bridge stp llc bonding ipmi_si ipmi_devintf ipmi_msghandler crct10dif_pclmul crc32_pclmul snd_pcm snd_timer mgag200 snd ttm soundcore drm_kms_helper pcspkr drm amd64_edac_mod fb_sys_fops edac_mce_amd syscopyarea sysfillrect sysimgblt k10temp edac_core fam15h_power ixgbe sp5100_tco i2c_piix4 uas joydev vxlan ip6_udp_tunnel udp_tunnel mdio input_leds usbkbd usbmouse 8250_fintek shpchp fjes dm_zero dm_thin_pool dm_persistent_data dm_bio_prison dm_service_time dm_round_robin dm_queue_length dm_multipath dm_log_userspace dm_flakey dm_delay xts aesni_intel lrw gf128mul [260175.412499] glue_helper ablk_helper cryptd aes_x86_64 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vmxnet3 virtio_net virtio_ring virtio tg3 sky2 r8169 pcnet32 mii igb ptp pps_core dca i2c_algo_bit e1000 bnx2 atl1c fuse zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) nfs lockd grace sunrpc fscache jfs reiserfs btrfs ext2 multipath linear raid0 dm_mirror dm_region_hash dm_log firewire_core crc_itu_t hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_monterey hid_microsoft hid_logitech_dj hid_logitech ff_memless hid_gyration hid_ezkey hid_cypress hid_chicony hid_cherry hid_belkin hid_apple hid_a4tech sl811_hcd hid_generic usbhid usb_storage hid aic94xx libsas lpfc qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 hpsa cciss 3w_9xxx 3w_xxxx 3w_sas [260175.412933] mptsas mptfc scsi_transport_fc mptspi mptscsih mptbase atp870u dc395x qla1280 imm parport dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi pdc_adma sata_inic162x sata_mv sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_via pata_jmicron pata_marvell pata_netcell pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_platform pata_cypress pata_oldpiix pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix [260175.413309] CPU: 0 PID: 8884 Comm: drbd_w_vm_ex13- Tainted: P O 4.4.6-gentoo-xen0u #3 [260175.413349] Hardware name: Supermicro AS -2022G-URF4+/H8DGU-LN4, BIOS 3.5a 09/25/2015 [260175.413387] task: ffff880415861b00 ti: ffff88034b964000 task.ti: ffff88034b964000 [260175.413420] RIP: e030:[<ffffffff813fb602>] [<ffffffff813fb602>] __memcpy+0x12/0x20 [260175.413458] RSP: e02b:ffff88034b967ad0 EFLAGS: 00010246 [260175.413482] RAX: ffff880402e94948 RBX: 0000000000000028 RCX: 0000000000000002 [260175.413514] RDX: 0000000000000000 RSI: 0000000000000028 RDI: ffff880402e94948 [260175.413546] RBP: ffff88034b967b08 R08: 00000000000006c0 R09: ffff88041b401400 [260175.413577] R10: ffff88041b401400 R11: 0000000000000000 R12: ffff88034b967c28 [260175.413609] R13: 0000000000000010 R14: 0000000000000010 R15: ffff88034b967c38 [260175.413648] FS: 00007fd9fffff700(0000) GS:ffff88041f800000(0000) knlGS:0000000000000000 [260175.413685] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [260175.413712] CR2: 0000000000000028 CR3: 0000000404edf000 CR4: 0000000000040660 [260175.413746] Stack: [260175.413757] ffffffff813fff95 ffff880402e94958 ffff8803e9128f00 ffff880365629800 [260175.413797] 0000000000000028 000000000000fe88 ffff88034b967c18 ffff88034b967ba8 [260175.413836] ffffffff8179ef25 ffffffff0000fe88 ffff8804158623a0 0000000000000003 [260175.413874] Call Trace: [260175.413892] [<ffffffff813fff95>] ? copy_from_iter+0x1f5/0x260 [260175.413923] [<ffffffff8179ef25>] tcp_sendmsg+0x605/0xaf0 [260175.413951] [<ffffffff817c9c45>] inet_sendmsg+0x65/0xa0 [260175.415401] [<ffffffff817340a5>] kernel_sendmsg+0x35/0x50 [260175.416848] [<ffffffffa0f5f8b1>] drbd_send+0xe1/0x200 [drbd] [260175.418262] [<ffffffffa0f5fa79>] __send_command.isra.37+0xa9/0x1d0 [drbd] [260175.419674] [<ffffffffa0f617a6>] drbd_send_dblock+0x286/0x640 [drbd] [260175.421063] [<ffffffffa0f57155>] ? mod_rq_state+0x355/0x610 [drbd] [260175.422431] [<ffffffff810c1981>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20 [260175.423812] [<ffffffffa0f46f57>] w_send_dblock+0x97/0x1d0 [drbd] [260175.425196] [<ffffffffa0f48358>] drbd_worker+0xf8/0x3a0 [drbd] [260175.426556] [<ffffffffa0f5dde0>] ? drbd_destroy_connection+0xf0/0xf0 [drbd] [260175.427895] [<ffffffffa0f5de2b>] drbd_thread_setup+0x4b/0x120 [drbd] [260175.429213] [<ffffffffa0f5dde0>] ? drbd_destroy_connection+0xf0/0xf0 [drbd] [260175.430500] [<ffffffff810984b8>] kthread+0xd8/0xf0 [260175.431756] [<ffffffff810983e0>] ? kthread_park+0x60/0x60 [260175.432983] [<ffffffff818514cf>] ret_from_fork+0x3f/0x70 [260175.434180] [<ffffffff810983e0>] ? kthread_park+0x60/0x60 [260175.435347] Code: 75 05 e8 72 fb ff ff 48 8b 43 60 48 2b 43 50 88 43 4e 5b 5d f3 c3 90 90 90 0f 1f 44 00 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 [260175.437792] RIP [<ffffffff813fb602>] __memcpy+0x12/0x20 [260175.438946] RSP <ffff88034b967ad0> [260175.440060] CR2: 0000000000000028 [260175.445188] ---[ end trace 004a8764b1dc737d ]--- [260209.881238] block drbd45: Remote failed to finish a request within ko-count * timeout [260209.882461] drbd vm_ex13-2: peer( Secondary -> Unknown ) conn( Connected -> Timeout ) pdsk( UpToDate -> DUnknown ) Thanks a lot! Please help! David