<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">BTW I tried again today after updating to latest DRBD bits in the ppa repo and still seeing the same problem.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><snip></div><div id="bloop_customfont" style="margin: 0px;"><div id="bloop_customfont" style="margin: 0px;">root@elysium-test01:~# drbd-overview </div><div id="bloop_customfont" style="margin: 0px;"> 0:.drbdctrl/0 Connected(2*) Secondary(2*) UpToDa/UpToDa </div><div id="bloop_customfont" style="margin: 0px;"> 1:.drbdctrl/1 Connected(2*) Secondary(2*) UpToDa/UpToDa </div><div id="bloop_customfont" style="margin: 0px;">100:data01/0 Connected(2*) Secondary(2*) UpToDa/UpToDa </div><div id="bloop_customfont" style="margin: 0px;">root@elysium-test01:~# mount -o discard,defaults /dev/drbd100 /mnt/disks/data01</div><div id="bloop_customfont" style="margin: 0px;"><div id="bloop_customfont" style="margin: 0px;">root@elysium-test01:~# drbd-overview </div><div id="bloop_customfont" style="margin: 0px;"> 0:.drbdctrl/0 Connected(2*) Secondary(2*) UpToDa/UpToDa </div><div id="bloop_customfont" style="margin: 0px;"> 1:.drbdctrl/1 Connected(2*) Secondary(2*) UpToDa/UpToDa </div><div id="bloop_customfont" style="margin: 0px;">100:data01/0 Connected(2*) Primar/Second UpToDa/UpToDa /mnt/disks/data01 ext4 83G 60M 79G 1% </div></div></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"></snip></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Then a second or so later on the second node:</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><snip></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:33 elysium-test02 kernel: [ 221.275172] drbd data01 elysium-test01: Preparing remote state change 3047474975 (primary_nodes=0, weak_nodes=0)</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:33 elysium-test02 kernel: [ 221.276435] drbd data01 elysium-test01: Committing remote state change 3047474975</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:33 elysium-test02 kernel: [ 221.276451] drbd data01 elysium-test01: peer( Secondary -> Primary )</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:33 elysium-test02 kernel: [ 221.291707] drbd data01/0 drbd100 elysium-test01: received new current UUID: 9032C1469604AE0B</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.650019] BUG: unable to handle kernel NULL pointer dereference at (null)</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.658340] IP: [<ffffffff813ff18d>] memcpy_orig+0x9d/0x110</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.664197] PGD ba137067 PUD ba136067 PMD 0 </div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.668930] Oops: 0002 [#1] SMP </div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.672650] Modules linked in: drbd_transport_tcp(OE) drbd(OE) ppdev ip6table_filter ip6_tables iptable_filter ip_tables x_tables serio_raw parport_pc parport pvpanic ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse virtio_scsi</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.726524] CPU: 0 PID: 2244 Comm: drbd_r_data01 Tainted: G OE 4.4.0-31-generic #50-Ubuntu</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.735933] Hardware name: Google Google/Google, BIOS Google 01/01/2011</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.742650] task: ffff880037639b80 ti: ffff8800afffc000 task.ti: ffff8800afffc000</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.750244] RIP: 0010:[<ffffffff813ff18d>] [<ffffffff813ff18d>] memcpy_orig+0x9d/0x110</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.758769] RSP: 0018:ffff8800affff9a8 EFLAGS: 00010202</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.764652] RAX: 0000000000000000 RBX: 0000000000000012 RCX: 0000000000000210</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.771889] RDX: 0000000000000012 RSI: ffff88003719a8ae RDI: 0000000000000000</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.779124] RBP: ffff8800affff9e0 R08: 0000000000000000 R09: 0000000000000000</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.786458] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800affffbb0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.793820] R13: 0000000000000012 R14: 0000000000000012 R15: ffff8800affffb90</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.801189] FS: 0000000000000000(0000) GS:ffff88012fc00000(0000) knlGS:0000000000000000</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.809529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.815376] CR2: 0000000000000000 CR3: 00000000ba130000 CR4: 00000000001406f0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.822636] Stack:</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.824788] ffffffff81403db6 ffff88003719a8c0 0000000000000210 000000000000003e</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.832781] 0000000000000012 0000000000000012 000000000000002c ffff8800affffa40</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.840779] ffffffff81715488 0000000000000000 ffff8800ba033480 0000000000000002</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.848761] Call Trace:</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.851320] [<ffffffff81403db6>] ? copy_to_iter+0x1b6/0x260</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.857101] [<ffffffff81715488>] skb_copy_datagram_iter+0x68/0x280</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.863649] [<ffffffff8176f953>] tcp_recvmsg+0x613/0xbe0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.869288] [<ffffffff8179d87e>] inet_recvmsg+0x7e/0xb0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.874709] [<ffffffff81705eab>] sock_recvmsg+0x3b/0x50</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.880235] [<ffffffff81706001>] kernel_recvmsg+0x61/0x80</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.885935] [<ffffffffc02e6703>] dtt_recv_short+0x63/0x80 [drbd_transport_tcp]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.893359] [<ffffffffc02e67e0>] dtt_recv+0xc0/0x180 [drbd_transport_tcp]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.900385] [<ffffffffc033b5c8>] drbd_recv+0x48/0x1f0 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.906353] [<ffffffff81705eab>] ? sock_recvmsg+0x3b/0x50</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.911974] [<ffffffffc0344878>] read_in_block+0xa8/0x3b0 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.918186] [<ffffffffc0346f00>] ? e_end_resync_block+0x130/0x130 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.925096] [<ffffffffc0347010>] receive_Data+0x110/0xcc0 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.931326] [<ffffffffc02e6803>] ? dtt_recv+0xe3/0x180 [drbd_transport_tcp]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.938496] [<ffffffffc033b5c8>] ? drbd_recv+0x48/0x1f0 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.944533] [<ffffffffc0346f00>] ? e_end_resync_block+0x130/0x130 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.951614] [<ffffffffc0346f00>] ? e_end_resync_block+0x130/0x130 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.958762] [<ffffffffc034bd26>] drbd_receiver+0x3e6/0x620 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.965069] [<ffffffffc03563c0>] ? idr_has_entry+0x10/0x10 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.971368] [<ffffffffc035641e>] drbd_thread_setup+0x5e/0x110 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.977943] [<ffffffffc03563c0>] ? idr_has_entry+0x10/0x10 [drbd]</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.984244] [<ffffffff810a0808>] kthread+0xd8/0xf0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.989235] [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 222.996225] [<ffffffff8182decf>] ret_from_fork+0x3f/0x70</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.001744] [<ffffffff810a0730>] ? kthread_create_on_node+0x1e0/0x1e0</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.008380] Code: 57 e8 4c 89 5f e0 48 8d 7f e0 73 d2 83 c2 20 48 29 d6 48 29 d7 83 fa 10 72 24 4c 8b 06 4c 8b 4e 08 4c 8b 54 16 f0 4c 8b 5c 16 f8 <4c> 89 07 4c 89 4f 08 4c 89 54 17 f0 4c 89 5c 17 f8 c3 90 83 fa </div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.035611] RIP [<ffffffff813ff18d>] memcpy_orig+0x9d/0x110</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.041512] RSP <ffff8800affff9a8></div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.045112] CR2: 0000000000000000</div><div id="bloop_customfont" style="margin: 0px;">Jul 15 17:52:35 elysium-test02 kernel: [ 223.048566] ---[ end trace 076e40e204e41f38 ]---</div></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"></snip></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Anyone have any idea whats going on and/or how to fix this?</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Thanks!</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">—jason</div></body></html>