[Drbd-dev] NULL pointer derefernce in 8.4.7-1 during drbd_destroy_connection()

Eric Wheeler drbd-dev at lists.ewheeler.net
Wed Mar 30 21:52:09 CEST 2016


On Wed, 30 Mar 2016, Lars Ellenberg wrote:

> On Wed, Mar 30, 2016 at 02:19:07AM +0000, Eric Wheeler wrote:
> > Hello all,
> > 
> > We are getting kernel crashes in linux 4.1.20 with the drbd-8.4.git tree 
> > at commit 3a6a769340ef93b1ba2792c6461250790795db49 .  
> > 
> > I don't see anything in the newer commits that addresses this issue so 
> > I'm posting---but I'll try the latest commit in master, too, just in case.
> > 
> > Please see the backtrace below.  I also included our global_common.conf 
> > further down.  This is protocol A and the link is quite slow.  This NULL 
> > ptr dereference appears to show up when the drbd kernel thread is blocked 
> > for a long time.  It might happen at reconnect time because the BUG didn't 
> > show up until 13 seconds after the P_BARRIER error.
> > 
> > The problem is pretty reproducable, so I can probably test patches.  
> > Please let me know what I can do to help test.
> 
> DRBD logs of both peers leading up to the incident may be useful.

See attached for the side that locked up at 18:05:51.  The first line 
starts at of the sending peer is 18:04:30 PST for remote correlation.  
The first line on the receiving peer is Mar 29 18:06:04 (15s after lockup) 
and both machines are ntp slaved.

The receiving side has the same module version but doesn't have any logs 
for 15 mins before the lockup, and the only logs after the lockup are 
"PingAck did not arrive in time." with related retries, but attached for 
reference.

Note that these are blank volumes on the receiver.  We just create-md'ed 
and started a fresh sync with proto A to move volumes to a different 
datacenter.
 
> check if older kernel versions are ok?
> as in 2.6.32, 3.10, ...
> if older seems to be ok, figure out which version breaks.
> 
> maybe check if older DRBD is still ok (maybe this is a more recent regression?)

I might be able to try earlier kernels, will see.  This is el7, not sure 
if I can go earlier than 3.10 for possible userspace requirements.

> try to resolve addresses to source code lines.

These correlate to the trace below in backtrace order.  It looks like a 
problem with drbd teardown since the bottom of the trace stack calls to 
drbd_destroy_connection:

(gdb) list *(drbd_send+0xe6)
0x29f56 is in drbd_send (drbd/drbd_main.c:1913).
1908			rcu_read_unlock();
1909			drbd_update_congested(connection);
1910		}
1911		do {
1912			rv = kernel_sendmsg(sock, &msg, &iov, 1, size);   <<<<<< Leaves DRBD
1913			if (rv == -EAGAIN) {
1914				if (we_should_drop_the_connection(connection, sock))
1915					break;
1916				else
1917					continue;

(gdb) list *(_drbd_no_send_page.isra.40+0x71)
A syntax error in expression, near `.40+0x71)'.
(gdb) list *(drbd_send_dblock+0x3e8)
0x2c1a8 is in drbd_send_dblock (drbd/drbd_main.c:1646).
1641			int err;
1642	
1643			err = _drbd_no_send_page(peer_device, bvec BVD bv_page,
1644						 bvec BVD bv_offset, bvec BVD bv_len,
1645						 bio_iter_last(bvec, iter) ? 0 : MSG_MORE);
1646			if (err)
1647				return err;
1648			/* REQ_WRITE_SAME has only one segment */
1649			if (bio->bi_rw & DRBD_REQ_WSAME)
1650				break;

(gdb) list *(complete_master_bio+0x94)
0x1e8a4 is in complete_master_bio (drbd/drbd_req.c:227).
222	void complete_master_bio(struct drbd_device *device,
223			struct bio_and_error *m)
224	{
225		bio_endio(m->bio, m->error);
226		dec_ap_bio(device);
227	}
228	
229	
230	/* Helper for __req_mod().
231	 * Set m->bio to the master bio, if it is fit to be completed,

(gdb) list *(w_send_dblock+0xaf)
0xc5ff is in w_send_dblock (drbd/drbd_req.h:321).
316	 * If you need it irqsave, do it your self!
317	 * Which means: don't use from bio endio callback. */
318	static inline int req_mod(struct drbd_request *req,
319			enum drbd_req_event what)
320	{
321		struct drbd_device *device = req->device;
322		struct bio_and_error m;
323		int rv;
324	
325		spin_lock_irq(&device->resource->req_lock);

(gdb) list *(drbd_worker+0xf9)
0xd9d9 is in drbd_worker (drbd/drbd_worker.c:2205).
2200	
2201			if (!list_empty(&work_list)) {
2202				w = list_first_entry(&work_list, struct drbd_work, list);
2203				list_del_init(&w->list);
2204				update_worker_timing_details(connection, w->cb);
2205				if (w->cb(w, connection->cstate < C_WF_REPORT_PARAMS) == 0)
2206					continue;
2207				if (connection->cstate >= C_WF_REPORT_PARAMS)
2208					conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
2209			}

(gdb) list *(drbd_destroy_connection+0x190)
0x27d30 is in drbd_thread_setup (drbd/drbd_main.c:362).
357		}
358		spin_unlock_irq(&connection->resource->req_lock);
359	}
360	
361	static int drbd_thread_setup(void *arg)
362	{
363		struct drbd_thread *thi = (struct drbd_thread *) arg;
364		struct drbd_resource *resource = thi->resource;
365		unsigned long flags;
366		int retval;

(gdb) list *(drbd_thread_setup+0x1d)
0x27d4d is in drbd_thread_setup (drbd/drbd_main.c:371).
366		int retval;
367	
368	restart:
369		retval = thi->function(thi);
370	
371		spin_lock_irqsave(&thi->t_lock, flags);
372	
373		/* if the receiver has been "EXITING", the last thing it did
374		 * was set the conn state to "StandAlone",
375		 * if now a re-connect request comes in, conn state goes C_UNCONNECTED,

(gdb) list *(drbd_destroy_connection+0x190)
0x27d30 is in drbd_thread_setup (drbd/drbd_main.c:362).
357		}
358		spin_unlock_irq(&connection->resource->req_lock);
359	}
360	
361	static int drbd_thread_setup(void *arg)
362	{
363		struct drbd_thread *thi = (struct drbd_thread *) arg;
364		struct drbd_resource *resource = thi->resource;
365		unsigned long flags;
366		int retval;


> 
> > [ 2480.751713]  [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
> > [ 2480.753608]  [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
> > [ 2480.755463]  [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
> > [ 2480.757263]  [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
> > [ 2480.759073]  [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
> > [ 2480.760844]  [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
> > [ 2480.762567]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
> > [ 2480.764181]  [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
> > [ 2480.765777]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
> > [ 2480.767337]  [<ffffffff810c0b08>] kthread+0xd8/0xf0
> > [ 2480.768873]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
> > [ 2480.770409]  [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
> > [ 2480.771868]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
> > 
> > 
> > ===> /etc/drbd.d/global_common.conf <===
> > common {
> > 	startup {
> > 		wfc-timeout 30;
> > 		outdated-wfc-timeout 20;
> > 		degr-wfc-timeout 30;
> > 	}
> > 	options {
> > 		on-no-data-accessible suspend-io;
> > 	}
> > 	syncer {
> > 		rate 500M;
> > 	}
> > 	disk {
> > 		al-extents 3389;
> > 		c-fill-target 10240;
> > 		c-delay-target 100;
> > 		c-plan-ahead 70;
> > 		c-min-rate 1024;
> > 		c-max-rate 400M;
> > 		on-io-error pass_on;
> > 		read-balancing when-congested-remote;
> > 	}
> > 	net {
> > 		after-sb-0pri discard-zero-changes;
> > 		after-sb-1pri call-pri-lost-after-sb;
> > 		after-sb-2pri disconnect;
> > 		allow-two-primaries no;
> > 		protocol A;
> > 		cram-hmac-alg sha1;
> > 		verify-alg crc32c;
> > 		csums-alg crc32c;
> > 		max-buffers 8192;
> > 		max-epoch-size 8192;
> > 		tcp-cork yes;
> > 		sndbuf-size 1M;
> > 		rcvbuf-size 2M;
> > 		unplug-watermark 128; 
> > 		ko-count 3;
> > 		timeout 90;
> > 		
> > 		ping-int 10;
> > 		ping-timeout 30;
> > 	}
> > }
> 


--
Eric Wheeler

> -- 
> : Lars Ellenberg
> : LINBIT | Keeping the Digital World Running
> : DRBD -- Heartbeat -- Corosync -- Pacemaker
> : R&D, Integration, Ops, Consulting, Support
> 
> DRBD® and LINBIT® are registered trademarks of LINBIT
> _______________________________________________
> drbd-dev mailing list
> drbd-dev at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
> 
-------------- next part --------------
Mar 29 18:04:30 san2 [ 2399.426673] block drbd7935: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
Mar 29 18:04:30 san2 [ 2399.431035] block drbd7935: drbd_sync_handshake:
Mar 29 18:04:30 san2 [ 2399.433091] block drbd7935: self D5FCC36B7DB360CA:0000000000000000:D0FDFBD85DF1B5A5:D0FCFBD85DF1B5A5 bits:879616 flags:0
Mar 29 18:04:30 san2 [ 2399.435147] block drbd7935: peer 05835EB3EEC6BD5D:D5FCC36B7DB360CA:D0FDFBD85DF1B5A4:D0FCFBD85DF1B5A5 bits:20590 flags:0
Mar 29 18:04:30 san2 [ 2399.437178] block drbd7935: uuid_compare()=-1 by rule 50
Mar 29 18:04:30 san2 [ 2399.439181] block drbd7935: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate ) 
Mar 29 18:04:30 san2 [ 2399.451110] block drbd7935: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 839(1), total 839; compression: 99.9%
Mar 29 18:04:30 san2 [ 2399.453159] block drbd7935: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 600(1), total 600; compression: 99.9%
Mar 29 18:04:30 san2 [ 2399.454990] block drbd7935: conn( WFBitMapT -> WFSyncUUID ) 
Mar 29 18:04:30 san2 [ 2399.488131] block drbd7935: updated sync uuid D5FDC36B7DB360CA:0000000000000000:D0FDFBD85DF1B5A5:D0FCFBD85DF1B5A5
Mar 29 18:04:30 san2 [ 2399.488806] block drbd7935: helper command: /sbin/drbdadm before-resync-target minor-7935
Mar 29 18:04:30 san2 [ 2399.499781] block drbd7935: helper command: /sbin/drbdadm before-resync-target minor-7935 exit code 0 (0x0)
Mar 29 18:04:30 san2 [ 2399.501347] block drbd7935: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) 
Mar 29 18:04:30 san2 [ 2399.502902] block drbd7935: Began resync as SyncTarget (will sync 3540040 KB [885010 bits set]).
Mar 29 18:04:40 san2 [ 2409.256237] block drbd7: We did not send a P_BARRIER for 27002ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:41 san2 [ 2410.178203] block drbd24: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:42 san2 [ 2411.278153] block drbd7994: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:42 san2 [ 2411.422145] block drbd7945: We did not send a P_BARRIER for 37575ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:09 san2 [ 2438.881042] block drbd7994: We did not send a P_BARRIER for 27001ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:22 san2 [ 2451.804514] block drbd7945: We did not send a P_BARRIER for 27044ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:29 san2 [ 2458.780210] block drbd15: We did not send a P_BARRIER for 27038ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:38 san2 [ 2467.187849] block drbd7994: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:51 san2 [ 2480.674208] BUG: unable to handle kernel 
Mar 29 18:05:51 san2  at 0000000000000003
Mar 29 18:05:51 san2 [ 2480.676403] IP:
Mar 29 18:05:51 san2  [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.678547] PGD 0 
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.680628] Oops: 0000 [#1] 
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.682675] Modules linked in:
Mar 29 18:05:51 san2  dm_snapshot
Mar 29 18:05:51 san2  xt_comment
Mar 29 18:05:51 san2  binfmt_misc
Mar 29 18:05:51 san2  xt_CHECKSUM
Mar 29 18:05:51 san2  iptable_mangle
Mar 29 18:05:51 san2  ipt_MASQUERADE
Mar 29 18:05:51 san2  nf_nat_masquerade_ipv4
Mar 29 18:05:51 san2  iptable_nat
Mar 29 18:05:51 san2  nf_nat_ipv4
Mar 29 18:05:51 san2  nf_nat
Mar 29 18:05:51 san2  nf_conntrack_ipv4
Mar 29 18:05:51 san2  nf_defrag_ipv4
Mar 29 18:05:51 san2  xt_conntrack
Mar 29 18:05:51 san2  nf_conntrack
Mar 29 18:05:51 san2  ipt_REJECT
Mar 29 18:05:51 san2  nf_reject_ipv4
Mar 29 18:05:51 san2  ebtable_filter
Mar 29 18:05:51 san2  ebtables
Mar 29 18:05:51 san2  ip6table_filter
Mar 29 18:05:51 san2  ip6_tables
Mar 29 18:05:51 san2  iptable_filter
Mar 29 18:05:51 san2  drbd(O)
Mar 29 18:05:51 san2  xfs
Mar 29 18:05:51 san2  dm_thin_pool
Mar 29 18:05:51 san2  dm_persistent_data
Mar 29 18:05:51 san2  dm_bio_prison
Mar 29 18:05:51 san2  dm_bufio
Mar 29 18:05:51 san2  libcrc32c
Mar 29 18:05:51 san2  bcache
Mar 29 18:05:51 san2  netconsole
Mar 29 18:05:51 san2  zram
Mar 29 18:05:51 san2  lz4_compress
Mar 29 18:05:51 san2  bridge
Mar 29 18:05:51 san2  8021q
Mar 29 18:05:51 san2  garp
Mar 29 18:05:51 san2  mrp
Mar 29 18:05:51 san2  stp
Mar 29 18:05:51 san2  llc
Mar 29 18:05:51 san2  x86_pkg_temp_thermal
Mar 29 18:05:51 san2  intel_powerclamp
Mar 29 18:05:51 san2  coretemp
Mar 29 18:05:51 san2  kvm_intel
Mar 29 18:05:51 san2  kvm
Mar 29 18:05:51 san2  crct10dif_pclmul
Mar 29 18:05:51 san2  iTCO_wdt
Mar 29 18:05:51 san2  crc32_pclmul
Mar 29 18:05:51 san2  iTCO_vendor_support
Mar 29 18:05:51 san2  sg
Mar 29 18:05:51 san2  ipmi_si
Mar 29 18:05:51 san2  ipmi_msghandler
Mar 29 18:05:51 san2  shpchp
Mar 29 18:05:51 san2  i2c_i801
Mar 29 18:05:51 san2  lpc_ich
Mar 29 18:05:51 san2  video
Mar 29 18:05:51 san2  mfd_core
Mar 29 18:05:51 san2  pcspkr
Mar 29 18:05:51 san2  nfsd
Mar 29 18:05:51 san2  auth_rpcgss
Mar 29 18:05:51 san2  nfs_acl
Mar 29 18:05:51 san2  lockd
Mar 29 18:05:51 san2  grace
Mar 29 18:05:51 san2  sunrpc
Mar 29 18:05:51 san2  ip_tables
Mar 29 18:05:51 san2  ext4
Mar 29 18:05:51 san2  mbcache
Mar 29 18:05:51 san2  jbd2
Mar 29 18:05:51 san2  mgag200
Mar 29 18:05:51 san2  syscopyarea
Mar 29 18:05:51 san2  sysfillrect
Mar 29 18:05:51 san2  sysimgblt
Mar 29 18:05:51 san2  i2c_algo_bit
Mar 29 18:05:51 san2  drm_kms_helper
Mar 29 18:05:51 san2  ttm
Mar 29 18:05:51 san2  ahci
Mar 29 18:05:51 san2  crc32c_intel
Mar 29 18:05:51 san2  libahci
Mar 29 18:05:51 san2  drm
Mar 29 18:05:51 san2  libata
Mar 29 18:05:51 san2  serio_raw
Mar 29 18:05:51 san2  ixgbe
Mar 29 18:05:51 san2  i2c_core
Mar 29 18:05:51 san2  e1000e
Mar 29 18:05:51 san2  mdio
Mar 29 18:05:51 san2  dca
Mar 29 18:05:51 san2  ptp
Mar 29 18:05:51 san2  arcmsr
Mar 29 18:05:51 san2  pps_core
Mar 29 18:05:51 san2  dm_mirror
Mar 29 18:05:51 san2  dm_region_hash
Mar 29 18:05:51 san2  dm_log
Mar 29 18:05:51 san2  dm_mod
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.700341] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G           O    4.1.20-3.el7.x86_64 #1
Mar 29 18:05:51 san2 [ 2480.702612] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:51 san2 [ 2480.704921] task: ffff8807c01d6e00 ti: ffff8807b7a88000 task.ti: ffff8807b7a88000
Mar 29 18:05:51 san2 [ 2480.707206] RIP: 0010:[<ffffffff81357a96>] 
Mar 29 18:05:51 san2  [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.709537] RSP: 0018:ffff8807b7a8ba50  EFLAGS: 00010286
Mar 29 18:05:51 san2 [ 2480.711774] RAX: ffff8807dc0cc4d8 RBX: 00000000000004a6 RCX: 00000000000004a6
Mar 29 18:05:51 san2 [ 2480.714079] RDX: 00000000000004a6 RSI: 0000000000000003 RDI: ffff8807dc0cc4d8
Mar 29 18:05:51 san2 [ 2480.716324] RBP: ffff8807b7a8ba98 R08: ffff8807b7a8bbd8 R09: ffff8807c01d7b28
Mar 29 18:05:51 san2 [ 2480.718605] R10: 0000000000000000 R11: ffff88075891b800 R12: 0000000000000a50
Mar 29 18:05:51 san2 [ 2480.720853] R13: ffff8807b7a8bbf8 R14: ffff8807b7a8bbf8 R15: 0000000000000000
Mar 29 18:05:51 san2 [ 2480.723052] FS:  0000000000000000(0000) GS:ffff88082fdc0000(0000) knlGS:0000000000000000
Mar 29 18:05:51 san2 [ 2480.725221] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 29 18:05:51 san2 [ 2480.727436] CR2: 0000000000000003 CR3: 0000000001a22000 CR4: 00000000001426e0
Mar 29 18:05:51 san2 [ 2480.729500] Stack:
Mar 29 18:05:51 san2 [ 2480.731538]  ffffffff8135c40f
Mar 29 18:05:51 san2  ffff8807b7a8bbd8
Mar 29 18:05:51 san2  ffff8807dc0cc97e
Mar 29 18:05:51 san2  ffff8807b7a8ba98
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.733562]  ffff8807738cb600
Mar 29 18:05:51 san2  ffff8807f7949000
Mar 29 18:05:51 san2  000000000000fa9b
Mar 29 18:05:51 san2  ffff8807b7a8bbe8
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.735620]  0000000000000a50
Mar 29 18:05:51 san2  ffff8807b7a8bb48
Mar 29 18:05:51 san2  ffffffff8161b76a
Mar 29 18:05:51 san2  ffff88070000000c
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.737704] Call Trace:
Mar 29 18:05:51 san2 [ 2480.739750]  [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:51 san2 [ 2480.741834]  [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:51 san2 [ 2480.743914]  [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:51 san2 [ 2480.745905]  [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:51 san2 [ 2480.747877]  [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:51 san2 [ 2480.749813]  [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:51 san2 [ 2480.751713]  [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:51 san2 [ 2480.753608]  [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:51 san2 [ 2480.755463]  [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.757263]  [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:51 san2 [ 2480.759073]  [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:51 san2 [ 2480.760844]  [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.762567]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.764181]  [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:51 san2 [ 2480.765777]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.767337]  [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:51 san2 [ 2480.768873]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.770409]  [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:51 san2 [ 2480.771868]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.773358] Code: 
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.776563] RIP 
Mar 29 18:05:51 san2  [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.778101]  RSP <ffff8807b7a8ba50>
Mar 29 18:05:51 san2 [ 2480.779584] CR2: 0000000000000003
Mar 29 18:05:51 san2 [ 2480.783016] ------------[ cut here ]------------
Mar 29 18:05:51 san2 [ 2480.784328] kernel BUG at arch/x86/mm/pageattr.c:214!
Mar 29 18:05:51 san2 [ 2480.785605] invalid opcode: 0000 [#2] 
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.786849] Modules linked in:
Mar 29 18:05:51 san2  dm_snapshot
Mar 29 18:05:51 san2  xt_comment
Mar 29 18:05:51 san2  binfmt_misc
Mar 29 18:05:51 san2  xt_CHECKSUM
Mar 29 18:05:51 san2  iptable_mangle
Mar 29 18:05:51 san2  ipt_MASQUERADE
Mar 29 18:05:51 san2  nf_nat_masquerade_ipv4
Mar 29 18:05:51 san2  iptable_nat
Mar 29 18:05:51 san2  nf_nat_ipv4
Mar 29 18:05:51 san2  nf_nat
Mar 29 18:05:51 san2  nf_conntrack_ipv4
Mar 29 18:05:51 san2  nf_defrag_ipv4
Mar 29 18:05:51 san2  xt_conntrack
Mar 29 18:05:51 san2  nf_conntrack
Mar 29 18:05:51 san2  ipt_REJECT
Mar 29 18:05:51 san2  nf_reject_ipv4
Mar 29 18:05:51 san2  ebtable_filter
Mar 29 18:05:51 san2  ebtables
Mar 29 18:05:51 san2  ip6table_filter
Mar 29 18:05:51 san2  ip6_tables
Mar 29 18:05:51 san2  iptable_filter
Mar 29 18:05:51 san2  drbd(O)
Mar 29 18:05:51 san2  xfs
Mar 29 18:05:51 san2  dm_thin_pool
Mar 29 18:05:51 san2  dm_persistent_data
Mar 29 18:05:51 san2  dm_bio_prison
Mar 29 18:05:51 san2  dm_bufio
Mar 29 18:05:51 san2  libcrc32c
Mar 29 18:05:51 san2  bcache
Mar 29 18:05:51 san2  netconsole
Mar 29 18:05:51 san2  zram
Mar 29 18:05:51 san2  lz4_compress
Mar 29 18:05:51 san2  bridge
Mar 29 18:05:51 san2  8021q
Mar 29 18:05:51 san2  garp
Mar 29 18:05:51 san2  mrp
Mar 29 18:05:51 san2  stp
Mar 29 18:05:51 san2  llc
Mar 29 18:05:51 san2  x86_pkg_temp_thermal
Mar 29 18:05:51 san2  intel_powerclamp
Mar 29 18:05:51 san2  coretemp
Mar 29 18:05:51 san2  kvm_intel
Mar 29 18:05:51 san2  kvm
Mar 29 18:05:51 san2  crct10dif_pclmul
Mar 29 18:05:51 san2  iTCO_wdt
Mar 29 18:05:51 san2  crc32_pclmul
Mar 29 18:05:51 san2  iTCO_vendor_support
Mar 29 18:05:51 san2  sg
Mar 29 18:05:51 san2  ipmi_si
Mar 29 18:05:51 san2  ipmi_msghandler
Mar 29 18:05:51 san2  shpchp
Mar 29 18:05:51 san2  i2c_i801
Mar 29 18:05:51 san2  lpc_ich
Mar 29 18:05:51 san2  video
Mar 29 18:05:51 san2  mfd_core
Mar 29 18:05:51 san2  pcspkr
Mar 29 18:05:51 san2  nfsd
Mar 29 18:05:51 san2  auth_rpcgss
Mar 29 18:05:51 san2  nfs_acl
Mar 29 18:05:51 san2  lockd
Mar 29 18:05:51 san2  grace
Mar 29 18:05:51 san2  sunrpc
Mar 29 18:05:51 san2  ip_tables
Mar 29 18:05:51 san2  ext4
Mar 29 18:05:51 san2  mbcache
Mar 29 18:05:51 san2  jbd2
Mar 29 18:05:51 san2  mgag200
Mar 29 18:05:51 san2  syscopyarea
Mar 29 18:05:51 san2  sysfillrect
Mar 29 18:05:51 san2  sysimgblt
Mar 29 18:05:51 san2  i2c_algo_bit
Mar 29 18:05:51 san2  drm_kms_helper
Mar 29 18:05:51 san2  ttm
Mar 29 18:05:51 san2  ahci
Mar 29 18:05:51 san2  crc32c_intel
Mar 29 18:05:51 san2  libahci
Mar 29 18:05:51 san2  drm
Mar 29 18:05:51 san2  libata
Mar 29 18:05:51 san2  serio_raw
Mar 29 18:05:51 san2  ixgbe
Mar 29 18:05:51 san2  i2c_core
Mar 29 18:05:51 san2  e1000e
Mar 29 18:05:51 san2  mdio
Mar 29 18:05:51 san2  dca
Mar 29 18:05:51 san2  ptp
Mar 29 18:05:51 san2  arcmsr
Mar 29 18:05:51 san2  pps_core
Mar 29 18:05:51 san2  dm_mirror
Mar 29 18:05:51 san2  dm_region_hash
Mar 29 18:05:51 san2  dm_log
Mar 29 18:05:51 san2  dm_mod
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.797990] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G           O    4.1.20-3.el7.x86_64 #1
Mar 29 18:05:51 san2 [ 2480.799528] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:51 san2 [ 2480.801087] task: ffff8807c01d6e00 ti: ffff8807b7a88000 task.ti: ffff8807b7a88000
Mar 29 18:05:51 san2 [ 2480.802611] RIP: 0010:[<ffffffff8106d197>] 
Mar 29 18:05:51 san2  [<ffffffff8106d197>] change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:51 san2 [ 2480.804179] RSP: 0018:ffff8807b7a8aba8  EFLAGS: 00010046
Mar 29 18:05:51 san2 [ 2480.805731] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000004
Mar 29 18:05:51 san2 [ 2480.807279] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000080000000
Mar 29 18:05:51 san2 [ 2480.808841] RBP: ffff8807b7a8ac58 R08: 80000000c9173101 R09: 00000000000c9173
Mar 29 18:05:51 san2 [ 2480.810373] R10: ffffea001edff0c0 R11: ffffffff813492f9 R12: 0000000000000010
Mar 29 18:05:51 san2 [ 2480.811884] R13: 0000000000000000 R14: 0000000000000200 R15: 0000000000000005
Mar 29 18:05:51 san2 [ 2480.813443] FS:  0000000000000000(0000) GS:ffff88082fdc0000(0000) knlGS:0000000000000000
Mar 29 18:05:51 san2 [ 2480.814951] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 29 18:05:51 san2 [ 2480.816465] CR2: 0000000000000003 CR3: 0000000001a22000 CR4: 00000000001426e0
Mar 29 18:05:51 san2 [ 2480.817981] Stack:
Mar 29 18:05:51 san2 [ 2480.819549]  0000000400000000
Mar 29 18:05:51 san2  0000000000000000
Mar 29 18:05:51 san2  0000000000000000
Mar 29 18:05:51 san2  ffff8806b321d000
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.821098]  00000000c9173000
Mar 29 18:05:51 san2  0000160000000000
Mar 29 18:05:51 san2  0000000000000000
Mar 29 18:05:51 san2  0000000000000000
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.822681]  0000000000000010
Mar 29 18:05:51 san2  0000000000000000
Mar 29 18:05:51 san2  0000000000000001
Mar 29 18:05:51 san2  0000000000000005
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.824197] Call Trace:
Mar 29 18:05:51 san2 [ 2480.825721]  [<ffffffff8106d4e8>] _set_pages_array+0xe8/0x140
Mar 29 18:05:51 san2 [ 2480.827320]  [<ffffffff8106d573>] set_pages_array_wc+0x13/0x20
Mar 29 18:05:51 san2 [ 2480.828921]  [<ffffffffa02d42ef>] ttm_set_pages_caching+0x2f/0x70 [ttm]
Mar 29 18:05:51 san2 [ 2480.830511]  [<ffffffffa02d4434>] ttm_alloc_new_pages.isra.6+0xb4/0x180 [ttm]
Mar 29 18:05:51 san2 [ 2480.832070]  [<ffffffffa02d0e31>] ? ttm_mem_reg_ioremap+0xd1/0x120 [ttm]
Mar 29 18:05:51 san2 [ 2480.833684]  [<ffffffffa02d4dd3>] ttm_pool_populate+0x3f3/0x510 [ttm]
Mar 29 18:05:51 san2 [ 2480.835242]  [<ffffffffa02fedde>] mgag200_ttm_tt_populate+0xe/0x10 [mgag200]
Mar 29 18:05:51 san2 [ 2480.836821]  [<ffffffffa02d184d>] ttm_bo_move_memcpy+0x61d/0x6a0 [ttm]
Mar 29 18:05:51 san2 [ 2480.838346]  [<ffffffffa02fed88>] mgag200_bo_move+0x18/0x20 [mgag200]
Mar 29 18:05:51 san2 [ 2480.839893]  [<ffffffffa02ced95>] ttm_bo_handle_move_mem+0x265/0x5c0 [ttm]
Mar 29 18:05:51 san2 [ 2480.841428]  [<ffffffffa02cf6e7>] ? ttm_bo_mem_space+0xe7/0x350 [ttm]
Mar 29 18:05:51 san2 [ 2480.843005]  [<ffffffffa02cfded>] ttm_bo_validate+0x20d/0x230 [ttm]
Mar 29 18:05:51 san2 [ 2480.844549]  [<ffffffff8106ab14>] ? iounmap+0x84/0xb0
Mar 29 18:05:51 san2 [ 2480.846111]  [<ffffffffa02ff653>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
Mar 29 18:05:51 san2 [ 2480.847626]  [<ffffffffa02faae5>] mga_crtc_do_set_base.isra.8.constprop.20+0x85/0x450 [mgag200]
Mar 29 18:05:51 san2 [ 2480.849171]  [<ffffffff81356c36>] ? delay_tsc+0x46/0x70
Mar 29 18:05:51 san2 [ 2480.850658]  [<ffffffffa02fbf0b>] mga_crtc_mode_set+0x105b/0x21a0 [mgag200]
Mar 29 18:05:51 san2 [ 2480.852200]  [<ffffffffa02364d3>] ? drm_mode_object_get+0x13/0x20 [drm]
Mar 29 18:05:51 san2 [ 2480.853730]  [<ffffffffa03179ad>] drm_crtc_helper_set_mode+0x33d/0x5a0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.855303]  [<ffffffffa0318a42>] drm_crtc_helper_set_config+0x892/0xab0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.856859]  [<ffffffffa02350ff>] drm_mode_set_config_internal+0x6f/0x110 [drm]
Mar 29 18:05:51 san2 [ 2480.858392]  [<ffffffffa0324530>] drm_fb_helper_pan_display+0xa0/0xf0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.859894]  [<ffffffff813b8381>] fb_pan_display+0xd1/0x1a0
Mar 29 18:05:51 san2 [ 2480.861653]  [<ffffffff813b2290>] bit_update_start+0x20/0x50
Mar 29 18:05:51 san2 [ 2480.863651]  [<ffffffff813b0a00>] fbcon_switch+0x3a0/0x5a0
Mar 29 18:05:51 san2 [ 2480.865706]  [<ffffffff81434bf9>] redraw_screen+0x1a9/0x250
Mar 29 18:05:51 san2 [ 2480.867722]  [<ffffffff813af15a>] fbcon_blank+0x22a/0x2f0
Mar 29 18:05:51 san2 [ 2480.869714]  [<ffffffff81194981>] ? irq_work_queue+0x11/0x90
Mar 29 18:05:51 san2 [ 2480.871556]  [<ffffffff810f9df2>] ? wake_up_klogd+0x32/0x40
Mar 29 18:05:51 san2 [ 2480.873270]  [<ffffffff810fa008>] ? console_unlock+0x208/0x480
Mar 29 18:05:51 san2 [ 2480.874940]  [<ffffffff8110b511>] ? internal_add_timer+0x91/0xb0
Mar 29 18:05:51 san2 [ 2480.876561]  [<ffffffff8110db3c>] ? mod_timer+0x10c/0x230
Mar 29 18:05:51 san2 [ 2480.878152]  [<ffffffff81435778>] do_unblank_screen+0xb8/0x1f0
Mar 29 18:05:51 san2 [ 2480.879756]  [<ffffffff814358c0>] unblank_screen+0x10/0x20
Mar 29 18:05:51 san2 [ 2480.881312]  [<ffffffff81359499>] bust_spinlocks+0x19/0x40
Mar 29 18:05:51 san2 [ 2480.882751]  [<ffffffff8101956c>] oops_end+0x3c/0x120
Mar 29 18:05:51 san2 [ 2480.884205]  [<ffffffff816db936>] no_context+0x2ee/0x366
Mar 29 18:05:51 san2 [ 2480.885651]  [<ffffffff816dba21>] __bad_area_nosemaphore+0x73/0x1cc
Mar 29 18:05:51 san2 [ 2480.887088]  [<ffffffff81014693>] ? __switch_to+0x1e3/0x580
Mar 29 18:05:51 san2 [ 2480.888526]  [<ffffffff816dbb8d>] bad_area_nosemaphore+0x13/0x15
Mar 29 18:05:51 san2 [ 2480.889963]  [<ffffffff81069fe6>] __do_page_fault+0x86/0x420
Mar 29 18:05:51 san2 [ 2480.891401]  [<ffffffff8110b7eb>] ? lock_timer_base.isra.35+0x2b/0x50
Mar 29 18:05:51 san2 [ 2480.892750]  [<ffffffff8106a3b0>] do_page_fault+0x30/0x80
Mar 29 18:05:51 san2 [ 2480.894078]  [<ffffffff816eb0d8>] page_fault+0x28/0x30
Mar 29 18:05:51 san2 [ 2480.895374]  [<ffffffff81357a96>] ? memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.896683]  [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:51 san2 [ 2480.898003]  [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:51 san2 [ 2480.899308]  [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:51 san2 [ 2480.900597]  [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:51 san2 [ 2480.901886]  [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:51 san2 [ 2480.903120]  [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:51 san2 [ 2480.904360]  [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:51 san2 [ 2480.905578]  [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:51 san2 [ 2480.906804]  [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.908035]  [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:51 san2 [ 2480.909265]  [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:51 san2 [ 2480.910494]  [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.911716]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.912853]  [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:51 san2 [ 2480.914034]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.915218]  [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:51 san2 [ 2480.916394]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.917564]  [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:51 san2 [ 2480.918706]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.919850] Code: 
Mar 29 18:05:51 san2  
Mar 29 18:05:51 san2 [ 2480.922520] RIP 
Mar 29 18:05:51 san2  [<ffffffff8106d197>] change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:51 san2 [ 2480.923697]  RSP <ffff8807b7a8aba8>
Mar 29 18:05:51 san2 [ 2480.924852] ---[ end trace 6b7ee2c36b3abf19 ]---
Mar 29 18:05:52 san2 [ 2481.050790] Kernel panic - not syncing: Fatal exception
Mar 29 18:05:52 san2 [ 2481.052371] Kernel Offset: disabled
Mar 29 18:05:52 san2 [ 2481.053400] drm_kms_helper: panic occurred, switching back to text console
Mar 29 18:05:52 san2 [ 2481.176995] ---[ end Kernel panic - not syncing: Fatal exception
Mar 29 18:05:52 san2 [ 2481.178078] ------------[ cut here ]------------
Mar 29 18:05:52 san2 [ 2481.179155] WARNING: CPU: 7 PID: 23962 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
Mar 29 18:05:52 san2 [ 2481.180414] Modules linked in:
Mar 29 18:05:52 san2  dm_snapshot
Mar 29 18:05:52 san2  xt_comment
Mar 29 18:05:52 san2  binfmt_misc
Mar 29 18:05:52 san2  xt_CHECKSUM
Mar 29 18:05:52 san2  iptable_mangle
Mar 29 18:05:52 san2  ipt_MASQUERADE
Mar 29 18:05:52 san2  nf_nat_masquerade_ipv4
Mar 29 18:05:52 san2  iptable_nat
Mar 29 18:05:52 san2  nf_nat_ipv4
Mar 29 18:05:52 san2  nf_nat
Mar 29 18:05:52 san2  nf_conntrack_ipv4
Mar 29 18:05:52 san2  nf_defrag_ipv4
Mar 29 18:05:52 san2  xt_conntrack
Mar 29 18:05:52 san2  nf_conntrack
Mar 29 18:05:52 san2  ipt_REJECT
Mar 29 18:05:52 san2  nf_reject_ipv4
Mar 29 18:05:52 san2  ebtable_filter
Mar 29 18:05:52 san2  ebtables
Mar 29 18:05:52 san2  ip6table_filter
Mar 29 18:05:52 san2  ip6_tables
Mar 29 18:05:52 san2  iptable_filter
Mar 29 18:05:52 san2  drbd(O)
Mar 29 18:05:52 san2  xfs
Mar 29 18:05:52 san2  dm_thin_pool
Mar 29 18:05:52 san2  dm_persistent_data
Mar 29 18:05:52 san2  dm_bio_prison
Mar 29 18:05:52 san2  dm_bufio
Mar 29 18:05:52 san2  libcrc32c
Mar 29 18:05:52 san2  bcache
Mar 29 18:05:52 san2  netconsole
Mar 29 18:05:52 san2  zram
Mar 29 18:05:52 san2  lz4_compress
Mar 29 18:05:52 san2  bridge
Mar 29 18:05:52 san2  8021q
Mar 29 18:05:52 san2  garp
Mar 29 18:05:52 san2  mrp
Mar 29 18:05:52 san2  stp
Mar 29 18:05:52 san2  llc
Mar 29 18:05:52 san2  x86_pkg_temp_thermal
Mar 29 18:05:52 san2  intel_powerclamp
Mar 29 18:05:52 san2  coretemp
Mar 29 18:05:52 san2  kvm_intel
Mar 29 18:05:52 san2  kvm
Mar 29 18:05:52 san2  crct10dif_pclmul
Mar 29 18:05:52 san2  iTCO_wdt
Mar 29 18:05:52 san2  crc32_pclmul
Mar 29 18:05:52 san2  iTCO_vendor_support
Mar 29 18:05:52 san2  sg
Mar 29 18:05:52 san2  ipmi_si
Mar 29 18:05:52 san2  ipmi_msghandler
Mar 29 18:05:52 san2  shpchp
Mar 29 18:05:52 san2  i2c_i801
Mar 29 18:05:52 san2  lpc_ich
Mar 29 18:05:52 san2  video
Mar 29 18:05:52 san2  mfd_core
Mar 29 18:05:52 san2  pcspkr
Mar 29 18:05:52 san2  nfsd
Mar 29 18:05:52 san2  auth_rpcgss
Mar 29 18:05:52 san2  nfs_acl
Mar 29 18:05:52 san2  lockd
Mar 29 18:05:52 san2  grace
Mar 29 18:05:52 san2  sunrpc
Mar 29 18:05:52 san2  ip_tables
Mar 29 18:05:52 san2  ext4
Mar 29 18:05:52 san2  mbcache
Mar 29 18:05:52 san2  jbd2
Mar 29 18:05:52 san2  mgag200
Mar 29 18:05:52 san2  syscopyarea
Mar 29 18:05:52 san2  sysfillrect
Mar 29 18:05:52 san2  sysimgblt
Mar 29 18:05:52 san2  i2c_algo_bit
Mar 29 18:05:52 san2  drm_kms_helper
Mar 29 18:05:52 san2  ttm
Mar 29 18:05:52 san2  ahci
Mar 29 18:05:52 san2  crc32c_intel
Mar 29 18:05:52 san2  libahci
Mar 29 18:05:52 san2  drm
Mar 29 18:05:52 san2  libata
Mar 29 18:05:52 san2  serio_raw
Mar 29 18:05:52 san2  ixgbe
Mar 29 18:05:52 san2  i2c_core
Mar 29 18:05:52 san2  e1000e
Mar 29 18:05:52 san2  mdio
Mar 29 18:05:52 san2  dca
Mar 29 18:05:52 san2  ptp
Mar 29 18:05:52 san2  arcmsr
Mar 29 18:05:52 san2  pps_core
Mar 29 18:05:52 san2  dm_mirror
Mar 29 18:05:52 san2  dm_region_hash
Mar 29 18:05:52 san2  dm_log
Mar 29 18:05:52 san2  dm_mod
Mar 29 18:05:52 san2  
Mar 29 18:05:52 san2 [ 2481.190970] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G      D    O    4.1.20-3.el7.x86_64 #1
Mar 29 18:05:52 san2 [ 2481.192411] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:52 san2 [ 2481.193863]  0000000000000086
Mar 29 18:05:52 san2  0000000088dca908
Mar 29 18:05:52 san2  ffff88082fdc3d58
Mar 29 18:05:52 san2  ffffffff816e16c4
Mar 29 18:05:52 san2  
Mar 29 18:05:52 san2 [ 2481.195340]  0000000000000000
Mar 29 18:05:52 san2  ffffffff8191df1e
Mar 29 18:05:52 san2  ffff88082fdc3d98
Mar 29 18:05:52 san2  ffffffff810a0dea
Mar 29 18:05:52 san2  
Mar 29 18:05:52 san2 [ 2481.196816]  ffff88082fdc3d88
Mar 29 18:05:52 san2  0000000000000000
Mar 29 18:05:52 san2  ffff88082fc177c0
Mar 29 18:05:52 san2  0000000000000007
Mar 29 18:05:52 san2  
Mar 29 18:05:52 san2 [ 2481.198295] Call Trace:
Mar 29 18:05:52 san2 [ 2481.199748]  <IRQ> 
Mar 29 18:05:52 san2  [<ffffffff816e16c4>] dump_stack+0x63/0x81
Mar 29 18:05:52 san2 [ 2481.201229]  [<ffffffff810a0dea>] warn_slowpath_common+0x8a/0xc0
Mar 29 18:05:52 san2 [ 2481.202720]  [<ffffffff810a0f1a>] warn_slowpath_null+0x1a/0x20
Mar 29 18:05:52 san2 [ 2481.204205]  [<ffffffff8104fd9d>] native_smp_send_reschedule+0x5d/0x60
Mar 29 18:05:52 san2 [ 2481.205693]  [<ffffffff810e0b15>] trigger_load_balance+0x145/0x1f0
Mar 29 18:05:52 san2 [ 2481.207187]  [<ffffffff810cde6c>] scheduler_tick+0x9c/0xe0
Mar 29 18:05:52 san2 [ 2481.208681]  [<ffffffff8110df41>] update_process_times+0x51/0x60
Mar 29 18:05:52 san2 [ 2481.210175]  [<ffffffff8111e525>] tick_sched_handle.isra.18+0x25/0x60
Mar 29 18:05:52 san2 [ 2481.211670]  [<ffffffff8111e5a4>] tick_sched_timer+0x44/0x80
Mar 29 18:05:52 san2 [ 2481.213163]  [<ffffffff8110ed27>] __run_hrtimer+0x77/0x220
Mar 29 18:05:52 san2 [ 2481.214650]  [<ffffffff8111e560>] ? tick_sched_handle.isra.18+0x60/0x60
Mar 29 18:05:52 san2 [ 2481.216145]  [<ffffffff8110f153>] hrtimer_interrupt+0x103/0x230
Mar 29 18:05:52 san2 [ 2481.217633]  [<ffffffff81052b69>] local_apic_timer_interrupt+0x39/0x60
Mar 29 18:05:52 san2 [ 2481.219123]  [<ffffffff816ebf35>] smp_apic_timer_interrupt+0x45/0x60
Mar 29 18:05:52 san2 [ 2481.220609]  [<ffffffff816e9fbe>] apic_timer_interrupt+0x6e/0x80
Mar 29 18:05:52 san2 [ 2481.222092]  <EOI> 
Mar 29 18:05:52 san2  [<ffffffff816dc166>] ? panic+0x1cd/0x20e
Mar 29 18:05:52 san2 [ 2481.223583]  [<ffffffff816dc15f>] ? panic+0x1c6/0x20e
Mar 29 18:05:52 san2 [ 2481.225064]  [<ffffffff81019639>] oops_end+0x109/0x120
Mar 29 18:05:52 san2 [ 2481.226536]  [<ffffffff81019beb>] die+0x4b/0x70
Mar 29 18:05:52 san2 [ 2481.227994]  [<ffffffff81015fac>] do_trap+0x14c/0x160
Mar 29 18:05:52 san2 [ 2481.229447]  [<ffffffff8101649c>] do_error_trap+0xac/0x190
Mar 29 18:05:52 san2 [ 2481.230892]  [<ffffffff8106d197>] ? change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:52 san2 [ 2481.232345]  [<ffffffff81070f28>] ? do_flush_tlb_all+0x48/0x50
Mar 29 18:05:52 san2 [ 2481.233769]  [<ffffffff8106c018>] ? lookup_address+0x28/0x30
Mar 29 18:05:52 san2 [ 2481.235160]  [<ffffffff8106c0db>] ? _lookup_address_cpa.isra.8+0x3b/0x40
Mar 29 18:05:52 san2 [ 2481.236540]  [<ffffffff8106c8fa>] ? __change_page_attr_set_clr+0x81a/0xba0
Mar 29 18:05:52 san2 [ 2481.237911]  [<ffffffff81016c50>] do_invalid_op+0x20/0x30
Mar 29 18:05:52 san2 [ 2481.239277]  [<ffffffff816eaa7e>] invalid_op+0x1e/0x30
Mar 29 18:05:52 san2 [ 2481.240635]  [<ffffffff813492f9>] ? free_cpumask_var+0x9/0x10
Mar 29 18:05:52 san2 [ 2481.241996]  [<ffffffff8106d197>] ? change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:52 san2 [ 2481.243351]  [<ffffffff8106d4e8>] _set_pages_array+0xe8/0x140
Mar 29 18:05:52 san2 [ 2481.244697]  [<ffffffff8106d573>] set_pages_array_wc+0x13/0x20
Mar 29 18:05:52 san2 [ 2481.246044]  [<ffffffffa02d42ef>] ttm_set_pages_caching+0x2f/0x70 [ttm]
Mar 29 18:05:52 san2 [ 2481.247373]  [<ffffffffa02d4434>] ttm_alloc_new_pages.isra.6+0xb4/0x180 [ttm]
Mar 29 18:05:52 san2 [ 2481.248671]  [<ffffffffa02d0e31>] ? ttm_mem_reg_ioremap+0xd1/0x120 [ttm]
Mar 29 18:05:52 san2 [ 2481.249936]  [<ffffffffa02d4dd3>] ttm_pool_populate+0x3f3/0x510 [ttm]
Mar 29 18:05:52 san2 [ 2481.251168]  [<ffffffffa02fedde>] mgag200_ttm_tt_populate+0xe/0x10 [mgag200]
Mar 29 18:05:52 san2 [ 2481.252368]  [<ffffffffa02d184d>] ttm_bo_move_memcpy+0x61d/0x6a0 [ttm]
Mar 29 18:05:52 san2 [ 2481.253534]  [<ffffffffa02fed88>] mgag200_bo_move+0x18/0x20 [mgag200]
Mar 29 18:05:52 san2 [ 2481.254667]  [<ffffffffa02ced95>] ttm_bo_handle_move_mem+0x265/0x5c0 [ttm]
Mar 29 18:05:52 san2 [ 2481.255766]  [<ffffffffa02cf6e7>] ? ttm_bo_mem_space+0xe7/0x350 [ttm]
Mar 29 18:05:52 san2 [ 2481.256832]  [<ffffffffa02cfded>] ttm_bo_validate+0x20d/0x230 [ttm]
Mar 29 18:05:52 san2 [ 2481.257871]  [<ffffffff8106ab14>] ? iounmap+0x84/0xb0
Mar 29 18:05:52 san2 [ 2481.258888]  [<ffffffffa02ff653>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
Mar 29 18:05:52 san2 [ 2481.259900]  [<ffffffffa02faae5>] mga_crtc_do_set_base.isra.8.constprop.20+0x85/0x450 [mgag200]
Mar 29 18:05:52 san2 [ 2481.260941]  [<ffffffff81356c36>] ? delay_tsc+0x46/0x70
Mar 29 18:05:52 san2 [ 2481.261975]  [<ffffffffa02fbf0b>] mga_crtc_mode_set+0x105b/0x21a0 [mgag200]
Mar 29 18:05:52 san2 [ 2481.263022]  [<ffffffffa02364d3>] ? drm_mode_object_get+0x13/0x20 [drm]
Mar 29 18:05:52 san2 [ 2481.264055]  [<ffffffffa03179ad>] drm_crtc_helper_set_mode+0x33d/0x5a0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.265099]  [<ffffffffa0318a42>] drm_crtc_helper_set_config+0x892/0xab0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.266161]  [<ffffffffa02350ff>] drm_mode_set_config_internal+0x6f/0x110 [drm]
Mar 29 18:05:52 san2 [ 2481.267217]  [<ffffffffa0324530>] drm_fb_helper_pan_display+0xa0/0xf0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.268276]  [<ffffffff813b8381>] fb_pan_display+0xd1/0x1a0
Mar 29 18:05:52 san2 [ 2481.269333]  [<ffffffff813b2290>] bit_update_start+0x20/0x50
Mar 29 18:05:52 san2 [ 2481.270386]  [<ffffffff813b0a00>] fbcon_switch+0x3a0/0x5a0
Mar 29 18:05:52 san2 [ 2481.271434]  [<ffffffff81434bf9>] redraw_screen+0x1a9/0x250
Mar 29 18:05:52 san2 [ 2481.272478]  [<ffffffff813af15a>] fbcon_blank+0x22a/0x2f0
Mar 29 18:05:52 san2 [ 2481.273526]  [<ffffffff81194981>] ? irq_work_queue+0x11/0x90
Mar 29 18:05:52 san2 [ 2481.274571]  [<ffffffff810f9df2>] ? wake_up_klogd+0x32/0x40
Mar 29 18:05:52 san2 [ 2481.275612]  [<ffffffff810fa008>] ? console_unlock+0x208/0x480
Mar 29 18:05:52 san2 [ 2481.276649]  [<ffffffff8110b511>] ? internal_add_timer+0x91/0xb0
Mar 29 18:05:52 san2 [ 2481.277684]  [<ffffffff8110db3c>] ? mod_timer+0x10c/0x230
Mar 29 18:05:52 san2 [ 2481.278714]  [<ffffffff81435778>] do_unblank_screen+0xb8/0x1f0
Mar 29 18:05:52 san2 [ 2481.279736]  [<ffffffff814358c0>] unblank_screen+0x10/0x20
Mar 29 18:05:52 san2 [ 2481.280749]  [<ffffffff81359499>] bust_spinlocks+0x19/0x40
Mar 29 18:05:52 san2 [ 2481.281764]  [<ffffffff8101956c>] oops_end+0x3c/0x120
Mar 29 18:05:52 san2 [ 2481.282781]  [<ffffffff816db936>] no_context+0x2ee/0x366
Mar 29 18:05:52 san2 [ 2481.283799]  [<ffffffff816dba21>] __bad_area_nosemaphore+0x73/0x1cc
Mar 29 18:05:52 san2 [ 2481.284828]  [<ffffffff81014693>] ? __switch_to+0x1e3/0x580
Mar 29 18:05:52 san2 [ 2481.285851]  [<ffffffff816dbb8d>] bad_area_nosemaphore+0x13/0x15
Mar 29 18:05:52 san2 [ 2481.286874]  [<ffffffff81069fe6>] __do_page_fault+0x86/0x420
Mar 29 18:05:52 san2 [ 2481.287897]  [<ffffffff8110b7eb>] ? lock_timer_base.isra.35+0x2b/0x50
Mar 29 18:05:52 san2 [ 2481.288925]  [<ffffffff8106a3b0>] do_page_fault+0x30/0x80
Mar 29 18:05:52 san2 [ 2481.289948]  [<ffffffff816eb0d8>] page_fault+0x28/0x30
Mar 29 18:05:52 san2 [ 2481.290966]  [<ffffffff81357a96>] ? memcpy_erms+0x6/0x10
Mar 29 18:05:52 san2 [ 2481.291982]  [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:52 san2 [ 2481.293002]  [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:52 san2 [ 2481.294012]  [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:52 san2 [ 2481.295017]  [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:52 san2 [ 2481.296031]  [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:52 san2 [ 2481.297036]  [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:52 san2 [ 2481.298042]  [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:52 san2 [ 2481.299047]  [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:52 san2 [ 2481.300059]  [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:52 san2 [ 2481.301057]  [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:52 san2 [ 2481.302066]  [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:52 san2 [ 2481.303071]  [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:52 san2 [ 2481.304072]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:52 san2 [ 2481.305082]  [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:52 san2 [ 2481.306090]  [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:52 san2 [ 2481.307101]  [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:52 san2 [ 2481.308097]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:52 san2 [ 2481.309096]  [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:52 san2 [ 2481.310066]  [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:52 san2 [ 2481.311043] ---[ end trace 6b7ee2c36b3abf1a ]---
-------------- next part --------------
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: PingAck did not arrive in time.
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: error receiving RSDataReply, e: -5 l: 65536!
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: ack_receiver terminated
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Terminating drbd_a_www3.ewh
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Connection closed
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: receiver terminated
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Restarting receiver thread
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: receiver (re)started
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: conn( Unconnected -> WFConnection ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: error receiving RSDataReply, e: -5 l: 32768!
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Terminating drbd_a_int-pbx.
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: conn( Unconnected -> WFConnection ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: error receiving RSDataReply, e: -5 l: 24576!
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Terminating drbd_a_rsinigsu
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: error receiving RSDataReply, e: -5 l: 61440!
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: conn( Unconnected -> WFConnection ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Terminating drbd_a_gls-moni
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: conn( Unconnected -> WFConnection ) 
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: PingAck did not arrive in time.
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: error receiving RSDataReply, e: -5 l: 8192!
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: ack_receiver terminated
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Terminating drbd_a_spuprot.
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Connection closed
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: receiver terminated
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Restarting receiver thread
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: receiver (re)started
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: conn( Unconnected -> WFConnection ) 
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: PingAck did not arrive in time.
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: error receiving RSDataReply, e: -5 l: 4096!
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: ack_receiver terminated
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Terminating drbd_a_mail.ewh
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Connection closed
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: conn( NetworkFailure -> Unconnected ) 
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: receiver terminated
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Restarting receiver thread
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: receiver (re)started
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: conn( Unconnected -> WFConnection ) 


More information about the drbd-dev mailing list