[Drbd-dev] NULL pointer derefernce in 8.4.7-1 during drbd_destroy_connection()
Eric Wheeler
drbd-dev at lists.ewheeler.net
Wed Mar 30 21:52:09 CEST 2016
On Wed, 30 Mar 2016, Lars Ellenberg wrote:
> On Wed, Mar 30, 2016 at 02:19:07AM +0000, Eric Wheeler wrote:
> > Hello all,
> >
> > We are getting kernel crashes in linux 4.1.20 with the drbd-8.4.git tree
> > at commit 3a6a769340ef93b1ba2792c6461250790795db49 .
> >
> > I don't see anything in the newer commits that addresses this issue so
> > I'm posting---but I'll try the latest commit in master, too, just in case.
> >
> > Please see the backtrace below. I also included our global_common.conf
> > further down. This is protocol A and the link is quite slow. This NULL
> > ptr dereference appears to show up when the drbd kernel thread is blocked
> > for a long time. It might happen at reconnect time because the BUG didn't
> > show up until 13 seconds after the P_BARRIER error.
> >
> > The problem is pretty reproducable, so I can probably test patches.
> > Please let me know what I can do to help test.
>
> DRBD logs of both peers leading up to the incident may be useful.
See attached for the side that locked up at 18:05:51. The first line
starts at of the sending peer is 18:04:30 PST for remote correlation.
The first line on the receiving peer is Mar 29 18:06:04 (15s after lockup)
and both machines are ntp slaved.
The receiving side has the same module version but doesn't have any logs
for 15 mins before the lockup, and the only logs after the lockup are
"PingAck did not arrive in time." with related retries, but attached for
reference.
Note that these are blank volumes on the receiver. We just create-md'ed
and started a fresh sync with proto A to move volumes to a different
datacenter.
> check if older kernel versions are ok?
> as in 2.6.32, 3.10, ...
> if older seems to be ok, figure out which version breaks.
>
> maybe check if older DRBD is still ok (maybe this is a more recent regression?)
I might be able to try earlier kernels, will see. This is el7, not sure
if I can go earlier than 3.10 for possible userspace requirements.
> try to resolve addresses to source code lines.
These correlate to the trace below in backtrace order. It looks like a
problem with drbd teardown since the bottom of the trace stack calls to
drbd_destroy_connection:
(gdb) list *(drbd_send+0xe6)
0x29f56 is in drbd_send (drbd/drbd_main.c:1913).
1908 rcu_read_unlock();
1909 drbd_update_congested(connection);
1910 }
1911 do {
1912 rv = kernel_sendmsg(sock, &msg, &iov, 1, size); <<<<<< Leaves DRBD
1913 if (rv == -EAGAIN) {
1914 if (we_should_drop_the_connection(connection, sock))
1915 break;
1916 else
1917 continue;
(gdb) list *(_drbd_no_send_page.isra.40+0x71)
A syntax error in expression, near `.40+0x71)'.
(gdb) list *(drbd_send_dblock+0x3e8)
0x2c1a8 is in drbd_send_dblock (drbd/drbd_main.c:1646).
1641 int err;
1642
1643 err = _drbd_no_send_page(peer_device, bvec BVD bv_page,
1644 bvec BVD bv_offset, bvec BVD bv_len,
1645 bio_iter_last(bvec, iter) ? 0 : MSG_MORE);
1646 if (err)
1647 return err;
1648 /* REQ_WRITE_SAME has only one segment */
1649 if (bio->bi_rw & DRBD_REQ_WSAME)
1650 break;
(gdb) list *(complete_master_bio+0x94)
0x1e8a4 is in complete_master_bio (drbd/drbd_req.c:227).
222 void complete_master_bio(struct drbd_device *device,
223 struct bio_and_error *m)
224 {
225 bio_endio(m->bio, m->error);
226 dec_ap_bio(device);
227 }
228
229
230 /* Helper for __req_mod().
231 * Set m->bio to the master bio, if it is fit to be completed,
(gdb) list *(w_send_dblock+0xaf)
0xc5ff is in w_send_dblock (drbd/drbd_req.h:321).
316 * If you need it irqsave, do it your self!
317 * Which means: don't use from bio endio callback. */
318 static inline int req_mod(struct drbd_request *req,
319 enum drbd_req_event what)
320 {
321 struct drbd_device *device = req->device;
322 struct bio_and_error m;
323 int rv;
324
325 spin_lock_irq(&device->resource->req_lock);
(gdb) list *(drbd_worker+0xf9)
0xd9d9 is in drbd_worker (drbd/drbd_worker.c:2205).
2200
2201 if (!list_empty(&work_list)) {
2202 w = list_first_entry(&work_list, struct drbd_work, list);
2203 list_del_init(&w->list);
2204 update_worker_timing_details(connection, w->cb);
2205 if (w->cb(w, connection->cstate < C_WF_REPORT_PARAMS) == 0)
2206 continue;
2207 if (connection->cstate >= C_WF_REPORT_PARAMS)
2208 conn_request_state(connection, NS(conn, C_NETWORK_FAILURE), CS_HARD);
2209 }
(gdb) list *(drbd_destroy_connection+0x190)
0x27d30 is in drbd_thread_setup (drbd/drbd_main.c:362).
357 }
358 spin_unlock_irq(&connection->resource->req_lock);
359 }
360
361 static int drbd_thread_setup(void *arg)
362 {
363 struct drbd_thread *thi = (struct drbd_thread *) arg;
364 struct drbd_resource *resource = thi->resource;
365 unsigned long flags;
366 int retval;
(gdb) list *(drbd_thread_setup+0x1d)
0x27d4d is in drbd_thread_setup (drbd/drbd_main.c:371).
366 int retval;
367
368 restart:
369 retval = thi->function(thi);
370
371 spin_lock_irqsave(&thi->t_lock, flags);
372
373 /* if the receiver has been "EXITING", the last thing it did
374 * was set the conn state to "StandAlone",
375 * if now a re-connect request comes in, conn state goes C_UNCONNECTED,
(gdb) list *(drbd_destroy_connection+0x190)
0x27d30 is in drbd_thread_setup (drbd/drbd_main.c:362).
357 }
358 spin_unlock_irq(&connection->resource->req_lock);
359 }
360
361 static int drbd_thread_setup(void *arg)
362 {
363 struct drbd_thread *thi = (struct drbd_thread *) arg;
364 struct drbd_resource *resource = thi->resource;
365 unsigned long flags;
366 int retval;
>
> > [ 2480.751713] [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
> > [ 2480.753608] [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
> > [ 2480.755463] [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
> > [ 2480.757263] [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
> > [ 2480.759073] [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
> > [ 2480.760844] [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
> > [ 2480.762567] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
> > [ 2480.764181] [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
> > [ 2480.765777] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
> > [ 2480.767337] [<ffffffff810c0b08>] kthread+0xd8/0xf0
> > [ 2480.768873] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
> > [ 2480.770409] [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
> > [ 2480.771868] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
> >
> >
> > ===> /etc/drbd.d/global_common.conf <===
> > common {
> > startup {
> > wfc-timeout 30;
> > outdated-wfc-timeout 20;
> > degr-wfc-timeout 30;
> > }
> > options {
> > on-no-data-accessible suspend-io;
> > }
> > syncer {
> > rate 500M;
> > }
> > disk {
> > al-extents 3389;
> > c-fill-target 10240;
> > c-delay-target 100;
> > c-plan-ahead 70;
> > c-min-rate 1024;
> > c-max-rate 400M;
> > on-io-error pass_on;
> > read-balancing when-congested-remote;
> > }
> > net {
> > after-sb-0pri discard-zero-changes;
> > after-sb-1pri call-pri-lost-after-sb;
> > after-sb-2pri disconnect;
> > allow-two-primaries no;
> > protocol A;
> > cram-hmac-alg sha1;
> > verify-alg crc32c;
> > csums-alg crc32c;
> > max-buffers 8192;
> > max-epoch-size 8192;
> > tcp-cork yes;
> > sndbuf-size 1M;
> > rcvbuf-size 2M;
> > unplug-watermark 128;
> > ko-count 3;
> > timeout 90;
> >
> > ping-int 10;
> > ping-timeout 30;
> > }
> > }
>
--
Eric Wheeler
> --
> : Lars Ellenberg
> : LINBIT | Keeping the Digital World Running
> : DRBD -- Heartbeat -- Corosync -- Pacemaker
> : R&D, Integration, Ops, Consulting, Support
>
> DRBD® and LINBIT® are registered trademarks of LINBIT
> _______________________________________________
> drbd-dev mailing list
> drbd-dev at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-dev
>
-------------- next part --------------
Mar 29 18:04:30 san2 [ 2399.426673] block drbd7935: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?
Mar 29 18:04:30 san2 [ 2399.431035] block drbd7935: drbd_sync_handshake:
Mar 29 18:04:30 san2 [ 2399.433091] block drbd7935: self D5FCC36B7DB360CA:0000000000000000:D0FDFBD85DF1B5A5:D0FCFBD85DF1B5A5 bits:879616 flags:0
Mar 29 18:04:30 san2 [ 2399.435147] block drbd7935: peer 05835EB3EEC6BD5D:D5FCC36B7DB360CA:D0FDFBD85DF1B5A4:D0FCFBD85DF1B5A5 bits:20590 flags:0
Mar 29 18:04:30 san2 [ 2399.437178] block drbd7935: uuid_compare()=-1 by rule 50
Mar 29 18:04:30 san2 [ 2399.439181] block drbd7935: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )
Mar 29 18:04:30 san2 [ 2399.451110] block drbd7935: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 839(1), total 839; compression: 99.9%
Mar 29 18:04:30 san2 [ 2399.453159] block drbd7935: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 600(1), total 600; compression: 99.9%
Mar 29 18:04:30 san2 [ 2399.454990] block drbd7935: conn( WFBitMapT -> WFSyncUUID )
Mar 29 18:04:30 san2 [ 2399.488131] block drbd7935: updated sync uuid D5FDC36B7DB360CA:0000000000000000:D0FDFBD85DF1B5A5:D0FCFBD85DF1B5A5
Mar 29 18:04:30 san2 [ 2399.488806] block drbd7935: helper command: /sbin/drbdadm before-resync-target minor-7935
Mar 29 18:04:30 san2 [ 2399.499781] block drbd7935: helper command: /sbin/drbdadm before-resync-target minor-7935 exit code 0 (0x0)
Mar 29 18:04:30 san2 [ 2399.501347] block drbd7935: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Mar 29 18:04:30 san2 [ 2399.502902] block drbd7935: Began resync as SyncTarget (will sync 3540040 KB [885010 bits set]).
Mar 29 18:04:40 san2 [ 2409.256237] block drbd7: We did not send a P_BARRIER for 27002ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:41 san2 [ 2410.178203] block drbd24: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:42 san2 [ 2411.278153] block drbd7994: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:04:42 san2 [ 2411.422145] block drbd7945: We did not send a P_BARRIER for 37575ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:09 san2 [ 2438.881042] block drbd7994: We did not send a P_BARRIER for 27001ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:22 san2 [ 2451.804514] block drbd7945: We did not send a P_BARRIER for 27044ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:29 san2 [ 2458.780210] block drbd15: We did not send a P_BARRIER for 27038ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:38 san2 [ 2467.187849] block drbd7994: We did not send a P_BARRIER for 27003ms > ko-count (3) * timeout (90 * 0.1s); drbd kernel thread blocked?
Mar 29 18:05:51 san2 [ 2480.674208] BUG: unable to handle kernel
Mar 29 18:05:51 san2 at 0000000000000003
Mar 29 18:05:51 san2 [ 2480.676403] IP:
Mar 29 18:05:51 san2 [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.678547] PGD 0
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.680628] Oops: 0000 [#1]
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.682675] Modules linked in:
Mar 29 18:05:51 san2 dm_snapshot
Mar 29 18:05:51 san2 xt_comment
Mar 29 18:05:51 san2 binfmt_misc
Mar 29 18:05:51 san2 xt_CHECKSUM
Mar 29 18:05:51 san2 iptable_mangle
Mar 29 18:05:51 san2 ipt_MASQUERADE
Mar 29 18:05:51 san2 nf_nat_masquerade_ipv4
Mar 29 18:05:51 san2 iptable_nat
Mar 29 18:05:51 san2 nf_nat_ipv4
Mar 29 18:05:51 san2 nf_nat
Mar 29 18:05:51 san2 nf_conntrack_ipv4
Mar 29 18:05:51 san2 nf_defrag_ipv4
Mar 29 18:05:51 san2 xt_conntrack
Mar 29 18:05:51 san2 nf_conntrack
Mar 29 18:05:51 san2 ipt_REJECT
Mar 29 18:05:51 san2 nf_reject_ipv4
Mar 29 18:05:51 san2 ebtable_filter
Mar 29 18:05:51 san2 ebtables
Mar 29 18:05:51 san2 ip6table_filter
Mar 29 18:05:51 san2 ip6_tables
Mar 29 18:05:51 san2 iptable_filter
Mar 29 18:05:51 san2 drbd(O)
Mar 29 18:05:51 san2 xfs
Mar 29 18:05:51 san2 dm_thin_pool
Mar 29 18:05:51 san2 dm_persistent_data
Mar 29 18:05:51 san2 dm_bio_prison
Mar 29 18:05:51 san2 dm_bufio
Mar 29 18:05:51 san2 libcrc32c
Mar 29 18:05:51 san2 bcache
Mar 29 18:05:51 san2 netconsole
Mar 29 18:05:51 san2 zram
Mar 29 18:05:51 san2 lz4_compress
Mar 29 18:05:51 san2 bridge
Mar 29 18:05:51 san2 8021q
Mar 29 18:05:51 san2 garp
Mar 29 18:05:51 san2 mrp
Mar 29 18:05:51 san2 stp
Mar 29 18:05:51 san2 llc
Mar 29 18:05:51 san2 x86_pkg_temp_thermal
Mar 29 18:05:51 san2 intel_powerclamp
Mar 29 18:05:51 san2 coretemp
Mar 29 18:05:51 san2 kvm_intel
Mar 29 18:05:51 san2 kvm
Mar 29 18:05:51 san2 crct10dif_pclmul
Mar 29 18:05:51 san2 iTCO_wdt
Mar 29 18:05:51 san2 crc32_pclmul
Mar 29 18:05:51 san2 iTCO_vendor_support
Mar 29 18:05:51 san2 sg
Mar 29 18:05:51 san2 ipmi_si
Mar 29 18:05:51 san2 ipmi_msghandler
Mar 29 18:05:51 san2 shpchp
Mar 29 18:05:51 san2 i2c_i801
Mar 29 18:05:51 san2 lpc_ich
Mar 29 18:05:51 san2 video
Mar 29 18:05:51 san2 mfd_core
Mar 29 18:05:51 san2 pcspkr
Mar 29 18:05:51 san2 nfsd
Mar 29 18:05:51 san2 auth_rpcgss
Mar 29 18:05:51 san2 nfs_acl
Mar 29 18:05:51 san2 lockd
Mar 29 18:05:51 san2 grace
Mar 29 18:05:51 san2 sunrpc
Mar 29 18:05:51 san2 ip_tables
Mar 29 18:05:51 san2 ext4
Mar 29 18:05:51 san2 mbcache
Mar 29 18:05:51 san2 jbd2
Mar 29 18:05:51 san2 mgag200
Mar 29 18:05:51 san2 syscopyarea
Mar 29 18:05:51 san2 sysfillrect
Mar 29 18:05:51 san2 sysimgblt
Mar 29 18:05:51 san2 i2c_algo_bit
Mar 29 18:05:51 san2 drm_kms_helper
Mar 29 18:05:51 san2 ttm
Mar 29 18:05:51 san2 ahci
Mar 29 18:05:51 san2 crc32c_intel
Mar 29 18:05:51 san2 libahci
Mar 29 18:05:51 san2 drm
Mar 29 18:05:51 san2 libata
Mar 29 18:05:51 san2 serio_raw
Mar 29 18:05:51 san2 ixgbe
Mar 29 18:05:51 san2 i2c_core
Mar 29 18:05:51 san2 e1000e
Mar 29 18:05:51 san2 mdio
Mar 29 18:05:51 san2 dca
Mar 29 18:05:51 san2 ptp
Mar 29 18:05:51 san2 arcmsr
Mar 29 18:05:51 san2 pps_core
Mar 29 18:05:51 san2 dm_mirror
Mar 29 18:05:51 san2 dm_region_hash
Mar 29 18:05:51 san2 dm_log
Mar 29 18:05:51 san2 dm_mod
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.700341] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G O 4.1.20-3.el7.x86_64 #1
Mar 29 18:05:51 san2 [ 2480.702612] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:51 san2 [ 2480.704921] task: ffff8807c01d6e00 ti: ffff8807b7a88000 task.ti: ffff8807b7a88000
Mar 29 18:05:51 san2 [ 2480.707206] RIP: 0010:[<ffffffff81357a96>]
Mar 29 18:05:51 san2 [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.709537] RSP: 0018:ffff8807b7a8ba50 EFLAGS: 00010286
Mar 29 18:05:51 san2 [ 2480.711774] RAX: ffff8807dc0cc4d8 RBX: 00000000000004a6 RCX: 00000000000004a6
Mar 29 18:05:51 san2 [ 2480.714079] RDX: 00000000000004a6 RSI: 0000000000000003 RDI: ffff8807dc0cc4d8
Mar 29 18:05:51 san2 [ 2480.716324] RBP: ffff8807b7a8ba98 R08: ffff8807b7a8bbd8 R09: ffff8807c01d7b28
Mar 29 18:05:51 san2 [ 2480.718605] R10: 0000000000000000 R11: ffff88075891b800 R12: 0000000000000a50
Mar 29 18:05:51 san2 [ 2480.720853] R13: ffff8807b7a8bbf8 R14: ffff8807b7a8bbf8 R15: 0000000000000000
Mar 29 18:05:51 san2 [ 2480.723052] FS: 0000000000000000(0000) GS:ffff88082fdc0000(0000) knlGS:0000000000000000
Mar 29 18:05:51 san2 [ 2480.725221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 29 18:05:51 san2 [ 2480.727436] CR2: 0000000000000003 CR3: 0000000001a22000 CR4: 00000000001426e0
Mar 29 18:05:51 san2 [ 2480.729500] Stack:
Mar 29 18:05:51 san2 [ 2480.731538] ffffffff8135c40f
Mar 29 18:05:51 san2 ffff8807b7a8bbd8
Mar 29 18:05:51 san2 ffff8807dc0cc97e
Mar 29 18:05:51 san2 ffff8807b7a8ba98
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.733562] ffff8807738cb600
Mar 29 18:05:51 san2 ffff8807f7949000
Mar 29 18:05:51 san2 000000000000fa9b
Mar 29 18:05:51 san2 ffff8807b7a8bbe8
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.735620] 0000000000000a50
Mar 29 18:05:51 san2 ffff8807b7a8bb48
Mar 29 18:05:51 san2 ffffffff8161b76a
Mar 29 18:05:51 san2 ffff88070000000c
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.737704] Call Trace:
Mar 29 18:05:51 san2 [ 2480.739750] [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:51 san2 [ 2480.741834] [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:51 san2 [ 2480.743914] [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:51 san2 [ 2480.745905] [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:51 san2 [ 2480.747877] [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:51 san2 [ 2480.749813] [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:51 san2 [ 2480.751713] [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:51 san2 [ 2480.753608] [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:51 san2 [ 2480.755463] [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.757263] [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:51 san2 [ 2480.759073] [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:51 san2 [ 2480.760844] [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.762567] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.764181] [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:51 san2 [ 2480.765777] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.767337] [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:51 san2 [ 2480.768873] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.770409] [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:51 san2 [ 2480.771868] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.773358] Code:
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.776563] RIP
Mar 29 18:05:51 san2 [<ffffffff81357a96>] memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.778101] RSP <ffff8807b7a8ba50>
Mar 29 18:05:51 san2 [ 2480.779584] CR2: 0000000000000003
Mar 29 18:05:51 san2 [ 2480.783016] ------------[ cut here ]------------
Mar 29 18:05:51 san2 [ 2480.784328] kernel BUG at arch/x86/mm/pageattr.c:214!
Mar 29 18:05:51 san2 [ 2480.785605] invalid opcode: 0000 [#2]
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.786849] Modules linked in:
Mar 29 18:05:51 san2 dm_snapshot
Mar 29 18:05:51 san2 xt_comment
Mar 29 18:05:51 san2 binfmt_misc
Mar 29 18:05:51 san2 xt_CHECKSUM
Mar 29 18:05:51 san2 iptable_mangle
Mar 29 18:05:51 san2 ipt_MASQUERADE
Mar 29 18:05:51 san2 nf_nat_masquerade_ipv4
Mar 29 18:05:51 san2 iptable_nat
Mar 29 18:05:51 san2 nf_nat_ipv4
Mar 29 18:05:51 san2 nf_nat
Mar 29 18:05:51 san2 nf_conntrack_ipv4
Mar 29 18:05:51 san2 nf_defrag_ipv4
Mar 29 18:05:51 san2 xt_conntrack
Mar 29 18:05:51 san2 nf_conntrack
Mar 29 18:05:51 san2 ipt_REJECT
Mar 29 18:05:51 san2 nf_reject_ipv4
Mar 29 18:05:51 san2 ebtable_filter
Mar 29 18:05:51 san2 ebtables
Mar 29 18:05:51 san2 ip6table_filter
Mar 29 18:05:51 san2 ip6_tables
Mar 29 18:05:51 san2 iptable_filter
Mar 29 18:05:51 san2 drbd(O)
Mar 29 18:05:51 san2 xfs
Mar 29 18:05:51 san2 dm_thin_pool
Mar 29 18:05:51 san2 dm_persistent_data
Mar 29 18:05:51 san2 dm_bio_prison
Mar 29 18:05:51 san2 dm_bufio
Mar 29 18:05:51 san2 libcrc32c
Mar 29 18:05:51 san2 bcache
Mar 29 18:05:51 san2 netconsole
Mar 29 18:05:51 san2 zram
Mar 29 18:05:51 san2 lz4_compress
Mar 29 18:05:51 san2 bridge
Mar 29 18:05:51 san2 8021q
Mar 29 18:05:51 san2 garp
Mar 29 18:05:51 san2 mrp
Mar 29 18:05:51 san2 stp
Mar 29 18:05:51 san2 llc
Mar 29 18:05:51 san2 x86_pkg_temp_thermal
Mar 29 18:05:51 san2 intel_powerclamp
Mar 29 18:05:51 san2 coretemp
Mar 29 18:05:51 san2 kvm_intel
Mar 29 18:05:51 san2 kvm
Mar 29 18:05:51 san2 crct10dif_pclmul
Mar 29 18:05:51 san2 iTCO_wdt
Mar 29 18:05:51 san2 crc32_pclmul
Mar 29 18:05:51 san2 iTCO_vendor_support
Mar 29 18:05:51 san2 sg
Mar 29 18:05:51 san2 ipmi_si
Mar 29 18:05:51 san2 ipmi_msghandler
Mar 29 18:05:51 san2 shpchp
Mar 29 18:05:51 san2 i2c_i801
Mar 29 18:05:51 san2 lpc_ich
Mar 29 18:05:51 san2 video
Mar 29 18:05:51 san2 mfd_core
Mar 29 18:05:51 san2 pcspkr
Mar 29 18:05:51 san2 nfsd
Mar 29 18:05:51 san2 auth_rpcgss
Mar 29 18:05:51 san2 nfs_acl
Mar 29 18:05:51 san2 lockd
Mar 29 18:05:51 san2 grace
Mar 29 18:05:51 san2 sunrpc
Mar 29 18:05:51 san2 ip_tables
Mar 29 18:05:51 san2 ext4
Mar 29 18:05:51 san2 mbcache
Mar 29 18:05:51 san2 jbd2
Mar 29 18:05:51 san2 mgag200
Mar 29 18:05:51 san2 syscopyarea
Mar 29 18:05:51 san2 sysfillrect
Mar 29 18:05:51 san2 sysimgblt
Mar 29 18:05:51 san2 i2c_algo_bit
Mar 29 18:05:51 san2 drm_kms_helper
Mar 29 18:05:51 san2 ttm
Mar 29 18:05:51 san2 ahci
Mar 29 18:05:51 san2 crc32c_intel
Mar 29 18:05:51 san2 libahci
Mar 29 18:05:51 san2 drm
Mar 29 18:05:51 san2 libata
Mar 29 18:05:51 san2 serio_raw
Mar 29 18:05:51 san2 ixgbe
Mar 29 18:05:51 san2 i2c_core
Mar 29 18:05:51 san2 e1000e
Mar 29 18:05:51 san2 mdio
Mar 29 18:05:51 san2 dca
Mar 29 18:05:51 san2 ptp
Mar 29 18:05:51 san2 arcmsr
Mar 29 18:05:51 san2 pps_core
Mar 29 18:05:51 san2 dm_mirror
Mar 29 18:05:51 san2 dm_region_hash
Mar 29 18:05:51 san2 dm_log
Mar 29 18:05:51 san2 dm_mod
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.797990] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G O 4.1.20-3.el7.x86_64 #1
Mar 29 18:05:51 san2 [ 2480.799528] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:51 san2 [ 2480.801087] task: ffff8807c01d6e00 ti: ffff8807b7a88000 task.ti: ffff8807b7a88000
Mar 29 18:05:51 san2 [ 2480.802611] RIP: 0010:[<ffffffff8106d197>]
Mar 29 18:05:51 san2 [<ffffffff8106d197>] change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:51 san2 [ 2480.804179] RSP: 0018:ffff8807b7a8aba8 EFLAGS: 00010046
Mar 29 18:05:51 san2 [ 2480.805731] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000004
Mar 29 18:05:51 san2 [ 2480.807279] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000080000000
Mar 29 18:05:51 san2 [ 2480.808841] RBP: ffff8807b7a8ac58 R08: 80000000c9173101 R09: 00000000000c9173
Mar 29 18:05:51 san2 [ 2480.810373] R10: ffffea001edff0c0 R11: ffffffff813492f9 R12: 0000000000000010
Mar 29 18:05:51 san2 [ 2480.811884] R13: 0000000000000000 R14: 0000000000000200 R15: 0000000000000005
Mar 29 18:05:51 san2 [ 2480.813443] FS: 0000000000000000(0000) GS:ffff88082fdc0000(0000) knlGS:0000000000000000
Mar 29 18:05:51 san2 [ 2480.814951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 29 18:05:51 san2 [ 2480.816465] CR2: 0000000000000003 CR3: 0000000001a22000 CR4: 00000000001426e0
Mar 29 18:05:51 san2 [ 2480.817981] Stack:
Mar 29 18:05:51 san2 [ 2480.819549] 0000000400000000
Mar 29 18:05:51 san2 0000000000000000
Mar 29 18:05:51 san2 0000000000000000
Mar 29 18:05:51 san2 ffff8806b321d000
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.821098] 00000000c9173000
Mar 29 18:05:51 san2 0000160000000000
Mar 29 18:05:51 san2 0000000000000000
Mar 29 18:05:51 san2 0000000000000000
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.822681] 0000000000000010
Mar 29 18:05:51 san2 0000000000000000
Mar 29 18:05:51 san2 0000000000000001
Mar 29 18:05:51 san2 0000000000000005
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.824197] Call Trace:
Mar 29 18:05:51 san2 [ 2480.825721] [<ffffffff8106d4e8>] _set_pages_array+0xe8/0x140
Mar 29 18:05:51 san2 [ 2480.827320] [<ffffffff8106d573>] set_pages_array_wc+0x13/0x20
Mar 29 18:05:51 san2 [ 2480.828921] [<ffffffffa02d42ef>] ttm_set_pages_caching+0x2f/0x70 [ttm]
Mar 29 18:05:51 san2 [ 2480.830511] [<ffffffffa02d4434>] ttm_alloc_new_pages.isra.6+0xb4/0x180 [ttm]
Mar 29 18:05:51 san2 [ 2480.832070] [<ffffffffa02d0e31>] ? ttm_mem_reg_ioremap+0xd1/0x120 [ttm]
Mar 29 18:05:51 san2 [ 2480.833684] [<ffffffffa02d4dd3>] ttm_pool_populate+0x3f3/0x510 [ttm]
Mar 29 18:05:51 san2 [ 2480.835242] [<ffffffffa02fedde>] mgag200_ttm_tt_populate+0xe/0x10 [mgag200]
Mar 29 18:05:51 san2 [ 2480.836821] [<ffffffffa02d184d>] ttm_bo_move_memcpy+0x61d/0x6a0 [ttm]
Mar 29 18:05:51 san2 [ 2480.838346] [<ffffffffa02fed88>] mgag200_bo_move+0x18/0x20 [mgag200]
Mar 29 18:05:51 san2 [ 2480.839893] [<ffffffffa02ced95>] ttm_bo_handle_move_mem+0x265/0x5c0 [ttm]
Mar 29 18:05:51 san2 [ 2480.841428] [<ffffffffa02cf6e7>] ? ttm_bo_mem_space+0xe7/0x350 [ttm]
Mar 29 18:05:51 san2 [ 2480.843005] [<ffffffffa02cfded>] ttm_bo_validate+0x20d/0x230 [ttm]
Mar 29 18:05:51 san2 [ 2480.844549] [<ffffffff8106ab14>] ? iounmap+0x84/0xb0
Mar 29 18:05:51 san2 [ 2480.846111] [<ffffffffa02ff653>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
Mar 29 18:05:51 san2 [ 2480.847626] [<ffffffffa02faae5>] mga_crtc_do_set_base.isra.8.constprop.20+0x85/0x450 [mgag200]
Mar 29 18:05:51 san2 [ 2480.849171] [<ffffffff81356c36>] ? delay_tsc+0x46/0x70
Mar 29 18:05:51 san2 [ 2480.850658] [<ffffffffa02fbf0b>] mga_crtc_mode_set+0x105b/0x21a0 [mgag200]
Mar 29 18:05:51 san2 [ 2480.852200] [<ffffffffa02364d3>] ? drm_mode_object_get+0x13/0x20 [drm]
Mar 29 18:05:51 san2 [ 2480.853730] [<ffffffffa03179ad>] drm_crtc_helper_set_mode+0x33d/0x5a0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.855303] [<ffffffffa0318a42>] drm_crtc_helper_set_config+0x892/0xab0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.856859] [<ffffffffa02350ff>] drm_mode_set_config_internal+0x6f/0x110 [drm]
Mar 29 18:05:51 san2 [ 2480.858392] [<ffffffffa0324530>] drm_fb_helper_pan_display+0xa0/0xf0 [drm_kms_helper]
Mar 29 18:05:51 san2 [ 2480.859894] [<ffffffff813b8381>] fb_pan_display+0xd1/0x1a0
Mar 29 18:05:51 san2 [ 2480.861653] [<ffffffff813b2290>] bit_update_start+0x20/0x50
Mar 29 18:05:51 san2 [ 2480.863651] [<ffffffff813b0a00>] fbcon_switch+0x3a0/0x5a0
Mar 29 18:05:51 san2 [ 2480.865706] [<ffffffff81434bf9>] redraw_screen+0x1a9/0x250
Mar 29 18:05:51 san2 [ 2480.867722] [<ffffffff813af15a>] fbcon_blank+0x22a/0x2f0
Mar 29 18:05:51 san2 [ 2480.869714] [<ffffffff81194981>] ? irq_work_queue+0x11/0x90
Mar 29 18:05:51 san2 [ 2480.871556] [<ffffffff810f9df2>] ? wake_up_klogd+0x32/0x40
Mar 29 18:05:51 san2 [ 2480.873270] [<ffffffff810fa008>] ? console_unlock+0x208/0x480
Mar 29 18:05:51 san2 [ 2480.874940] [<ffffffff8110b511>] ? internal_add_timer+0x91/0xb0
Mar 29 18:05:51 san2 [ 2480.876561] [<ffffffff8110db3c>] ? mod_timer+0x10c/0x230
Mar 29 18:05:51 san2 [ 2480.878152] [<ffffffff81435778>] do_unblank_screen+0xb8/0x1f0
Mar 29 18:05:51 san2 [ 2480.879756] [<ffffffff814358c0>] unblank_screen+0x10/0x20
Mar 29 18:05:51 san2 [ 2480.881312] [<ffffffff81359499>] bust_spinlocks+0x19/0x40
Mar 29 18:05:51 san2 [ 2480.882751] [<ffffffff8101956c>] oops_end+0x3c/0x120
Mar 29 18:05:51 san2 [ 2480.884205] [<ffffffff816db936>] no_context+0x2ee/0x366
Mar 29 18:05:51 san2 [ 2480.885651] [<ffffffff816dba21>] __bad_area_nosemaphore+0x73/0x1cc
Mar 29 18:05:51 san2 [ 2480.887088] [<ffffffff81014693>] ? __switch_to+0x1e3/0x580
Mar 29 18:05:51 san2 [ 2480.888526] [<ffffffff816dbb8d>] bad_area_nosemaphore+0x13/0x15
Mar 29 18:05:51 san2 [ 2480.889963] [<ffffffff81069fe6>] __do_page_fault+0x86/0x420
Mar 29 18:05:51 san2 [ 2480.891401] [<ffffffff8110b7eb>] ? lock_timer_base.isra.35+0x2b/0x50
Mar 29 18:05:51 san2 [ 2480.892750] [<ffffffff8106a3b0>] do_page_fault+0x30/0x80
Mar 29 18:05:51 san2 [ 2480.894078] [<ffffffff816eb0d8>] page_fault+0x28/0x30
Mar 29 18:05:51 san2 [ 2480.895374] [<ffffffff81357a96>] ? memcpy_erms+0x6/0x10
Mar 29 18:05:51 san2 [ 2480.896683] [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:51 san2 [ 2480.898003] [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:51 san2 [ 2480.899308] [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:51 san2 [ 2480.900597] [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:51 san2 [ 2480.901886] [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:51 san2 [ 2480.903120] [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:51 san2 [ 2480.904360] [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:51 san2 [ 2480.905578] [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:51 san2 [ 2480.906804] [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.908035] [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:51 san2 [ 2480.909265] [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:51 san2 [ 2480.910494] [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:51 san2 [ 2480.911716] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.912853] [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:51 san2 [ 2480.914034] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:51 san2 [ 2480.915218] [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:51 san2 [ 2480.916394] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.917564] [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:51 san2 [ 2480.918706] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:51 san2 [ 2480.919850] Code:
Mar 29 18:05:51 san2
Mar 29 18:05:51 san2 [ 2480.922520] RIP
Mar 29 18:05:51 san2 [<ffffffff8106d197>] change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:51 san2 [ 2480.923697] RSP <ffff8807b7a8aba8>
Mar 29 18:05:51 san2 [ 2480.924852] ---[ end trace 6b7ee2c36b3abf19 ]---
Mar 29 18:05:52 san2 [ 2481.050790] Kernel panic - not syncing: Fatal exception
Mar 29 18:05:52 san2 [ 2481.052371] Kernel Offset: disabled
Mar 29 18:05:52 san2 [ 2481.053400] drm_kms_helper: panic occurred, switching back to text console
Mar 29 18:05:52 san2 [ 2481.176995] ---[ end Kernel panic - not syncing: Fatal exception
Mar 29 18:05:52 san2 [ 2481.178078] ------------[ cut here ]------------
Mar 29 18:05:52 san2 [ 2481.179155] WARNING: CPU: 7 PID: 23962 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x5d/0x60()
Mar 29 18:05:52 san2 [ 2481.180414] Modules linked in:
Mar 29 18:05:52 san2 dm_snapshot
Mar 29 18:05:52 san2 xt_comment
Mar 29 18:05:52 san2 binfmt_misc
Mar 29 18:05:52 san2 xt_CHECKSUM
Mar 29 18:05:52 san2 iptable_mangle
Mar 29 18:05:52 san2 ipt_MASQUERADE
Mar 29 18:05:52 san2 nf_nat_masquerade_ipv4
Mar 29 18:05:52 san2 iptable_nat
Mar 29 18:05:52 san2 nf_nat_ipv4
Mar 29 18:05:52 san2 nf_nat
Mar 29 18:05:52 san2 nf_conntrack_ipv4
Mar 29 18:05:52 san2 nf_defrag_ipv4
Mar 29 18:05:52 san2 xt_conntrack
Mar 29 18:05:52 san2 nf_conntrack
Mar 29 18:05:52 san2 ipt_REJECT
Mar 29 18:05:52 san2 nf_reject_ipv4
Mar 29 18:05:52 san2 ebtable_filter
Mar 29 18:05:52 san2 ebtables
Mar 29 18:05:52 san2 ip6table_filter
Mar 29 18:05:52 san2 ip6_tables
Mar 29 18:05:52 san2 iptable_filter
Mar 29 18:05:52 san2 drbd(O)
Mar 29 18:05:52 san2 xfs
Mar 29 18:05:52 san2 dm_thin_pool
Mar 29 18:05:52 san2 dm_persistent_data
Mar 29 18:05:52 san2 dm_bio_prison
Mar 29 18:05:52 san2 dm_bufio
Mar 29 18:05:52 san2 libcrc32c
Mar 29 18:05:52 san2 bcache
Mar 29 18:05:52 san2 netconsole
Mar 29 18:05:52 san2 zram
Mar 29 18:05:52 san2 lz4_compress
Mar 29 18:05:52 san2 bridge
Mar 29 18:05:52 san2 8021q
Mar 29 18:05:52 san2 garp
Mar 29 18:05:52 san2 mrp
Mar 29 18:05:52 san2 stp
Mar 29 18:05:52 san2 llc
Mar 29 18:05:52 san2 x86_pkg_temp_thermal
Mar 29 18:05:52 san2 intel_powerclamp
Mar 29 18:05:52 san2 coretemp
Mar 29 18:05:52 san2 kvm_intel
Mar 29 18:05:52 san2 kvm
Mar 29 18:05:52 san2 crct10dif_pclmul
Mar 29 18:05:52 san2 iTCO_wdt
Mar 29 18:05:52 san2 crc32_pclmul
Mar 29 18:05:52 san2 iTCO_vendor_support
Mar 29 18:05:52 san2 sg
Mar 29 18:05:52 san2 ipmi_si
Mar 29 18:05:52 san2 ipmi_msghandler
Mar 29 18:05:52 san2 shpchp
Mar 29 18:05:52 san2 i2c_i801
Mar 29 18:05:52 san2 lpc_ich
Mar 29 18:05:52 san2 video
Mar 29 18:05:52 san2 mfd_core
Mar 29 18:05:52 san2 pcspkr
Mar 29 18:05:52 san2 nfsd
Mar 29 18:05:52 san2 auth_rpcgss
Mar 29 18:05:52 san2 nfs_acl
Mar 29 18:05:52 san2 lockd
Mar 29 18:05:52 san2 grace
Mar 29 18:05:52 san2 sunrpc
Mar 29 18:05:52 san2 ip_tables
Mar 29 18:05:52 san2 ext4
Mar 29 18:05:52 san2 mbcache
Mar 29 18:05:52 san2 jbd2
Mar 29 18:05:52 san2 mgag200
Mar 29 18:05:52 san2 syscopyarea
Mar 29 18:05:52 san2 sysfillrect
Mar 29 18:05:52 san2 sysimgblt
Mar 29 18:05:52 san2 i2c_algo_bit
Mar 29 18:05:52 san2 drm_kms_helper
Mar 29 18:05:52 san2 ttm
Mar 29 18:05:52 san2 ahci
Mar 29 18:05:52 san2 crc32c_intel
Mar 29 18:05:52 san2 libahci
Mar 29 18:05:52 san2 drm
Mar 29 18:05:52 san2 libata
Mar 29 18:05:52 san2 serio_raw
Mar 29 18:05:52 san2 ixgbe
Mar 29 18:05:52 san2 i2c_core
Mar 29 18:05:52 san2 e1000e
Mar 29 18:05:52 san2 mdio
Mar 29 18:05:52 san2 dca
Mar 29 18:05:52 san2 ptp
Mar 29 18:05:52 san2 arcmsr
Mar 29 18:05:52 san2 pps_core
Mar 29 18:05:52 san2 dm_mirror
Mar 29 18:05:52 san2 dm_region_hash
Mar 29 18:05:52 san2 dm_log
Mar 29 18:05:52 san2 dm_mod
Mar 29 18:05:52 san2
Mar 29 18:05:52 san2 [ 2481.190970] CPU: 7 PID: 23962 Comm: drbd_w_www3.ewh Tainted: G D O 4.1.20-3.el7.x86_64 #1
Mar 29 18:05:52 san2 [ 2481.192411] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.2 02/20/2015
Mar 29 18:05:52 san2 [ 2481.193863] 0000000000000086
Mar 29 18:05:52 san2 0000000088dca908
Mar 29 18:05:52 san2 ffff88082fdc3d58
Mar 29 18:05:52 san2 ffffffff816e16c4
Mar 29 18:05:52 san2
Mar 29 18:05:52 san2 [ 2481.195340] 0000000000000000
Mar 29 18:05:52 san2 ffffffff8191df1e
Mar 29 18:05:52 san2 ffff88082fdc3d98
Mar 29 18:05:52 san2 ffffffff810a0dea
Mar 29 18:05:52 san2
Mar 29 18:05:52 san2 [ 2481.196816] ffff88082fdc3d88
Mar 29 18:05:52 san2 0000000000000000
Mar 29 18:05:52 san2 ffff88082fc177c0
Mar 29 18:05:52 san2 0000000000000007
Mar 29 18:05:52 san2
Mar 29 18:05:52 san2 [ 2481.198295] Call Trace:
Mar 29 18:05:52 san2 [ 2481.199748] <IRQ>
Mar 29 18:05:52 san2 [<ffffffff816e16c4>] dump_stack+0x63/0x81
Mar 29 18:05:52 san2 [ 2481.201229] [<ffffffff810a0dea>] warn_slowpath_common+0x8a/0xc0
Mar 29 18:05:52 san2 [ 2481.202720] [<ffffffff810a0f1a>] warn_slowpath_null+0x1a/0x20
Mar 29 18:05:52 san2 [ 2481.204205] [<ffffffff8104fd9d>] native_smp_send_reschedule+0x5d/0x60
Mar 29 18:05:52 san2 [ 2481.205693] [<ffffffff810e0b15>] trigger_load_balance+0x145/0x1f0
Mar 29 18:05:52 san2 [ 2481.207187] [<ffffffff810cde6c>] scheduler_tick+0x9c/0xe0
Mar 29 18:05:52 san2 [ 2481.208681] [<ffffffff8110df41>] update_process_times+0x51/0x60
Mar 29 18:05:52 san2 [ 2481.210175] [<ffffffff8111e525>] tick_sched_handle.isra.18+0x25/0x60
Mar 29 18:05:52 san2 [ 2481.211670] [<ffffffff8111e5a4>] tick_sched_timer+0x44/0x80
Mar 29 18:05:52 san2 [ 2481.213163] [<ffffffff8110ed27>] __run_hrtimer+0x77/0x220
Mar 29 18:05:52 san2 [ 2481.214650] [<ffffffff8111e560>] ? tick_sched_handle.isra.18+0x60/0x60
Mar 29 18:05:52 san2 [ 2481.216145] [<ffffffff8110f153>] hrtimer_interrupt+0x103/0x230
Mar 29 18:05:52 san2 [ 2481.217633] [<ffffffff81052b69>] local_apic_timer_interrupt+0x39/0x60
Mar 29 18:05:52 san2 [ 2481.219123] [<ffffffff816ebf35>] smp_apic_timer_interrupt+0x45/0x60
Mar 29 18:05:52 san2 [ 2481.220609] [<ffffffff816e9fbe>] apic_timer_interrupt+0x6e/0x80
Mar 29 18:05:52 san2 [ 2481.222092] <EOI>
Mar 29 18:05:52 san2 [<ffffffff816dc166>] ? panic+0x1cd/0x20e
Mar 29 18:05:52 san2 [ 2481.223583] [<ffffffff816dc15f>] ? panic+0x1c6/0x20e
Mar 29 18:05:52 san2 [ 2481.225064] [<ffffffff81019639>] oops_end+0x109/0x120
Mar 29 18:05:52 san2 [ 2481.226536] [<ffffffff81019beb>] die+0x4b/0x70
Mar 29 18:05:52 san2 [ 2481.227994] [<ffffffff81015fac>] do_trap+0x14c/0x160
Mar 29 18:05:52 san2 [ 2481.229447] [<ffffffff8101649c>] do_error_trap+0xac/0x190
Mar 29 18:05:52 san2 [ 2481.230892] [<ffffffff8106d197>] ? change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:52 san2 [ 2481.232345] [<ffffffff81070f28>] ? do_flush_tlb_all+0x48/0x50
Mar 29 18:05:52 san2 [ 2481.233769] [<ffffffff8106c018>] ? lookup_address+0x28/0x30
Mar 29 18:05:52 san2 [ 2481.235160] [<ffffffff8106c0db>] ? _lookup_address_cpa.isra.8+0x3b/0x40
Mar 29 18:05:52 san2 [ 2481.236540] [<ffffffff8106c8fa>] ? __change_page_attr_set_clr+0x81a/0xba0
Mar 29 18:05:52 san2 [ 2481.237911] [<ffffffff81016c50>] do_invalid_op+0x20/0x30
Mar 29 18:05:52 san2 [ 2481.239277] [<ffffffff816eaa7e>] invalid_op+0x1e/0x30
Mar 29 18:05:52 san2 [ 2481.240635] [<ffffffff813492f9>] ? free_cpumask_var+0x9/0x10
Mar 29 18:05:52 san2 [ 2481.241996] [<ffffffff8106d197>] ? change_page_attr_set_clr+0x517/0x520
Mar 29 18:05:52 san2 [ 2481.243351] [<ffffffff8106d4e8>] _set_pages_array+0xe8/0x140
Mar 29 18:05:52 san2 [ 2481.244697] [<ffffffff8106d573>] set_pages_array_wc+0x13/0x20
Mar 29 18:05:52 san2 [ 2481.246044] [<ffffffffa02d42ef>] ttm_set_pages_caching+0x2f/0x70 [ttm]
Mar 29 18:05:52 san2 [ 2481.247373] [<ffffffffa02d4434>] ttm_alloc_new_pages.isra.6+0xb4/0x180 [ttm]
Mar 29 18:05:52 san2 [ 2481.248671] [<ffffffffa02d0e31>] ? ttm_mem_reg_ioremap+0xd1/0x120 [ttm]
Mar 29 18:05:52 san2 [ 2481.249936] [<ffffffffa02d4dd3>] ttm_pool_populate+0x3f3/0x510 [ttm]
Mar 29 18:05:52 san2 [ 2481.251168] [<ffffffffa02fedde>] mgag200_ttm_tt_populate+0xe/0x10 [mgag200]
Mar 29 18:05:52 san2 [ 2481.252368] [<ffffffffa02d184d>] ttm_bo_move_memcpy+0x61d/0x6a0 [ttm]
Mar 29 18:05:52 san2 [ 2481.253534] [<ffffffffa02fed88>] mgag200_bo_move+0x18/0x20 [mgag200]
Mar 29 18:05:52 san2 [ 2481.254667] [<ffffffffa02ced95>] ttm_bo_handle_move_mem+0x265/0x5c0 [ttm]
Mar 29 18:05:52 san2 [ 2481.255766] [<ffffffffa02cf6e7>] ? ttm_bo_mem_space+0xe7/0x350 [ttm]
Mar 29 18:05:52 san2 [ 2481.256832] [<ffffffffa02cfded>] ttm_bo_validate+0x20d/0x230 [ttm]
Mar 29 18:05:52 san2 [ 2481.257871] [<ffffffff8106ab14>] ? iounmap+0x84/0xb0
Mar 29 18:05:52 san2 [ 2481.258888] [<ffffffffa02ff653>] mgag200_bo_push_sysram+0x93/0xe0 [mgag200]
Mar 29 18:05:52 san2 [ 2481.259900] [<ffffffffa02faae5>] mga_crtc_do_set_base.isra.8.constprop.20+0x85/0x450 [mgag200]
Mar 29 18:05:52 san2 [ 2481.260941] [<ffffffff81356c36>] ? delay_tsc+0x46/0x70
Mar 29 18:05:52 san2 [ 2481.261975] [<ffffffffa02fbf0b>] mga_crtc_mode_set+0x105b/0x21a0 [mgag200]
Mar 29 18:05:52 san2 [ 2481.263022] [<ffffffffa02364d3>] ? drm_mode_object_get+0x13/0x20 [drm]
Mar 29 18:05:52 san2 [ 2481.264055] [<ffffffffa03179ad>] drm_crtc_helper_set_mode+0x33d/0x5a0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.265099] [<ffffffffa0318a42>] drm_crtc_helper_set_config+0x892/0xab0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.266161] [<ffffffffa02350ff>] drm_mode_set_config_internal+0x6f/0x110 [drm]
Mar 29 18:05:52 san2 [ 2481.267217] [<ffffffffa0324530>] drm_fb_helper_pan_display+0xa0/0xf0 [drm_kms_helper]
Mar 29 18:05:52 san2 [ 2481.268276] [<ffffffff813b8381>] fb_pan_display+0xd1/0x1a0
Mar 29 18:05:52 san2 [ 2481.269333] [<ffffffff813b2290>] bit_update_start+0x20/0x50
Mar 29 18:05:52 san2 [ 2481.270386] [<ffffffff813b0a00>] fbcon_switch+0x3a0/0x5a0
Mar 29 18:05:52 san2 [ 2481.271434] [<ffffffff81434bf9>] redraw_screen+0x1a9/0x250
Mar 29 18:05:52 san2 [ 2481.272478] [<ffffffff813af15a>] fbcon_blank+0x22a/0x2f0
Mar 29 18:05:52 san2 [ 2481.273526] [<ffffffff81194981>] ? irq_work_queue+0x11/0x90
Mar 29 18:05:52 san2 [ 2481.274571] [<ffffffff810f9df2>] ? wake_up_klogd+0x32/0x40
Mar 29 18:05:52 san2 [ 2481.275612] [<ffffffff810fa008>] ? console_unlock+0x208/0x480
Mar 29 18:05:52 san2 [ 2481.276649] [<ffffffff8110b511>] ? internal_add_timer+0x91/0xb0
Mar 29 18:05:52 san2 [ 2481.277684] [<ffffffff8110db3c>] ? mod_timer+0x10c/0x230
Mar 29 18:05:52 san2 [ 2481.278714] [<ffffffff81435778>] do_unblank_screen+0xb8/0x1f0
Mar 29 18:05:52 san2 [ 2481.279736] [<ffffffff814358c0>] unblank_screen+0x10/0x20
Mar 29 18:05:52 san2 [ 2481.280749] [<ffffffff81359499>] bust_spinlocks+0x19/0x40
Mar 29 18:05:52 san2 [ 2481.281764] [<ffffffff8101956c>] oops_end+0x3c/0x120
Mar 29 18:05:52 san2 [ 2481.282781] [<ffffffff816db936>] no_context+0x2ee/0x366
Mar 29 18:05:52 san2 [ 2481.283799] [<ffffffff816dba21>] __bad_area_nosemaphore+0x73/0x1cc
Mar 29 18:05:52 san2 [ 2481.284828] [<ffffffff81014693>] ? __switch_to+0x1e3/0x580
Mar 29 18:05:52 san2 [ 2481.285851] [<ffffffff816dbb8d>] bad_area_nosemaphore+0x13/0x15
Mar 29 18:05:52 san2 [ 2481.286874] [<ffffffff81069fe6>] __do_page_fault+0x86/0x420
Mar 29 18:05:52 san2 [ 2481.287897] [<ffffffff8110b7eb>] ? lock_timer_base.isra.35+0x2b/0x50
Mar 29 18:05:52 san2 [ 2481.288925] [<ffffffff8106a3b0>] do_page_fault+0x30/0x80
Mar 29 18:05:52 san2 [ 2481.289948] [<ffffffff816eb0d8>] page_fault+0x28/0x30
Mar 29 18:05:52 san2 [ 2481.290966] [<ffffffff81357a96>] ? memcpy_erms+0x6/0x10
Mar 29 18:05:52 san2 [ 2481.291982] [<ffffffff8135c40f>] ? copy_from_iter+0x2bf/0x2e0
Mar 29 18:05:52 san2 [ 2481.293002] [<ffffffff8161b76a>] tcp_sendmsg+0xa2a/0xb50
Mar 29 18:05:52 san2 [ 2481.294012] [<ffffffff81646c54>] inet_sendmsg+0x64/0xa0
Mar 29 18:05:52 san2 [ 2481.295017] [<ffffffff812d1403>] ? selinux_socket_sendmsg+0x23/0x30
Mar 29 18:05:52 san2 [ 2481.296031] [<ffffffff815ac54d>] sock_sendmsg+0x3d/0x50
Mar 29 18:05:52 san2 [ 2481.297036] [<ffffffff815ac67b>] kernel_sendmsg+0x2b/0x30
Mar 29 18:05:52 san2 [ 2481.298042] [<ffffffffa06b0f26>] drbd_send+0xe6/0x200 [drbd]
Mar 29 18:05:52 san2 [ 2481.299047] [<ffffffffa06b2b81>] _drbd_no_send_page.isra.40+0x71/0xb0 [drbd]
Mar 29 18:05:52 san2 [ 2481.300059] [<ffffffffa06b3178>] drbd_send_dblock+0x3e8/0x7a0 [drbd]
Mar 29 18:05:52 san2 [ 2481.301057] [<ffffffffa06a5874>] ? complete_master_bio+0x94/0x170 [drbd]
Mar 29 18:05:52 san2 [ 2481.302066] [<ffffffffa06935cf>] w_send_dblock+0xaf/0x1e0 [drbd]
Mar 29 18:05:52 san2 [ 2481.303071] [<ffffffffa06949a9>] drbd_worker+0xf9/0x3a0 [drbd]
Mar 29 18:05:52 san2 [ 2481.304072] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:52 san2 [ 2481.305082] [<ffffffffa06aed1d>] drbd_thread_setup+0x1d/0x110 [drbd]
Mar 29 18:05:52 san2 [ 2481.306090] [<ffffffffa06aed00>] ? drbd_destroy_connection+0x190/0x190 [drbd]
Mar 29 18:05:52 san2 [ 2481.307101] [<ffffffff810c0b08>] kthread+0xd8/0xf0
Mar 29 18:05:52 san2 [ 2481.308097] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:52 san2 [ 2481.309096] [<ffffffff816e94e2>] ret_from_fork+0x42/0x70
Mar 29 18:05:52 san2 [ 2481.310066] [<ffffffff810c0a30>] ? kthread_create_on_node+0x1b0/0x1b0
Mar 29 18:05:52 san2 [ 2481.311043] ---[ end trace 6b7ee2c36b3abf1a ]---
-------------- next part --------------
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: PingAck did not arrive in time.
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: error receiving RSDataReply, e: -5 l: 65536!
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: ack_receiver terminated
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Terminating drbd_a_www3.ewh
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Connection closed
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: receiver terminated
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: Restarting receiver thread
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: receiver (re)started
Mar 29 18:06:04 importer-peer1 kernel: drbd www3.ewh: conn( Unconnected -> WFConnection )
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: error receiving RSDataReply, e: -5 l: 32768!
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Terminating drbd_a_int-pbx.
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd int-pbx.: conn( Unconnected -> WFConnection )
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: error receiving RSDataReply, e: -5 l: 24576!
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Terminating drbd_a_rsinigsu
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: PingAck did not arrive in time.
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: error receiving RSDataReply, e: -5 l: 61440!
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd rsinigsu: conn( Unconnected -> WFConnection )
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: ack_receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Terminating drbd_a_gls-moni
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Connection closed
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: receiver terminated
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: Restarting receiver thread
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: receiver (re)started
Mar 29 18:06:08 importer-peer1 kernel: drbd gls-moni: conn( Unconnected -> WFConnection )
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: PingAck did not arrive in time.
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: error receiving RSDataReply, e: -5 l: 8192!
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: ack_receiver terminated
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Terminating drbd_a_spuprot.
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Connection closed
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: receiver terminated
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: Restarting receiver thread
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: receiver (re)started
Mar 29 18:06:10 importer-peer1 kernel: drbd spuprot.: conn( Unconnected -> WFConnection )
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: PingAck did not arrive in time.
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: peer( Primary -> Unknown ) conn( SyncTarget -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: error receiving RSDataReply, e: -5 l: 4096!
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: ack_receiver terminated
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Terminating drbd_a_mail.ewh
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Connection closed
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: conn( NetworkFailure -> Unconnected )
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: receiver terminated
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: Restarting receiver thread
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: receiver (re)started
Mar 29 18:06:13 importer-peer1 kernel: drbd mail.ewh: conn( Unconnected -> WFConnection )
More information about the drbd-dev
mailing list