[DRBD-user] Heads up, kernel 4.4 users (ubuntu xenial), possible crash "bad: scheduling from the idle thread"

Lars Ellenberg lars.ellenberg at linbit.com
Tue May 16 14:07:37 CEST 2017

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


For reference, if you run some kernel 4.0 to 4.8 inclusive,
e.g. a Ubuntu Xenial kernel 4.4 something,
and you manage to hit a very specific race,
you could potentially trigger this BUG:
(with a 

| BUG: scheduling while atomic: swapper/19/0/0x00000100
| 
| Modules linked in: ...
| CPU: 19 PID: 0 Comm: swapper/19 Tainted: P      D    OE   4.4.0-51-generic #72-Ubuntu
| Call Trace:
|   <IRQ>  dump_stack+0x63/0x90
|   __schedule_bug+0x4b/0x60
|   __schedule+0x726/0xa30
|   schedule+0x35/0x80
|   blk_mq_freeze_queue_wait+0x57/0xc0
|   ? wake_atomic_t_function+0x60/0x60
|   blk_mq_freeze_queue+0x1a/0x20
|   blk_freeze_queue+0xe/0x10
| 
| [above could be different]
| 
|   blk_cleanup_queue+0x7f/0x150
|   drbd_destroy_device+0xfe/0x160 [drbd]
|   forget_state_change+0xf1/0x110 [drbd]
|   free_state_changes+0x43/0x50 [drbd]
|   drbd_adm_get_initial_state_done+0x53/0x70 [drbd]
|   netlink_sock_destruct+0x3e/0x160
|   sk_destruct+0x1f/0x120
|   __sk_free+0x43/0xa0
|   sk_free+0x18/0x20
|   deferred_put_nlk_sk+0x1f/0x30
|   rcu_process_callbacks+0x204/0x600
| 
| [below could be different]
| 
|   __do_softirq+0x101/0x290
|   irq_exit+0xa3/0xb0
|   smp_apic_timer_interrupt+0x42/0x50
|   apic_timer_interrupt+0x82/0x90
|   <EOI>  ? cpuidle_enter_state+0x10e/0x2b0
|   cpuidle_enter+0x17/0x20
|   call_cpuidle+0x32/0x60
|   ? cpuidle_select+0x13/0x20
|   cpu_startup_entry+0x290/0x350
|   start_secondary+0x154/0x190
| bad: scheduling from the idle thread!

Tell-tale indicators for this specific issue
are "deferred_put_nlk_sk" and "bad: scheduling from the idle thread".

This is not a bug in DRBD, but a bug in the kernel,
and has been fixed since, so please upgrade your kernel.

Broken by 4.0
21e4902aea80 netlink: Lockless lookup with RCU grace period in socket release

Upstream fixed in 4.9,
fix backported to for example 4.4.38
baaf0c65bc8e netlink: Do not schedule work from sk_destruct
d1ed9c1dba63 netlink: Call cb->done from a worker threads

Fix included in various vendor kernel upgrades already.

Thanks,

-- 
: Lars Ellenberg
: LINBIT | Keeping the Digital World Running
: DRBD -- Heartbeat -- Corosync -- Pacemaker
: R&D, Integration, Ops, Consulting, Support

DRBD® and LINBIT® are registered trademarks of LINBIT



More information about the drbd-user mailing list