Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
I have been using drbd for a number of years on a number of servers but
have a new issue that is causing me problems.
The problem:
I had to upgrade a cluster quickly on the fly and upgraded the
secondary. When it resynced the drbd devices, I got a high load and both
servers hung.
I then had to enable one device at a time to sync but still had 1
(800GB) device that caused issues so disabled this device (not critical
data).
We tested falling over to the upgraded server as the primary which did
work but decided not to keep this as the primary as we did not have time
to test that all our services were working as expected. The cluster was
stable for 3 days before I went on holiday and I am now back to hear
that the primary, running the older versions of the software still
intermittently experiences a load that grows to >300 over a number of hours.
The only way to resolve is to reboot the secondary (which does not have
a high load) and there is a containment script in to monitor load and
reboot.
I have upgraded this particular cluster before using the same method
without any issues
The versions:
current primary:
CentOS 5.5
drbd 8.3.8, 88/86-94
heartbeat 2
new secondary:
Cent)S 6.5
drbd 8.3.16, 88/86-97
heartbeat 3
We are still using haresources which appears to be working but do know
that when we go to upgrade the other node, we should be looking at
heartbeat v2 (or v1).
The only thing we can see in the logs that are relevant is:
from current primary, running latest vn
when we first try to sync:
kernel: block drbd1: skipping unknown optional packet type 39, l: 0!
kernel: block drbd0: Handshake successful: Agreed network protocol
version 94
So it looks like drbd should be working ok but when we have this issue,
even when the load is just starting to grow we can't stop drbd on the
secondary as it won't release the drbd resource. The secondary with the
newer versions shows the following in /var/log/messages with no similar
kernel traces in the original node.
any ideas before we go to upgrade the other node bringing it to the same
level as drbd 8.3.16?
Tam McLaughlin
----------------------------------------
Jul 15 18:15:45 hostname kernel: block drbd2: bitmap WRITE of 2171 pages
took 1337 jiffies
Jul 15 18:15:45 hostname kernel: block drbd2: 0 KB (0 bits) marked
out-of-sync by on disk bit-map.
Jul 19 22:03:13 hostname kernel: INFO: task drbd0_receiver:3222 blocked
for more than 120 seconds.
Jul 19 22:03:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:03:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:03:13 hostname kernel: drbd0_receive D 0000000000000000 0
3222 2 0x00000080
Jul 19 22:03:13 hostname kernel: ffff8800f1f67920 0000000000000046
0000000000000000 ffff8800f3412080
Jul 19 22:03:13 hostname kernel: 0000000000000001 ffff88007c8129c0
ffff8800f3412080 ffff880037c45ca0
Jul 19 22:03:13 hostname kernel: ffff8800f3412638 ffff8800f1f67fd8
000000000000fbc8 ffff8800f3412638
Jul 19 22:03:13 hostname kernel: Call Trace:
Jul 19 22:03:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:03:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:03:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:03:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:03:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:03:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa0335066>] ?
drbd_send_b_ack+0x46/0x50 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa0318f76>] ?
drbd_may_finish_epoch+0x106/0x430 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:03:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:03:13 hostname kernel: INFO: task drbd1_receiver:3231 blocked
for more than 120 seconds.
Jul 19 22:03:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:03:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:03:13 hostname kernel: drbd1_receive D 0000000000000000 0
3231 2 0x00000080
Jul 19 22:03:13 hostname kernel: ffff8800f304b920 0000000000000046
ffff8800f1ddbbb8 ffff8800f1ddb540
Jul 19 22:03:13 hostname kernel: 0000000000000001 ffff88007c812840
ffff8800f1ddb540 ffff880037c45ca0
Jul 19 22:03:13 hostname kernel: ffff8800f1ddbaf8 ffff8800f304bfd8
000000000000fbc8 ffff8800f1ddbaf8
Jul 19 22:03:13 hostname kernel: Call Trace:
Jul 19 22:03:13 hostname kernel: [<ffffffff810a6d21>] ?
ktime_get_ts+0xb1/0xf0
Jul 19 22:03:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:03:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:03:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:03:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:03:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:03:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:03:13 hostname kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Jul 19 22:03:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:03:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:03:13 hostname kernel: INFO: task drbd2_receiver:3237 blocked
for more than 120 seconds.
Jul 19 22:03:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:03:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:03:13 hostname kernel: drbd2_receive D 0000000000000000 0
3237 2 0x00000080
Jul 19 22:03:13 hostname kernel: ffff88007d5f1920 0000000000000046
0000000000000000 ffff88007d5ecaa0
Jul 19 22:03:13 hostname kernel: 0000000000000001 ffff88007c812900
ffff88007d5ecaa0 ffff880037c45ca0
Jul 19 22:03:13 hostname kernel: ffff88007d5ed058 ffff88007d5f1fd8
000000000000fbc8 ffff88007d5ed058
Jul 19 22:03:13 hostname kernel: Call Trace:
Jul 19 22:03:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:03:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:03:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:03:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:03:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:03:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:03:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:03:13 hostname kernel: [<ffffffff814a24f2>] ?
do_tcp_setsockopt+0x102/0x490
Jul 19 22:03:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:03:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:03:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:03:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:03:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:05:13 hostname kernel: INFO: task drbd0_receiver:3222 blocked
for more than 120 seconds.
Jul 19 22:05:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:05:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:05:13 hostname kernel: drbd0_receive D 0000000000000000 0
3222 2 0x00000080
Jul 19 22:05:13 hostname kernel: ffff8800f1f67920 0000000000000046
0000000000000000 ffff8800f3412080
Jul 19 22:05:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:05:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:05:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:05:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:05:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:05:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa0335066>] ?
drbd_send_b_ack+0x46/0x50 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa0318f76>] ?
drbd_may_finish_epoch+0x106/0x430 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:05:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:05:13 hostname kernel: INFO: task drbd1_receiver:3231 blocked
for more than 120 seconds.
Jul 19 22:05:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:05:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:05:13 hostname kernel: drbd1_receive D 0000000000000000 0
3231 2 0x00000080
Jul 19 22:05:13 hostname kernel: ffff8800f304b920 0000000000000046
ffff8800f1ddbbb8 ffff8800f1ddb540
Jul 19 22:05:13 hostname kernel: 0000000000000001 ffff88007c812840
ffff8800f1ddb540 ffff880037c45ca0
Jul 19 22:05:13 hostname kernel: ffff8800f1ddbaf8 ffff8800f304bfd8
000000000000fbc8 ffff8800f1ddbaf8
Jul 19 22:05:13 hostname kernel: Call Trace:
Jul 19 22:05:13 hostname kernel: [<ffffffff810a6d21>] ?
ktime_get_ts+0xb1/0xf0
Jul 19 22:05:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:05:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:05:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:05:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:05:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:05:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:05:13 hostname kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Jul 19 22:05:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:05:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:05:13 hostname kernel: INFO: task drbd2_receiver:3237 blocked
for more than 120 seconds.
Jul 19 22:05:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:05:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:05:13 hostname kernel: drbd2_receive D 0000000000000000 0
3237 2 0x00000080
Jul 19 22:05:13 hostname kernel: ffff88007d5f1920 0000000000000046
0000000000000000 ffff88007d5ecaa0
Jul 19 22:05:13 hostname kernel: 0000000000000001 ffff88007c812900
ffff88007d5ecaa0 ffff880037c45ca0
Jul 19 22:05:13 hostname kernel: ffff88007d5ed058 ffff88007d5f1fd8
000000000000fbc8 ffff88007d5ed058
Jul 19 22:05:13 hostname kernel: Call Trace:
Jul 19 22:05:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:05:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:05:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:05:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:05:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:05:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:05:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:05:13 hostname kernel: [<ffffffff814a24f2>] ?
do_tcp_setsockopt+0x102/0x490
Jul 19 22:05:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:05:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:05:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:05:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:05:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:07:13 hostname kernel: INFO: task drbd0_receiver:3222 blocked
for more than 120 seconds.
Jul 19 22:07:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:07:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:07:13 hostname kernel: drbd0_receive D 0000000000000000 0
3222 2 0x00000080
Jul 19 22:07:13 hostname kernel: ffff8800f1f67920 0000000000000046
0000000000000000 ffff8800f3412080
Jul 19 22:07:13 hostname kernel: 0000000000000001 ffff88007c8129c0
ffff8800f3412080 ffff880037c45ca0
Jul 19 22:07:13 hostname kernel: ffff8800f3412638 ffff8800f1f67fd8
000000000000fbc8 ffff8800f3412638
Jul 19 22:07:13 hostname kernel: Call Trace:
Jul 19 22:07:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:07:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:07:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:07:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:07:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:07:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa0335066>] ?
drbd_send_b_ack+0x46/0x50 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa0318f76>] ?
drbd_may_finish_epoch+0x106/0x430 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:07:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:07:13 hostname kernel: INFO: task drbd1_receiver:3231 blocked
for more than 120 seconds.
Jul 19 22:07:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:07:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:07:13 hostname kernel: drbd1_receive D 0000000000000000 0
3231 2 0x00000080
Jul 19 22:07:13 hostname kernel: ffff8800f304b920 0000000000000046
ffff8800f1ddbbb8 ffff8800f1ddb540
Jul 19 22:07:13 hostname kernel: 0000000000000001 ffff88007c812840
ffff8800f1ddb540 ffff880037c45ca0
Jul 19 22:07:13 hostname kernel: ffff8800f1ddbaf8 ffff8800f304bfd8
000000000000fbc8 ffff8800f1ddbaf8
Jul 19 22:07:13 hostname kernel: Call Trace:
Jul 19 22:07:13 hostname kernel: [<ffffffff810a6d21>] ?
ktime_get_ts+0xb1/0xf0
Jul 19 22:07:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:07:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:07:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:07:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:07:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:07:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:07:13 hostname kernel: [<ffffffff8100b9ce>] ?
common_interrupt+0xe/0x13
Jul 19 22:07:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:07:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:07:13 hostname kernel: INFO: task drbd2_receiver:3237 blocked
for more than 120 seconds.
Jul 19 22:07:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:07:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:07:13 hostname kernel: drbd2_receive D 0000000000000000 0
3237 2 0x00000080
Jul 19 22:07:13 hostname kernel: ffff88007d5f1920 0000000000000046
0000000000000000 ffff88007d5ecaa0
Jul 19 22:07:13 hostname kernel: 0000000000000001 ffff88007c812900
ffff88007d5ecaa0 ffff880037c45ca0
Jul 19 22:07:13 hostname kernel: ffff88007d5ed058 ffff88007d5f1fd8
000000000000fbc8 ffff88007d5ed058
Jul 19 22:07:13 hostname kernel: Call Trace:
Jul 19 22:07:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:07:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:07:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:07:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:07:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:07:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:07:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8152b40b>] ?
_spin_unlock_bh+0x1b/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffff8144bee5>] ?
release_sock+0xe5/0x110
Jul 19 22:07:13 hostname kernel: [<ffffffff814a24f2>] ?
do_tcp_setsockopt+0x102/0x490
Jul 19 22:07:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:07:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:07:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:07:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:07:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 19 22:09:13 hostname kernel: INFO: task drbd0_receiver:3222 blocked
for more than 120 seconds.
Jul 19 22:09:13 hostname kernel: Not tainted
2.6.32-431.20.3.el6.x86_64 #1
Jul 19 22:09:13 hostname kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 22:09:13 hostname kernel: drbd0_receive D 0000000000000000 0
3222 2 0x00000080
Jul 19 22:09:13 hostname kernel: ffff8800f1f67920 0000000000000046
0000000000000000 ffff8800f3412080
Jul 19 22:09:13 hostname kernel: 0000000000000001 ffff88007c8129c0
ffff8800f3412080 ffff880037c45ca0
Jul 19 22:09:13 hostname kernel: ffff8800f3412638 ffff8800f1f67fd8
000000000000fbc8 ffff8800f3412638
Jul 19 22:09:13 hostname kernel: Call Trace:
Jul 19 22:09:13 hostname kernel: [<ffffffff81528fc3>] io_schedule+0x73/0xc0
Jul 19 22:09:13 hostname kernel: [<ffffffff81268418>]
get_request_wait+0x108/0x1d0
Jul 19 22:09:13 hostname kernel: [<ffffffff8109afa0>] ?
autoremove_wake_function+0x0/0x40
Jul 19 22:09:13 hostname kernel: [<ffffffff81261fce>] ?
elv_merge+0x17e/0x1c0
Jul 19 22:09:13 hostname kernel: [<ffffffff81268579>]
blk_queue_bio+0x99/0x620
Jul 19 22:09:13 hostname kernel: [<ffffffff81267600>]
generic_make_request+0x240/0x5a0
Jul 19 22:09:13 hostname kernel: [<ffffffff811c492b>] ?
bio_alloc_bioset+0x5b/0xf0
Jul 19 22:09:13 hostname kernel: [<ffffffffa031cc0b>]
drbd_submit_ee+0x20b/0x4f0 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa031dca1>]
receive_Data+0x231/0xe00 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa0335066>] ?
drbd_send_b_ack+0x46/0x50 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa0318f76>] ?
drbd_may_finish_epoch+0x106/0x430 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa03172c3>] drbdd+0xe3/0x380
[drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:09:13 hostname kernel: [<ffffffff81528235>] ? printk+0x41/0x44
Jul 19 22:09:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa031bd14>]
drbdd_init+0xa4/0x1d0 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffff8152883e>] ?
thread_return+0x4e/0x760
Jul 19 22:09:13 hostname kernel: [<ffffffff81061d12>] ?
default_wake_function+0x12/0x20
Jul 19 22:09:13 hostname kernel: [<ffffffffa032e1fe>]
drbd_thread_setup+0x3e/0x120 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffffa032e1c0>] ?
drbd_thread_setup+0x0/0x120 [drbd]
Jul 19 22:09:13 hostname kernel: [<ffffffff8109abf6>] kthread+0x96/0xa0
Jul 19 22:09:13 hostname kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Jul 19 22:09:13 hostname kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Jul 19 22:09:13 hostname kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Jul 20 09:25:10 hostname kernel: Initializing cgroup subsys cpuset