Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi,
We have a three node setup with DRBD 9, CentOS 7 (kernel-4.9.63-29) and
Xen and we still experience problems with out of sync data after doing a
verify. I did some investigation on the last run on verify and luckily
enough this time the affected data was on a log file, so I had a
timestamp on when the problem occurred. Two entries went missing on one
of the nodes, one from 22:33:55, one at 22:34:04.
Looking at the drbd log file, the time of when the data got missing
corresponded with errors in the drbd.log (attached). I think the problem
is related to:
FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap'
by drbd_r_backup[4718]
Scanning for other 'FIXME' in the log files showed entries for the
volumes that are out of sync as well ('forget_bitmap()', 'receive
bitmap' and 'send_bitmap (WFBitMapS)'), and the error "SyncSource still
sees bits set!!".
What causes this to happen and how do I prevent this from happening again?
Another problem we are experiencing is that a verify is stuck at 99/100 %:
xen2 role:Secondary
replication:VerifyS peer-disk:UpToDate done:100.00
xen3 role:Secondary
replication:VerifyS peer-disk:UpToDate done:100.00
and:
102: cs:VerifyS ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
ns:4858133 nr:0 dw:4860338 dr:179138008 al:568 bm:0 lo:0 pe:[0;0]
ua:0 ap:[0;0] ep:1 wo:2 oos:0
[===================>] verified:100.0% (0/33376)M
finish: 0:00:00 speed: 92 (24 -- 2,020) want: 86,680 K/sec
(stalled)
99% sector pos: 68356856/68359376
resync: used:0/61 hits:8544667 misses:522 starving:0 locked:0
changed:261
act_log: used:0/1237 hits:687304 misses:2221 starving:0
locked:0 changed:698
blocked on activity log: 0
Any idea what is happening here?
Kind regards,
Maarten Bremer
-------------- next part --------------
Dec 1 22:00:34 xen1 kernel: drbd backup xen3: peer( Primary -> Secondary )
Dec 1 22:00:34 xen1 kernel: drbd backup/0 drbd104: No resync, but 1 bits in bitmap!
Dec 1 22:00:34 xen1 kernel: drbd backup/0 drbd104: No resync, but 1 bits in bitmap!
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: PingAck did not arrive in time.
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Dec 1 22:10:48 xen1 kernel: drbd backup/0 drbd104 xen3: pdsk( UpToDate -> DUnknown ) repl( VerifyT -> Off )
Dec 1 22:10:48 xen1 kernel: drbd backup/0 drbd104 xen3: Online Verify reached sector 1952890896
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: ack_receiver terminated
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: Terminating ack_recv thread
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: Connection closed
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: conn( NetworkFailure -> Unconnected )
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: Restarting receiver thread
Dec 1 22:10:48 xen1 kernel: drbd backup xen3: conn( Unconnected -> Connecting )
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Handshake to peer 0 successful: Agreed network protocol version 112
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Peer authenticated using 20 bytes HMAC
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Starting ack_recv thread (from drbd_r_backup [4098])
Dec 1 22:13:18 xen1 kernel: drbd backup xen2: Preparing remote state change 4134868138 (primary_nodes=0, weak_nodes=0)
Dec 1 22:13:18 xen1 kernel: drbd backup xen2: Committing remote state change 4134868138
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Preparing remote state change 160529494 (primary_nodes=0, weak_nodes=0)
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: Committing remote state change 160529494
Dec 1 22:13:18 xen1 kernel: drbd backup xen3: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104 xen3: drbd_sync_handshake:
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104 xen3: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:2 flags:120
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104 xen3: peer 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:2 flags:20
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104 xen3: uuid_compare()=0 by rule 38
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104: No resync, but 2 bits in bitmap!
Dec 1 22:13:18 xen1 kernel: drbd backup/0 drbd104 xen3: pdsk( DUnknown -> UpToDate ) repl( Off -> Established )
Dec 1 22:14:01 xen1 kernel: drbd backup xen3: Preparing remote state change 3166784408 (primary_nodes=0, weak_nodes=0)
Dec 1 22:14:01 xen1 kernel: drbd backup xen3: Committing remote state change 3166784408
Dec 1 22:14:01 xen1 kernel: drbd backup xen3: peer( Secondary -> Primary )
Dec 1 22:14:06 xen1 kernel: drbd backup xen3: peer( Primary -> Secondary )
Dec 1 22:14:06 xen1 kernel: drbd backup/0 drbd104: No resync, but 1 bits in bitmap!
Dec 1 22:14:06 xen1 kernel: drbd backup/0 drbd104: No resync, but 1 bits in bitmap!
Dec 1 22:14:07 xen1 kernel: drbd backup xen3: Preparing remote state change 200069919 (primary_nodes=0, weak_nodes=0)
Dec 1 22:14:07 xen1 kernel: drbd backup xen3: Committing remote state change 200069919
Dec 1 22:14:07 xen1 kernel: drbd backup xen3: peer( Secondary -> Primary )
Dec 1 22:36:32 xen1 kernel: drbd backup: Starting worker thread (from drbdsetup [2315])
Dec 1 22:36:32 xen1 kernel: drbd backup xen3: Starting sender thread (from drbdsetup [2485])
Dec 1 22:36:32 xen1 kernel: drbd backup xen2: Starting sender thread (from drbdsetup [2498])
Dec 1 22:36:33 xen1 kernel: drbd backup/0 drbd104: disk( Diskless -> Attaching )
Dec 1 22:36:33 xen1 kernel: drbd backup/0 drbd104: Maximum number of peer devices = 7
Dec 1 22:36:33 xen1 kernel: drbd backup: Method to ensure write ordering: flush
Dec 1 22:36:33 xen1 kernel: drbd backup/0 drbd104: drbd_bm_resize called with capacity == 1953125000
Dec 1 22:36:33 xen1 kernel: drbd backup/0 drbd104: resync bitmap: bits=244140625 words=26702886 pages=52155
Dec 1 22:36:33 xen1 kernel: drbd backup/0 drbd104: size = 931 GB (976562500 KB)
Dec 1 22:36:34 xen1 kernel: drbd backup/0 drbd104: recounting of set bits took additional 259ms
Dec 1 22:36:34 xen1 kernel: drbd backup/0 drbd104: disk( Attaching -> UpToDate )
Dec 1 22:36:34 xen1 kernel: drbd backup/0 drbd104: attached to current UUID: 024FEC5F33230400
Dec 1 22:36:35 xen1 kernel: drbd backup xen3: conn( StandAlone -> Unconnected )
Dec 1 22:36:35 xen1 kernel: drbd backup xen3: Starting receiver thread (from drbd_w_backup [2317])
Dec 1 22:36:35 xen1 kernel: drbd backup xen3: conn( Unconnected -> Connecting )
Dec 1 22:36:35 xen1 kernel: drbd backup xen2: conn( StandAlone -> Unconnected )
Dec 1 22:36:35 xen1 kernel: drbd backup xen2: Starting receiver thread (from drbd_w_backup [2317])
Dec 1 22:36:35 xen1 kernel: drbd backup xen2: conn( Unconnected -> Connecting )
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Handshake to peer 0 successful: Agreed network protocol version 112
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Peer authenticated using 20 bytes HMAC
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Starting ack_recv thread (from drbd_r_backup [3381])
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Handshake to peer 1 successful: Agreed network protocol version 112
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Peer authenticated using 20 bytes HMAC
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Starting ack_recv thread (from drbd_r_backup [3383])
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Preparing remote state change 256926584 (primary_nodes=0, weak_nodes=0)
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Committing remote state change 256926584
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104: disk( UpToDate -> Outdated )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: drbd_sync_handshake:
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:1 flags:20
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: peer 05BD4D0CA42ED132:024FEC5F33230400:EB28BEFBE590B852:1C7D4D00E62F67BA bits:63 flags:100
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: uuid_compare()=-2 by rule 50
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Preparing remote state change 437337321 (primary_nodes=0, weak_nodes=0)
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: helper command: /sbin/drbdadm before-resync-target
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Committing remote state change 437337321
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: conn( Connecting -> Connected ) peer( Unknown -> Primary )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104: before-resync-target handler returned 1, dropping connection.
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen2: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: ack_receiver terminated
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Terminating ack_recv thread
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: drbd_sync_handshake:
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:2 flags:20
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: peer 05BD4D0CA42ED133:024FEC5F33230400:EB28BEFBE590B852:1C7D4D00E62F67BA bits:65 flags:120
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: uuid_compare()=-2 by rule 50
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 107(1), total 107; compression: 100.0%
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 107(1), total 107; compression: 100.0%
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm before-resync-target
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104: before-resync-target handler returned 1, dropping connection.
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: conn( Connected -> Disconnecting ) peer( Primary -> Unknown )
Dec 1 22:36:36 xen1 kernel: drbd backup/0 drbd104 xen3: pdsk( UpToDate -> DUnknown ) repl( WFBitMapT -> Off )
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: ack_receiver terminated
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Terminating ack_recv thread
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Connection closed
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: conn( Disconnecting -> StandAlone )
Dec 1 22:36:36 xen1 kernel: drbd backup xen2: Terminating receiver thread
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Connection closed
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: conn( Disconnecting -> StandAlone )
Dec 1 22:36:36 xen1 kernel: drbd backup xen3: Terminating receiver thread
Dec 1 22:36:59 xen1 kernel: drbd backup xen3: conn( StandAlone -> Unconnected )
Dec 1 22:36:59 xen1 kernel: drbd backup xen3: Starting receiver thread (from drbd_w_backup [2317])
Dec 1 22:36:59 xen1 kernel: drbd backup xen3: conn( Unconnected -> Connecting )
Dec 1 22:36:59 xen1 kernel: drbd backup xen2: conn( StandAlone -> Unconnected )
Dec 1 22:36:59 xen1 kernel: drbd backup xen2: Starting receiver thread (from drbd_w_backup [2317])
Dec 1 22:36:59 xen1 kernel: drbd backup xen2: conn( Unconnected -> Connecting )
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Handshake to peer 1 successful: Agreed network protocol version 112
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Peer authenticated using 20 bytes HMAC
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Starting ack_recv thread (from drbd_r_backup [4718])
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Handshake to peer 0 successful: Agreed network protocol version 112
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Peer authenticated using 20 bytes HMAC
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Starting ack_recv thread (from drbd_r_backup [4716])
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Preparing remote state change 2781476550 (primary_nodes=0, weak_nodes=0)
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: Committing remote state change 2781476550
Dec 1 22:37:00 xen1 kernel: drbd backup xen3: conn( Connecting -> Connected ) peer( Unknown -> Primary )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: drbd_sync_handshake:
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:65 flags:120
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: peer 05BD4D0CA42ED133:024FEC5F33230400:EB28BEFBE590B852:1C7D4D00E62F67BA bits:69 flags:120
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: uuid_compare()=-2 by rule 50
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 107(1), total 107; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Preparing remote state change 2133914811 (primary_nodes=0, weak_nodes=0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 107(1), total 107; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm before-resync-target
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Committing remote state change 2133914811
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: disk( Outdated -> Inconsistent )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: resync-susp( no -> connection dependency )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: repl( WFBitMapT -> SyncTarget )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: Began resync as SyncTarget (will sync 276 KB [69 bits set]).
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: drbd_sync_handshake:
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:63 flags:104
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: peer 05BD4D0CA42ED132:024FEC5F33230400:EB28BEFBE590B852:1C7D4D00E62F67BA bits:67 flags:100
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: uuid_compare()=-2 by rule 50
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME drbd_a_backup[4761] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME drbd_a_backup[4761] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Preparing remote state change 2133914811 (primary_nodes=0, weak_nodes=0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 107(1), total 107; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm before-resync-target
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: Committing remote state change 2133914811
Dec 1 22:37:00 xen1 kernel: drbd backup xen2: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: disk( Outdated -> Inconsistent )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: resync-susp( no -> connection dependency )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: repl( WFBitMapT -> SyncTarget )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: Began resync as SyncTarget (will sync 276 KB [69 bits set]).
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: drbd_sync_handshake:
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: self 024FEC5F33230400:0000000000000000:EB28BEFBE590B852:1C7D4D00E62F67BA bits:63 flags:104
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: peer 05BD4D0CA42ED132:024FEC5F33230400:EB28BEFBE590B852:1C7D4D00E62F67BA bits:67 flags:100
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: uuid_compare()=-2 by rule 50
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME drbd_a_backup[4761] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME drbd_a_backup[4761] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 98(1), total 98; compression: 100.0%
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: helper command: /sbin/drbdadm before-resync-target
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME drbd_a_backup[4763] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: FIXME kworker/u64:8[415] in __bm_op, bitmap locked for 'receive bitmap' by drbd_r_backup[4718]
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: Resync done (total 1 sec; paused 0 sec; 276 K/sec)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: updated UUIDs 05BD4D0CA42ED132:0000000000000000:024FEC5F33230400:EB28BEFBE590B852
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: disk( Inconsistent -> Outdated )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: resync-susp( connection dependency -> no )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: repl( SyncTarget -> Established )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm after-resync-target
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: helper command: /sbin/drbdadm before-resync-target exit code 0 (0x0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: disk( Outdated -> Inconsistent )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: repl( WFBitMapT -> SyncTarget )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: resync-susp( no -> connection dependency )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: Began resync as SyncTarget (will sync 16 KB [4 bits set]).
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: Retrying drbd_rs_del_all() later. refcnt=1
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: Resync done (total 1 sec; paused 0 sec; 16 K/sec)
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: updated UUIDs 05BD4D0CA42ED132:0000000000000000:05BD4D0CA42ED132:024FEC5F33230400
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104: disk( Inconsistent -> UpToDate )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: repl( SyncTarget -> Established )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen3: resync-susp( connection dependency -> no )
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: helper command: /sbin/drbdadm after-resync-target
Dec 1 22:37:00 xen1 kernel: drbd backup/0 drbd104 xen2: helper command: /sbin/drbdadm after-resync-target exit code 0 (0x0)