Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello,
I have a problem with DRBD.
Strangely, I first had some out of sync resources.
Writing to FS over DRBD was also very very slow, even sometimes hangued.
I did some tests : I stopped DRBD on secondary node.
Writing to FS was then OK.
I re-started DRBD on secondary node.
Synchronisation was abnormally very very slow, around 100KB/s (instead
of my usual 90-100MB/s).
And writing to FS was again let's say impossible.
As soon as "drbdadm verify all" started, kernel said Oops with a NULL
pointer dereference.
DRBD is of course still in a bad state, unusable.
Could you tell me more ?
Could this issue be corrected ?
Following are versions used, logs etc...
Of course feel free to ask me if you need further information !
Thank you very much !
Best regards,
Ben
### Configuration on both nodes :
# OS
Debian 6 "squeeze", stable
# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ ea9e28dbff98e331a62bcbcc63a6135808fe2917\
build\ by\ @murphy\,\ 2010-07-17\ 12:38:16
DRBDADM_API_VERSION=88
DRBD_KERNEL_VERSION_CODE=0x080307
DRBDADM_VERSION_CODE=0x080307
DRBDADM_VERSION=8.3.7
# Kernel version
ftp://ftp.ovh.net/made-in-ovh/bzImage/2.6.34.6-3/
bzImage-2.6.34.6-xxxx-std-ipv6-32
# drbd8-utils
2:8.3.7-2.1
http://packages.debian.org/squeeze/drbd8-utils
### kern.log (from the beginning of the issue to the kernel BUG) :
Aug 27 20:42:30 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 20:42:30 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Aug 27 20:42:30 srv2-2 kernel: block drbd2: asender terminated
Aug 27 20:42:30 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 20:42:30 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 20:42:30 srv2-2 kernel: block drbd2: Connection closed
Aug 27 20:42:30 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 20:42:30 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 20:42:30 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 20:42:30 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 20:42:30 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 20:42:32 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 20:42:32 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 20:42:32 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [2604])
Aug 27 20:42:32 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 20:42:32 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 20:42:32 srv2-2 kernel: block drbd2: self
7AC135B0DDA176B6:0000000000000000:9F1B469E8F2919D0:7C82F73AC1847EED
bits:0 flags:0
Aug 27 20:42:32 srv2-2 kernel: block drbd2: peer
50B3DFC70675E881:7AC135B0DDA176B7:9F1B469E8F2919D1:7C82F73AC1847EED
bits:0 flags:0
Aug 27 20:42:32 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 20:42:32 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 20:45:17 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 20:45:17 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 20:45:17 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 20:45:17 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Aug 27 20:45:17 srv2-2 kernel: block drbd2: Began resync as SyncTarget
(will sync 0 KB [0 bits set]).
Aug 27 20:45:18 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 20:45:18 srv2-2 kernel: block drbd2: conn( SyncTarget ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 20:45:18 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 20:45:18 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:03:02 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 21:03:02 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Aug 27 21:03:02 srv2-2 kernel: block drbd2: asender terminated
Aug 27 21:03:02 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 21:03:02 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 21:03:02 srv2-2 kernel: block drbd2: Connection closed
Aug 27 21:03:02 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 21:03:02 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 21:03:02 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 21:03:02 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 21:03:02 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 21:03:03 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 21:03:03 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 21:03:03 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [2604])
Aug 27 21:03:03 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 21:03:03 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 21:03:03 srv2-2 kernel: block drbd2: self
50B3DFC70675E880:0000000000000000:93E448BA70E67C8E:7AC135B0DDA176B7
bits:0 flags:0
Aug 27 21:03:03 srv2-2 kernel: block drbd2: peer
A4B0DBA65DC23955:50B3DFC70675E881:93E448BA70E67C8E:7AC135B0DDA176B7
bits:0 flags:0
Aug 27 21:03:03 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 21:03:03 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 21:06:00 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 21:06:00 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 21:06:00 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:06:00 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Aug 27 21:06:00 srv2-2 kernel: block drbd2: Began resync as SyncTarget
(will sync 0 KB [0 bits set]).
Aug 27 21:06:00 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 21:06:00 srv2-2 kernel: block drbd2: conn( SyncTarget ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 21:06:00 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 21:06:00 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:10:23 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 21:10:23 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
Aug 27 21:10:23 srv2-2 kernel: block drbd2: asender terminated
Aug 27 21:10:23 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 21:10:23 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 21:10:23 srv2-2 kernel: block drbd2: Connection closed
Aug 27 21:10:23 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 21:10:23 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 21:10:23 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 21:10:23 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 21:10:23 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 21:10:24 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 21:10:24 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 21:10:24 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [2604])
Aug 27 21:10:24 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 21:10:24 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 21:10:24 srv2-2 kernel: block drbd2: self
A4B0DBA65DC23954:0000000000000000:A9E7DB06198932D4:50B3DFC70675E881
bits:0 flags:0
Aug 27 21:10:24 srv2-2 kernel: block drbd2: peer
C0808022EBB34FD3:A4B0DBA65DC23955:A9E7DB06198932D4:50B3DFC70675E881
bits:0 flags:0
Aug 27 21:10:24 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 21:10:24 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 21:13:28 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 21:13:29 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 21:13:29 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:13:29 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Aug 27 21:13:29 srv2-2 kernel: block drbd2: Began resync as SyncTarget
(will sync 0 KB [0 bits set]).
Aug 27 21:13:30 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 21:13:30 srv2-2 kernel: block drbd2: conn( SyncTarget ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 21:13:30 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 21:13:30 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:13:30 srv2-2 kernel: block drbd2: Connected in w_make_resync_request
Aug 27 21:41:58 srv2-2 kernel: block drbd1: peer( Primary -> Unknown )
conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
Aug 27 21:41:58 srv2-2 kernel: block drbd1: short read receiving data:
read 2096 expected 4096
Aug 27 21:41:58 srv2-2 kernel: block drbd1: error receiving Data, l: 4120!
Aug 27 21:41:58 srv2-2 kernel: block drbd1: asender terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd1: Terminating drbd1_asender
Aug 27 21:41:58 srv2-2 kernel: block drbd1: Connection closed
Aug 27 21:41:58 srv2-2 kernel: block drbd1: conn( Disconnecting -> StandAlone )
Aug 27 21:41:58 srv2-2 kernel: block drbd1: receiver terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd1: Terminating drbd1_receiver
Aug 27 21:41:58 srv2-2 kernel: block drbd1: disk( UpToDate -> Diskless )
Aug 27 21:41:58 srv2-2 kernel: block drbd1: drbd_bm_resize called with
capacity == 0
Aug 27 21:41:58 srv2-2 kernel: block drbd1: worker terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd1: Terminating drbd1_worker
Aug 27 21:41:58 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
Aug 27 21:41:58 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 21:41:58 srv2-2 kernel: block drbd2: asender terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 21:41:58 srv2-2 kernel: block drbd2: Connection closed
Aug 27 21:41:58 srv2-2 kernel: block drbd2: conn( Disconnecting -> StandAlone )
Aug 27 21:41:58 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd2: Terminating drbd2_receiver
Aug 27 21:41:58 srv2-2 kernel: block drbd2: disk( UpToDate -> Diskless )
Aug 27 21:41:58 srv2-2 kernel: block drbd2: drbd_bm_resize called with
capacity == 0
Aug 27 21:41:58 srv2-2 kernel: block drbd2: worker terminated
Aug 27 21:41:58 srv2-2 kernel: block drbd2: Terminating drbd2_worker
Aug 27 21:43:50 srv2-2 kernel: block drbd1: Starting worker thread
(from cqueue [2561])
Aug 27 21:43:50 srv2-2 kernel: block drbd1: disk( Diskless -> Attaching )
Aug 27 21:43:50 srv2-2 kernel: block drbd1: No usable activity log found.
Aug 27 21:43:50 srv2-2 kernel: block drbd1: Method to ensure write
ordering: barrier
Aug 27 21:43:50 srv2-2 kernel: block drbd1: max_segment_size ( = BIO
size ) = 32768
Aug 27 21:43:50 srv2-2 kernel: block drbd1: drbd_bm_resize called with
capacity == 1450419656
Aug 27 21:43:50 srv2-2 kernel: block drbd1: resync bitmap:
bits=181302457 words=5665702
Aug 27 21:43:50 srv2-2 kernel: block drbd1: size = 692 GB (725209828 KB)
Aug 27 21:43:51 srv2-2 kernel: block drbd1: recounting of set bits
took additional 6 jiffies
Aug 27 21:43:51 srv2-2 kernel: block drbd1: 0 KB (0 bits) marked
out-of-sync by on disk bit-map.
Aug 27 21:43:51 srv2-2 kernel: block drbd1: disk( Attaching -> UpToDate )
Aug 27 21:43:51 srv2-2 kernel: block drbd2: Starting worker thread
(from cqueue [2561])
Aug 27 21:43:51 srv2-2 kernel: block drbd2: disk( Diskless -> Attaching )
Aug 27 21:43:51 srv2-2 kernel: block drbd2: No usable activity log found.
Aug 27 21:43:51 srv2-2 kernel: block drbd2: Method to ensure write
ordering: barrier
Aug 27 21:43:51 srv2-2 kernel: block drbd2: max_segment_size ( = BIO
size ) = 32768
Aug 27 21:43:51 srv2-2 kernel: block drbd2: drbd_bm_resize called with
capacity == 1465104376
Aug 27 21:43:51 srv2-2 kernel: block drbd2: resync bitmap:
bits=183138047 words=5723064
Aug 27 21:43:51 srv2-2 kernel: block drbd2: size = 699 GB (732552188 KB)
Aug 27 21:43:51 srv2-2 kernel: block drbd2: recounting of set bits
took additional 6 jiffies
Aug 27 21:43:51 srv2-2 kernel: block drbd2: 0 KB (0 bits) marked
out-of-sync by on disk bit-map.
Aug 27 21:43:51 srv2-2 kernel: block drbd2: disk( Attaching -> UpToDate )
Aug 27 21:43:51 srv2-2 kernel: block drbd1: conn( StandAlone -> Unconnected )
Aug 27 21:43:51 srv2-2 kernel: block drbd1: Starting receiver thread
(from drbd1_worker [27314])
Aug 27 21:43:51 srv2-2 kernel: block drbd1: receiver (re)started
Aug 27 21:43:51 srv2-2 kernel: block drbd1: conn( Unconnected -> WFConnection )
Aug 27 21:43:51 srv2-2 kernel: block drbd2: conn( StandAlone -> Unconnected )
Aug 27 21:43:51 srv2-2 kernel: block drbd2: Starting receiver thread
(from drbd2_worker [27319])
Aug 27 21:43:51 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 21:43:51 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 21:43:51 srv2-2 kernel: block drbd1: Handshake successful:
Agreed network protocol version 92
Aug 27 21:43:51 srv2-2 kernel: block drbd1: conn( WFConnection ->
WFReportParams )
Aug 27 21:43:51 srv2-2 kernel: block drbd1: Starting asender thread
(from drbd1_receiver [27334])
Aug 27 21:43:51 srv2-2 kernel: block drbd1: data-integrity-alg: <not-used>
Aug 27 21:43:52 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 21:43:52 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 21:43:52 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [27340])
Aug 27 21:43:52 srv2-2 kernel: block drbd1: drbd_sync_handshake:
Aug 27 21:43:52 srv2-2 kernel: block drbd1: self
D935FD158B48D358:0000000000000000:44A7A13DD9733674:1DF75354805DC515
bits:0 flags:0
Aug 27 21:43:52 srv2-2 kernel: block drbd1: peer
DE0D4E700325B9AB:D935FD158B48D359:44A7A13DD9733675:1DF75354805DC515
bits:31985 flags:0
Aug 27 21:43:52 srv2-2 kernel: block drbd1: uuid_compare()=-1 by rule 50
Aug 27 21:43:52 srv2-2 kernel: block drbd1: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 21:43:52 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 21:43:52 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 21:43:52 srv2-2 kernel: block drbd2: self
C0808022EBB34FD2:0000000000000000:471662D73DA91BF8:A4B0DBA65DC23955
bits:0 flags:0
Aug 27 21:43:52 srv2-2 kernel: block drbd2: peer
F099BEA2390E084D:C0808022EBB34FD3:471662D73DA91BF8:A4B0DBA65DC23955
bits:0 flags:0
Aug 27 21:43:52 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 21:43:52 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 21:44:37 srv2-2 kernel: block drbd1: conn( WFBitMapT -> WFSyncUUID )
Aug 27 21:44:37 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm before-resync-target minor-1
Aug 27 21:44:37 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
Aug 27 21:44:37 srv2-2 kernel: block drbd1: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Aug 27 21:44:37 srv2-2 kernel: block drbd2: aftr_isp( 0 -> 1 )
Aug 27 21:44:37 srv2-2 kernel: block drbd1: Began resync as SyncTarget
(will sync 127940 KB [31985 bits set]).
Aug 27 21:45:48 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 21:45:48 srv2-2 kernel: block drbd2: peer_isp( 0 -> 1 )
Aug 27 21:45:48 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 21:45:48 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:45:48 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 21:45:48 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 21:45:48 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 21:45:48 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 21:45:48 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 21:45:48 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 21:52:21 srv2-2 kernel: block drbd1: peer( Primary -> Unknown )
conn( SyncTarget -> Disconnecting ) pdsk( UpToDate -> DUnknown )
Aug 27 21:52:21 srv2-2 kernel: block drbd1: short read receiving data:
read 3264 expected 4096
Aug 27 21:52:21 srv2-2 kernel: block drbd2: aftr_isp( 1 -> 0 )
Aug 27 21:52:21 srv2-2 kernel: block drbd1: error receiving
RSDataReply, l: 32792!
Aug 27 21:52:21 srv2-2 kernel: block drbd1: asender terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd1: Terminating drbd1_asender
Aug 27 21:52:21 srv2-2 kernel: block drbd1: Connection closed
Aug 27 21:52:21 srv2-2 kernel: block drbd1: conn( Disconnecting -> StandAlone )
Aug 27 21:52:21 srv2-2 kernel: block drbd1: receiver terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd1: Terminating drbd1_receiver
Aug 27 21:52:21 srv2-2 kernel: block drbd1: disk( Inconsistent -> Diskless )
Aug 27 21:52:21 srv2-2 kernel: block drbd1: drbd_bm_resize called with
capacity == 0
Aug 27 21:52:21 srv2-2 kernel: block drbd1: worker terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd1: Terminating drbd1_worker
Aug 27 21:52:21 srv2-2 kernel: block drbd2: peer_isp( 1 -> 0 )
Aug 27 21:52:21 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
Aug 27 21:52:21 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 21:52:21 srv2-2 kernel: block drbd2: asender terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 21:52:21 srv2-2 kernel: block drbd2: Connection closed
Aug 27 21:52:21 srv2-2 kernel: block drbd2: conn( Disconnecting -> StandAlone )
Aug 27 21:52:21 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd2: Terminating drbd2_receiver
Aug 27 21:52:21 srv2-2 kernel: block drbd2: disk( UpToDate -> Diskless )
Aug 27 21:52:21 srv2-2 kernel: block drbd2: drbd_bm_resize called with
capacity == 0
Aug 27 21:52:21 srv2-2 kernel: block drbd2: worker terminated
Aug 27 21:52:21 srv2-2 kernel: block drbd2: Terminating drbd2_worker
Aug 27 21:59:09 srv2-2 kernel: block drbd1: Starting worker thread
(from cqueue [2561])
Aug 27 21:59:09 srv2-2 kernel: block drbd1: disk( Diskless -> Attaching )
Aug 27 21:59:09 srv2-2 kernel: block drbd1: No usable activity log found.
Aug 27 21:59:09 srv2-2 kernel: block drbd1: Method to ensure write
ordering: barrier
Aug 27 21:59:09 srv2-2 kernel: block drbd1: max_segment_size ( = BIO
size ) = 32768
Aug 27 21:59:09 srv2-2 kernel: block drbd1: drbd_bm_resize called with
capacity == 1450419656
Aug 27 21:59:09 srv2-2 kernel: block drbd1: resync bitmap:
bits=181302457 words=5665702
Aug 27 21:59:09 srv2-2 kernel: block drbd1: size = 692 GB (725209828 KB)
Aug 27 21:59:10 srv2-2 kernel: block drbd1: recounting of set bits
took additional 6 jiffies
Aug 27 21:59:10 srv2-2 kernel: block drbd1: 0 KB (0 bits) marked
out-of-sync by on disk bit-map.
Aug 27 21:59:10 srv2-2 kernel: block drbd1: disk( Attaching -> Inconsistent )
Aug 27 21:59:10 srv2-2 kernel: block drbd2: Starting worker thread
(from cqueue [2561])
Aug 27 21:59:10 srv2-2 kernel: block drbd2: disk( Diskless -> Attaching )
Aug 27 21:59:10 srv2-2 kernel: block drbd2: No usable activity log found.
Aug 27 21:59:10 srv2-2 kernel: block drbd2: Method to ensure write
ordering: barrier
Aug 27 21:59:10 srv2-2 kernel: block drbd2: max_segment_size ( = BIO
size ) = 32768
Aug 27 21:59:10 srv2-2 kernel: block drbd2: drbd_bm_resize called with
capacity == 1465104376
Aug 27 21:59:10 srv2-2 kernel: block drbd2: resync bitmap:
bits=183138047 words=5723064
Aug 27 21:59:10 srv2-2 kernel: block drbd2: size = 699 GB (732552188 KB)
Aug 27 21:59:10 srv2-2 kernel: block drbd2: recounting of set bits
took additional 6 jiffies
Aug 27 21:59:10 srv2-2 kernel: block drbd2: 0 KB (0 bits) marked
out-of-sync by on disk bit-map.
Aug 27 21:59:10 srv2-2 kernel: block drbd2: disk( Attaching -> UpToDate )
Aug 27 21:59:10 srv2-2 kernel: block drbd1: conn( StandAlone -> Unconnected )
Aug 27 21:59:10 srv2-2 kernel: block drbd1: Starting receiver thread
(from drbd1_worker [376])
Aug 27 21:59:10 srv2-2 kernel: block drbd1: receiver (re)started
Aug 27 21:59:10 srv2-2 kernel: block drbd1: conn( Unconnected -> WFConnection )
Aug 27 21:59:10 srv2-2 kernel: block drbd2: conn( StandAlone -> Unconnected )
Aug 27 21:59:10 srv2-2 kernel: block drbd2: Starting receiver thread
(from drbd2_worker [384])
Aug 27 21:59:10 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 21:59:10 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 21:59:11 srv2-2 kernel: block drbd1: Handshake successful:
Agreed network protocol version 92
Aug 27 21:59:11 srv2-2 kernel: block drbd1: conn( WFConnection ->
WFReportParams )
Aug 27 21:59:11 srv2-2 kernel: block drbd1: Starting asender thread
(from drbd1_receiver [398])
Aug 27 21:59:11 srv2-2 kernel: block drbd1: data-integrity-alg: <not-used>
Aug 27 21:59:11 srv2-2 kernel: block drbd1: drbd_sync_handshake:
Aug 27 21:59:11 srv2-2 kernel: block drbd1: self
D3554F9F9A0C63F2:0000000000000000:44A7A13DD9733674:1DF75354805DC515
bits:0 flags:0
Aug 27 21:59:11 srv2-2 kernel: block drbd1: peer
DE0D4E700325B9AB:D3554F9F9A0C63F3:D935FD158B48D359:44A7A13DD9733675
bits:168622 flags:0
Aug 27 21:59:11 srv2-2 kernel: block drbd1: uuid_compare()=-1 by rule 50
Aug 27 21:59:11 srv2-2 kernel: block drbd1: Becoming sync target due
to disk states.
Aug 27 21:59:11 srv2-2 kernel: block drbd1: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 21:59:21 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 21:59:21 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 21:59:21 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [404])
Aug 27 21:59:21 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 21:59:21 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 21:59:21 srv2-2 kernel: block drbd2: self
F099BEA2390E084C:0000000000000000:9A1E199817FB0F6C:C0808022EBB34FD3
bits:0 flags:0
Aug 27 21:59:21 srv2-2 kernel: block drbd2: peer
1AF04270CD442A73:F099BEA2390E084D:9A1E199817FB0F6C:C0808022EBB34FD3
bits:0 flags:0
Aug 27 21:59:21 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 21:59:21 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Aug 27 22:01:58 srv2-2 kernel: block drbd1: conn( WFBitMapT -> WFSyncUUID )
Aug 27 22:01:58 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm before-resync-target minor-1
Aug 27 22:01:58 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
Aug 27 22:01:58 srv2-2 kernel: block drbd1: conn( WFSyncUUID -> SyncTarget )
Aug 27 22:01:58 srv2-2 kernel: block drbd2: aftr_isp( 0 -> 1 )
Aug 27 22:01:58 srv2-2 kernel: block drbd1: Began resync as SyncTarget
(will sync 674492 KB [168623 bits set]).
Aug 27 22:02:23 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 22:02:23 srv2-2 kernel: block drbd2: peer_isp( 0 -> 1 )
Aug 27 22:02:24 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 22:02:24 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:02:24 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 22:02:24 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 22:02:24 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 22:02:24 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 22:02:24 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 22:02:24 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:11:02 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 22:11:02 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
peer_isp( 1 -> 0 )
Aug 27 22:11:02 srv2-2 kernel: block drbd2: asender terminated
Aug 27 22:11:02 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 22:11:02 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 22:11:02 srv2-2 kernel: block drbd2: Connection closed
Aug 27 22:11:02 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 22:11:02 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 22:11:02 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 22:11:02 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 22:11:02 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 22:11:36 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 22:11:36 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 22:11:36 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [404])
Aug 27 22:11:36 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 22:11:36 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 22:11:36 srv2-2 kernel: block drbd2: self
1AF04270CD442A72:0000000000000000:C352B1424CFF5664:F099BEA2390E084D
bits:0 flags:0
Aug 27 22:11:36 srv2-2 kernel: block drbd2: peer
DF2499C4BA4BC311:1AF04270CD442A73:C352B1424CFF5665:F099BEA2390E084D
bits:0 flags:0
Aug 27 22:11:36 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 22:11:36 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
peer_isp( 0 -> 1 )
Aug 27 22:14:43 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 22:14:44 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 22:14:44 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:14:44 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 22:14:44 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 22:14:44 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 22:14:44 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 22:14:44 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 22:14:44 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:25:24 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 22:25:24 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
peer_isp( 1 -> 0 )
Aug 27 22:25:24 srv2-2 kernel: block drbd2: asender terminated
Aug 27 22:25:24 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 22:25:24 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 22:25:24 srv2-2 kernel: block drbd2: Connection closed
Aug 27 22:25:24 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 22:25:24 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 22:25:24 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 22:25:24 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 22:25:24 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 22:25:24 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 22:25:24 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 22:25:24 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [404])
Aug 27 22:25:24 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 22:25:24 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 22:25:24 srv2-2 kernel: block drbd2: self
DF2499C4BA4BC310:0000000000000000:CDC8E05504B8FB14:1AF04270CD442A73
bits:0 flags:0
Aug 27 22:25:24 srv2-2 kernel: block drbd2: peer
3EB6B0593B27167F:DF2499C4BA4BC311:CDC8E05504B8FB14:1AF04270CD442A73
bits:0 flags:0
Aug 27 22:25:24 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 22:25:24 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
peer_isp( 0 -> 1 )
Aug 27 22:25:36 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 22:25:37 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 22:25:37 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:25:37 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 22:25:37 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 22:25:37 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 22:25:37 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 22:25:37 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 22:25:37 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:27:59 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 22:27:59 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
peer_isp( 1 -> 0 )
Aug 27 22:27:59 srv2-2 kernel: block drbd2: asender terminated
Aug 27 22:27:59 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 22:27:59 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 22:27:59 srv2-2 kernel: block drbd2: Connection closed
Aug 27 22:27:59 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 22:27:59 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 22:27:59 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 22:27:59 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 22:27:59 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 22:28:00 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 22:28:00 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 22:28:00 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [404])
Aug 27 22:28:00 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 22:28:01 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 22:28:01 srv2-2 kernel: block drbd2: self
3EB6B0593B27167E:0000000000000000:AA9FD926B6C34724:DF2499C4BA4BC311
bits:0 flags:0
Aug 27 22:28:01 srv2-2 kernel: block drbd2: peer
874C08E599332431:3EB6B0593B27167F:AA9FD926B6C34724:DF2499C4BA4BC311
bits:0 flags:0
Aug 27 22:28:01 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 22:28:01 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
peer_isp( 0 -> 1 )
Aug 27 22:31:12 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 22:31:12 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 22:31:12 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 22:31:12 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 22:31:12 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 22:31:12 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 22:31:12 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 22:31:12 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 22:31:12 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 23:15:20 srv2-2 kernel: block drbd2: PingAck did not arrive in time.
Aug 27 23:15:20 srv2-2 kernel: block drbd2: peer( Primary -> Unknown )
conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )
peer_isp( 1 -> 0 )
Aug 27 23:15:20 srv2-2 kernel: block drbd2: asender terminated
Aug 27 23:15:20 srv2-2 kernel: block drbd2: short read expecting
header on sock: r=-512
Aug 27 23:15:20 srv2-2 kernel: block drbd2: Terminating drbd2_asender
Aug 27 23:15:20 srv2-2 kernel: block drbd2: Connection closed
Aug 27 23:15:20 srv2-2 kernel: block drbd2: conn( NetworkFailure ->
Unconnected )
Aug 27 23:15:20 srv2-2 kernel: block drbd2: receiver terminated
Aug 27 23:15:20 srv2-2 kernel: block drbd2: Restarting drbd2_receiver
Aug 27 23:15:20 srv2-2 kernel: block drbd2: receiver (re)started
Aug 27 23:15:20 srv2-2 kernel: block drbd2: conn( Unconnected -> WFConnection )
Aug 27 23:15:29 srv2-2 kernel: block drbd2: Handshake successful:
Agreed network protocol version 92
Aug 27 23:15:29 srv2-2 kernel: block drbd2: conn( WFConnection ->
WFReportParams )
Aug 27 23:15:29 srv2-2 kernel: block drbd2: Starting asender thread
(from drbd2_receiver [404])
Aug 27 23:15:29 srv2-2 kernel: block drbd2: data-integrity-alg: <not-used>
Aug 27 23:15:29 srv2-2 kernel: block drbd2: drbd_sync_handshake:
Aug 27 23:15:29 srv2-2 kernel: block drbd2: self
874C08E599332430:0000000000000000:ABE6A42F05CD58BA:3EB6B0593B27167F
bits:0 flags:0
Aug 27 23:15:29 srv2-2 kernel: block drbd2: peer
FBC90E77650AF305:874C08E599332431:ABE6A42F05CD58BB:3EB6B0593B27167F
bits:0 flags:0
Aug 27 23:15:29 srv2-2 kernel: block drbd2: uuid_compare()=-1 by rule 50
Aug 27 23:15:29 srv2-2 kernel: block drbd2: peer( Unknown -> Primary )
conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
peer_isp( 0 -> 1 )
Aug 27 23:17:08 srv2-2 kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )
Aug 27 23:17:09 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2
Aug 27 23:17:09 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)
Aug 27 23:17:09 srv2-2 kernel: block drbd2: conn( WFSyncUUID ->
PausedSyncT ) disk( UpToDate -> Inconsistent )
Aug 27 23:17:09 srv2-2 kernel: block drbd2: Began resync as
PausedSyncT (will sync 0 KB [0 bits set]).
Aug 27 23:17:09 srv2-2 kernel: block drbd2: Resync done (total 1 sec;
paused 0 sec; 0 K/sec)
Aug 27 23:17:09 srv2-2 kernel: block drbd2: conn( PausedSyncT ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 23:17:09 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2
Aug 27 23:17:09 srv2-2 kernel: block drbd2: helper command:
/sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)
Aug 27 23:17:14 srv2-2 kernel: block drbd2: md_sync_timer expired!
Worker calls drbd_md_sync().
Aug 27 23:27:28 srv2-2 kernel: block drbd1: Resync done (total 5129
sec; paused 0 sec; 128 K/sec)
Aug 27 23:27:28 srv2-2 kernel: block drbd1: conn( SyncTarget ->
Connected ) disk( Inconsistent -> UpToDate )
Aug 27 23:27:28 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm after-resync-target minor-1
Aug 27 23:27:28 srv2-2 kernel: block drbd1: helper command:
/sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)
Aug 27 23:27:28 srv2-2 kernel: block drbd2: aftr_isp( 1 -> 0 )
Aug 27 23:28:06 srv2-2 kernel: block drbd2: peer_isp( 1 -> 0 )
Aug 28 04:00:01 srv2-2 kernel: block drbd1: conn( Connected -> VerifyS )
Aug 28 04:00:01 srv2-2 kernel: block drbd1: Starting Online Verify from sector 0
Aug 28 04:00:01 srv2-2 kernel: block drbd2: conn( Connected -> VerifyS )
Aug 28 04:00:01 srv2-2 kernel: block drbd2: Starting Online Verify from sector 0
Aug 28 04:00:01 srv2-2 kernel: BUG: unable to handle kernel NULL
pointer dereference at 00000018
Aug 28 04:00:01 srv2-2 kernel: IP: [<c153ca60>] w_e_end_ov_reply+0x50/0x400
Aug 28 04:00:01 srv2-2 kernel: *pdpt = 0000000037112001 *pde = 0000000000000000
Aug 28 04:00:01 srv2-2 kernel: Oops: 0000 [#1] SMP
Aug 28 04:00:01 srv2-2 kernel: last sysfs file:
/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/class
Aug 28 04:00:01 srv2-2 kernel:
Aug 28 04:00:01 srv2-2 kernel: Pid: 376, comm: drbd1_worker Not
tainted 2.6.34.6-xxxx-std-ipv6-32 #3 DG31PR/SP
Aug 28 04:00:01 srv2-2 kernel: EIP: 0060:[<c153ca60>] EFLAGS: 00010202 CPU: 1
Aug 28 04:00:01 srv2-2 kernel: EIP is at w_e_end_ov_reply+0x50/0x400
Aug 28 04:00:01 srv2-2 kernel: EAX: 00000000 EBX: f64ee800 ECX:
f6b35160 EDX: 00000010
Aug 28 04:00:01 srv2-2 kernel: ESI: f64ee800 EDI: f6c6f040 EBP:
f6667f30 ESP: f6667ec4
Aug 28 04:00:01 srv2-2 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Aug 28 04:00:01 srv2-2 kernel: Process drbd1_worker (pid: 376,
ti=f6666000 task=f6ad0a60 task.ti=f6666000)
Aug 28 04:00:01 srv2-2 kernel: Stack:
Aug 28 04:00:01 srv2-2 kernel: c106b67c 7fffffff 00000000 00000286
00000330 00000000 f64ee800 f6667f30
Aug 28 04:00:01 srv2-2 kernel: <0> c153c855 00001000 7fffffff f6b35160
f6aff7c0 c1b95800 f6c6f040 f6667f18
Aug 28 04:00:01 srv2-2 kernel: <0> c180ab2f f6667f7c 00000004 f6667f7c
c1b95800 f6667f30 c17b8132 f6667f7c
Aug 28 04:00:01 srv2-2 kernel: Call Trace:
Aug 28 04:00:01 srv2-2 kernel: [<c106b67c>] ? mod_timer+0xcc/0x150
Aug 28 04:00:01 srv2-2 kernel: [<c153c855>] ? w_make_ov_request+0x195/0x2a0
Aug 28 04:00:01 srv2-2 kernel: [<c180ab2f>] ? tcp_setsockopt+0x4f/0x60
Aug 28 04:00:01 srv2-2 kernel: [<c17b8132>] ? sock_common_setsockopt+0x22/0x30
Aug 28 04:00:01 srv2-2 kernel: [<c1539e06>] ? drbd_worker+0x1a6/0x4d0
Aug 28 04:00:01 srv2-2 kernel: [<c1554a90>] ? drbd_thread_setup+0x0/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1554aba>] ? drbd_thread_setup+0x2a/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1554a90>] ? drbd_thread_setup+0x0/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1075244>] ? kthread+0x74/0x80
Aug 28 04:00:01 srv2-2 kernel: [<c10751d0>] ? kthread+0x0/0x80
Aug 28 04:00:01 srv2-2 kernel: [<c102b736>] ? kernel_thread_helper+0x6/0x10
Aug 28 04:00:01 srv2-2 kernel: Code: 57 18 01 00 8b 55 c0 8b 4d c0 8b
52 30 89 55 c4 8b 41 10 f6 40 10 01 0f 84 ae 02 00 00 8b 5d ac ba 10
00 00 00 8b 83 88 04 00 00 <8b> 40 18 89 45 b0 e8 45 31 b9 ff 85 c0 89
c3 0f 84 6b 01 00 00
Aug 28 04:00:01 srv2-2 kernel: EIP: [<c153ca60>]
w_e_end_ov_reply+0x50/0x400 SS:ESP 0068:f6667ec4
Aug 28 04:00:01 srv2-2 kernel: CR2: 0000000000000018
Aug 28 04:00:01 srv2-2 kernel: ---[ end trace 9fd9b59239ba972a ]---
Aug 28 04:00:01 srv2-2 kernel: BUG: unable to handle kernel NULL
pointer dereference at 00000018
Aug 28 04:00:01 srv2-2 kernel: IP: [<c153ca60>] w_e_end_ov_reply+0x50/0x400
Aug 28 04:00:01 srv2-2 kernel: *pdpt = 0000000033d48001 *pde = 0000000000000000
Aug 28 04:00:01 srv2-2 kernel: Oops: 0000 [#2] SMP
Aug 28 04:00:01 srv2-2 kernel: last sysfs file:
/sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/class
Aug 28 04:00:01 srv2-2 kernel:
Aug 28 04:00:01 srv2-2 kernel: Pid: 384, comm: drbd2_worker Tainted: G
D 2.6.34.6-xxxx-std-ipv6-32 #3 DG31PR/SP
Aug 28 04:00:01 srv2-2 kernel: EIP: 0060:[<c153ca60>] EFLAGS: 00010202 CPU: 1
Aug 28 04:00:01 srv2-2 kernel: EIP is at w_e_end_ov_reply+0x50/0x400
Aug 28 04:00:01 srv2-2 kernel: EAX: 00000000 EBX: f6a9c800 ECX:
f62a8908 EDX: 00000010
Aug 28 04:00:01 srv2-2 kernel: ESI: f6a9c800 EDI: f6c72e40 EBP:
f3bfdf30 ESP: f3bfdec4
Aug 28 04:00:01 srv2-2 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Aug 28 04:00:01 srv2-2 kernel: Process drbd2_worker (pid: 384,
ti=f3bfc000 task=f70d6180 task.ti=f3bfc000)
Aug 28 04:00:01 srv2-2 kernel: Stack:
Aug 28 04:00:01 srv2-2 kernel: c106b67c 7fffffff 00000000 00000286
00000660 00000000 f6a9c800 f3bfdf30
Aug 28 04:00:01 srv2-2 kernel: <0> c153c855 00001000 7fffffff f62a8908
f629bea0 c1b95800 f6c72e40 f3bfdf18
Aug 28 04:00:01 srv2-2 kernel: <0> c180ab2f f3bfdf7c 00000004 f3bfdf7c
c1b95800 f3bfdf30 c17b8132 f3bfdf7c
Aug 28 04:00:01 srv2-2 kernel: Call Trace:
Aug 28 04:00:01 srv2-2 kernel: [<c106b67c>] ? mod_timer+0xcc/0x150
Aug 28 04:00:01 srv2-2 kernel: [<c153c855>] ? w_make_ov_request+0x195/0x2a0
Aug 28 04:00:01 srv2-2 kernel: [<c180ab2f>] ? tcp_setsockopt+0x4f/0x60
Aug 28 04:00:01 srv2-2 kernel: [<c17b8132>] ? sock_common_setsockopt+0x22/0x30
Aug 28 04:00:01 srv2-2 kernel: [<c1539e06>] ? drbd_worker+0x1a6/0x4d0
Aug 28 04:00:01 srv2-2 kernel: [<c1554a90>] ? drbd_thread_setup+0x0/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1554aba>] ? drbd_thread_setup+0x2a/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1554a90>] ? drbd_thread_setup+0x0/0xf0
Aug 28 04:00:01 srv2-2 kernel: [<c1075244>] ? kthread+0x74/0x80
Aug 28 04:00:01 srv2-2 kernel: [<c10751d0>] ? kthread+0x0/0x80
Aug 28 04:00:01 srv2-2 kernel: [<c102b736>] ? kernel_thread_helper+0x6/0x10
Aug 28 04:00:01 srv2-2 kernel: Code: 57 18 01 00 8b 55 c0 8b 4d c0 8b
52 30 89 55 c4 8b 41 10 f6 40 10 01 0f 84 ae 02 00 00 8b 5d ac ba 10
00 00 00 8b 83 88 04 00 00 <8b> 40 18 89 45 b0 e8 45 31 b9 ff 85 c0 89
c3 0f 84 6b 01 00 00
Aug 28 04:00:01 srv2-2 kernel: EIP: [<c153ca60>]
w_e_end_ov_reply+0x50/0x400 SS:ESP 0068:f3bfdec4
Aug 28 04:00:01 srv2-2 kernel: CR2: 0000000000000018
Aug 28 04:00:01 srv2-2 kernel: ---[ end trace 9fd9b59239ba972b ]---