Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi all, can someone explain what happened here. Secondary node disconnected and did not connect back (became Standalone) . Also there was no email notification (I have those handlers in config: pri-on-incon-degr, pri-lost-after-sb, local-io-error, split-brain). I was lucky to notice it. Is this bug? Centos 6.1, drbd 8.3 from elrepo Thanks. Dec 13 09:56:32 kvm kernel: block drbd2: Digest integrity check FAILED: 275489360s +4096 Dec 13 09:56:32 kvm kernel: block drbd2: error receiving Data, l: 4124! Dec 13 09:56:32 kvm kernel: block drbd2: peer( Primary -> Unknown ) conn( Connected -> ProtocolError ) pdsk( UpToDate -> DUnknown ) Dec 13 09:56:32 kvm kernel: block drbd2: asender terminated Dec 13 09:56:32 kvm kernel: block drbd2: Terminating asender thread Dec 13 09:56:32 kvm kernel: block drbd2: Connection closed Dec 13 09:56:32 kvm kernel: block drbd2: conn( ProtocolError -> Unconnected ) Dec 13 09:56:32 kvm kernel: block drbd2: receiver terminated Dec 13 09:56:32 kvm kernel: block drbd2: Restarting receiver thread Dec 13 09:56:32 kvm kernel: block drbd2: receiver (re)started Dec 13 09:56:32 kvm kernel: block drbd2: conn( Unconnected -> WFConnection ) Dec 13 09:56:32 kvm kernel: block drbd2: Handshake successful: Agreed network protocol version 96 Dec 13 09:56:32 kvm kernel: block drbd2: conn( WFConnection -> WFReportParams ) Dec 13 09:56:32 kvm kernel: block drbd2: Starting asender thread (from drbd2_receiver [7791]) Dec 13 09:56:32 kvm kernel: block drbd2: data-integrity-alg: crc32c Dec 13 09:56:32 kvm kernel: block drbd2: drbd_sync_handshake: Dec 13 09:56:32 kvm kernel: block drbd2: self FD8148B5BCB1E19C:0000000000000000:5C43BE86AC37E9FC:5C42BE86AC37E9FD bits:0 flags:0 Dec 13 09:56:32 kvm kernel: block drbd2: peer E44EAC3D29F4825F:FD8148B5BCB1E19D:5C43BE86AC37E9FD:5C42BE86AC37E9FD bits:82 flags:0 Dec 13 09:56:32 kvm kernel: block drbd2: uuid_compare()=-1 by rule 50 Dec 13 09:56:32 kvm kernel: block drbd2: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate ) Dec 13 09:56:32 kvm kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID ) Dec 13 09:56:32 kvm kernel: block drbd2: updated sync uuid FD8248B5BCB1E19C:0000000000000000:5C43BE86AC37E9FC:5C42BE86AC37E9FD Dec 13 09:56:32 kvm kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 Dec 13 09:56:32 kvm kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0) Dec 13 09:56:32 kvm kernel: block drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) Dec 13 09:56:32 kvm kernel: block drbd2: Began resync as SyncTarget (will sync 336 KB [84 bits set]). Dec 13 09:56:33 kvm kernel: block drbd2: BAD! sector=216526720s enr=6607 rs_left=-1 rs_failed=0 count=1 Dec 13 09:56:33 kvm kernel: Pid: 9607, comm: drbd2_asender Tainted: G W ---------------- 2.6.32-131.21.1.el6.x86_64 #1 Dec 13 09:56:33 kvm kernel: Call Trace: Dec 13 09:56:33 kvm kernel: [<ffffffffa046426a>] ? drbd_try_clear_on_disk_bm+0x3ca/0x4e0 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffff8100bace>] ? common_interrupt+0xe/0x13 Dec 13 09:56:33 kvm kernel: [<ffffffffa0444d85>] ? _drbd_bm_total_weight+0x45/0xf0 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffffa0464723>] ? __drbd_set_in_sync+0x153/0x270 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffffa045163f>] ? e_end_resync_block+0x4f/0x130 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffffa0453de2>] ? drbd_process_done_ee+0x112/0x1a0 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffff8107dda2>] ? flush_sigqueue+0x42/0x60 Dec 13 09:56:33 kvm kernel: [<ffffffffa0455b8c>] ? drbd_asender+0x10c/0x960 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffff8107960c>] ? lock_timer_base+0x3c/0x70 Dec 13 09:56:33 kvm kernel: [<ffffffff8107a0bb>] ? try_to_del_timer_sync+0x7b/0xe0 Dec 13 09:56:33 kvm kernel: [<ffffffff8107a142>] ? del_timer_sync+0x22/0x30 Dec 13 09:56:33 kvm kernel: [<ffffffff814dbbda>] ? schedule_timeout+0x19a/0x2e0 Dec 13 09:56:33 kvm kernel: [<ffffffff81079720>] ? process_timeout+0x0/0x10 Dec 13 09:56:33 kvm kernel: [<ffffffffa046a77c>] ? drbd_thread_setup+0xdc/0x260 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffff8100c1ca>] ? child_rip+0xa/0x20 Dec 13 09:56:33 kvm kernel: [<ffffffffa046a6a0>] ? drbd_thread_setup+0x0/0x260 [drbd] Dec 13 09:56:33 kvm kernel: [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 Dec 13 09:56:33 kvm kernel: block drbd2: peer( Primary -> Unknown ) conn( SyncTarget -> Disconnecting ) pdsk( UpToDate -> DUnknown ) Dec 13 09:56:33 kvm kernel: block drbd2: asender terminated Dec 13 09:56:33 kvm kernel: block drbd2: Terminating asender thread Dec 13 09:56:33 kvm kernel: block drbd2: bitmap WRITE of 1997 pages took 144 jiffies Dec 13 09:56:33 kvm kernel: block drbd2: 28 KB (7 bits) marked out-of-sync by on disk bit-map. Dec 13 09:56:33 kvm kernel: block drbd2: Connection closed Dec 13 09:56:33 kvm kernel: block drbd2: conn( Disconnecting -> StandAlone ) Dec 13 09:56:33 kvm kernel: block drbd2: receiver terminated Dec 13 09:56:33 kvm kernel: block drbd2: Terminating receiver thread -- Vladimir Vassiliev