Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
There was another note mentioning backups...
DRBD is designed to protect against server and disk failures. Backups
primarily protect against human errors, disasters and so on - and I do
have backups... Snarky comments aren't very helpful and don't have
much place in civil discourse except maybe with your friends. The fact
that you don't want your system to recover from I/O errors is your
choice. I'm funny that way - I want my system to do all it can to
recover from problems, and minimize data loss...
In this case, I have a disk failure which I am having trouble getting
DRBD to protect me against. I'm perfectly willing to accept that I
should have configured things differently - which would be why I came
here asking for help. In the 10+ years I've been using and recommending
DRBD, it's never come up for me before.
On 09/18/2012 05:16 AM, Lars Ellenberg wrote:
>
> Alan Robertson <alanr at unix.sh> schrieb:
>
>> I have read errors on the primary side, which caused the secondary to
>> go
>> into an "inconsistent" state. This means that the disk which
>> desperately needs backing up, is no longer being backed up (!).
>>
>> In an ideal world, it seems to me what one would like for DRBD to do
>> would be:
>> get the data from the secondary
>> write it to the primary - which often "fixes" read errors
>> continue on syncing everything else to the secondary
> Well, we don't do this yet.
> We detach the faulty disk, and resync when you reattach.
>
> Platform, kernel version, drbd version, configuration and logs...
> ;-)
I actually figured you'd just tell me what I needed to change - so I
didn't go grab them the first time. Nevertheless - mea culpa... ;-)
The original occurrence is lost to antiquity, unfortunately so logs
could only be recent - not when it first happened. I included a good
bit of recent logs from both sides. I grepped out drbd issues. Let me
know what else you want.
What _seems_ to have happened, is that the primary continued on, and the
secondary became inconsistent because the two sides were disconnected.
Attempts to resync the two failed because of the read error on the
primary - making it impossible to switch to the secondary using normal
methods.
----------------------------------------------------------------------------
Linux paul 2.6.38-15-server #66-Ubuntu SMP Tue Aug 14 17:42:23 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux
drbd8-utils
2:8.3.9-1ubuntu1 RAID 1 over tcp/ip for Linux
utilities
----------------------------------------------------------------------------
$ cat /proc/drbd (on 'paul' - primary for the problematic partition)
version: 8.3.9 (api:88/proto:86-95)
srcversion: 8925C35502BC976C622CF7A
0: cs:Connected ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:3173100 nr:0 dw:3244204 dr:20104349 al:5775 bm:836 lo:0 pe:0 ua:0
ap:0 ep:1 wo:f oos:512
1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:1420 nr:5328 dw:6780 dr:6866 al:0 bm:178 lo:0 pe:0 ua:0 ap:0 ep:1
wo:f oos:0
----------------------------------------------------------------------------
paul:/etc/drbd.d $ cat home.res (this is the partition with problems)
resource home {
device /dev/drbd0;
meta-disk internal;
on paul {
disk /dev/paul/home;
address 10.1.1.31:7789;
}
on silas {
disk /dev/silas/home;
address 10.1.1.32:7789;
}
}
----------------------------------------------------------------------------
paul:/etc/drbd.d $ cat etc.res (this partition is a happy camper)
resource etc {
device /dev/drbd1;
meta-disk internal;
on paul {
disk /dev/paul/etc;
address 10.1.1.31:7790;
}
on silas {
disk /dev/silas/etc;
address 10.1.1.32:7790;
}
}
----------------------------------------------------------------------------
paul:/etc/drbd.d $ cat global_common.conf
global {
usage-count yes;
# minor-count dialog-refresh disable-ip-verification
}
common {
protocol C;
handlers {
pri-on-incon-degr
"/usr/lib/drbd/notify-pri-on-incon-degr.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
pri-lost-after-sb
"/usr/lib/drbd/notify-pri-lost-after-sb.sh;
/usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ;
reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh;
/usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger
; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
# split-brain "/usr/lib/drbd/notify-split-brain.sh root";
# out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target
"/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target
/usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}
startup {
# wfc-timeout degr-wfc-timeout outdated-wfc-timeout
wait-after-sb
}
disk {
# on-io-error fencing use-bmbv no-disk-barrier
no-disk-flushes
# no-disk-drain no-md-flushes max-bio-bvecs
}
net {
# sndbuf-size rcvbuf-size timeout connect-int ping-int
ping-timeout max-buffers
# max-epoch-size ko-count allow-two-primaries
cram-hmac-alg shared-secret
# after-sb-0pri after-sb-1pri after-sb-2pri
data-integrity-alg no-tcp-cork
}
syncer {
# rate after al-extents use-rle cpu-mask verify-alg
csums-alg
}
}
----------------------------------------------------------------------------
Sep 17 19:47:32 paul kernel: [164558.023607] block drbd0: Restarting
drbd0_receiver
Sep 17 19:47:32 paul kernel: [164558.023613] block drbd0: receiver
(re)started
Sep 17 19:47:32 paul kernel: [164558.023622] block drbd0: conn(
Unconnected -> WFConnection )
Sep 17 19:47:37 paul kernel: [164563.014702] block drbd1: peer( Primary
-> Secondary )
Sep 17 19:47:39 paul kernel: [164564.994392] block drbd1: peer(
Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate ->
DUnknown )
Sep 17 19:47:39 paul kernel: [164564.994470] block drbd1: asender terminated
Sep 17 19:47:39 paul kernel: [164564.994479] block drbd1: Terminating
drbd1_asender
Sep 17 19:47:39 paul kernel: [164565.009255] block drbd1: Connection closed
Sep 17 19:47:39 paul kernel: [164565.009268] block drbd1: conn( TearDown
-> Unconnected )
Sep 17 19:47:39 paul kernel: [164565.009282] block drbd1: receiver
terminated
Sep 17 19:47:39 paul kernel: [164565.009290] block drbd1: Restarting
drbd1_receiver
Sep 17 19:47:39 paul kernel: [164565.009296] block drbd1: receiver
(re)started
Sep 17 19:47:39 paul kernel: [164565.009305] block drbd1: conn(
Unconnected -> WFConnection )
Sep 17 19:47:40 paul kernel: [164565.740844] block drbd1: role(
Secondary -> Primary )
Sep 17 19:47:40 paul kernel: [164565.746094] block drbd1: Creating new
current UUID
Sep 17 19:47:41 paul kernel: [164566.909040] EXT4-fs (drbd1): mounted
filesystem with ordered data mode. Opts: (null)
Sep 17 19:48:15 paul kernel: [164600.720065] block drbd0: Handshake
successful: Agreed network protocol version 95
Sep 17 19:48:15 paul kernel: [164600.720074] block drbd0: conn(
WFConnection -> WFReportParams )
Sep 17 19:48:15 paul kernel: [164600.720122] block drbd0: Starting
asender thread (from drbd0_receiver [16865])
Sep 17 19:48:15 paul kernel: [164600.720496] block drbd0:
data-integrity-alg: <not-used>
Sep 17 19:48:15 paul kernel: [164600.720530] block drbd0:
max_segment_size ( = BIO size ) = 65536
Sep 17 19:48:15 paul kernel: [164600.720542] block drbd0:
drbd_sync_handshake:
Sep 17 19:48:15 paul kernel: [164600.720550] block drbd0: self
01408D626068BA73:E44E88C02198AD7B:A0415500B675BCC6:D1693A79CE98B05C
bits:234 flags:0
Sep 17 19:48:15 paul kernel: [164600.720560] block drbd0: peer
E44E88C02198AD7A:0000000000000000:0F8964D1877294D0:D3BAACF0A864D9DF
bits:128 flags:0
Sep 17 19:48:15 paul kernel: [164600.720568] block drbd0:
uuid_compare()=1 by rule 70
Sep 17 19:48:15 paul kernel: [164600.720573] block drbd0: Becoming sync
source due to disk states.
Sep 17 19:48:15 paul kernel: [164600.720584] block drbd0: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS )
Sep 17 19:48:15 paul kernel: [164600.770093] block drbd1: Handshake
successful: Agreed network protocol version 95
Sep 17 19:48:15 paul kernel: [164600.770110] block drbd1: conn(
WFConnection -> WFReportParams )
Sep 17 19:48:15 paul kernel: [164600.770172] block drbd1: Starting
asender thread (from drbd1_receiver [11268])
Sep 17 19:48:15 paul kernel: [164600.772228] block drbd1:
data-integrity-alg: <not-used>
Sep 17 19:48:15 paul kernel: [164600.772276] block drbd1:
max_segment_size ( = BIO size ) = 65536
Sep 17 19:48:15 paul kernel: [164600.772313] block drbd1:
drbd_sync_handshake:
Sep 17 19:48:15 paul kernel: [164600.772335] block drbd1: self
359D13E313FE85EB:95D5ABF15E518CF6:7941A513D7A77D8F:809CDAEBF881533E
bits:33 flags:0
Sep 17 19:48:15 paul kernel: [164600.772338] block drbd1: peer
95D5ABF15E518CF6:0000000000000000:7941A513D7A77D8E:809CDAEBF881533E
bits:0 flags:0
Sep 17 19:48:15 paul kernel: [164600.772341] block drbd1:
uuid_compare()=1 by rule 70
Sep 17 19:48:15 paul kernel: [164600.772347] block drbd1: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown ->
Outdated )
Sep 17 19:48:15 paul kernel: [164600.895699] block drbd1: conn(
WFBitMapS -> SyncSource ) pdsk( Outdated -> Inconsistent )
Sep 17 19:48:15 paul kernel: [164600.895722] block drbd1: Began resync
as SyncSource (will sync 132 KB [33 bits set]).
Sep 17 19:48:15 paul kernel: [164601.171438] block drbd1: Resync done
(total 1 sec; paused 0 sec; 132 K/sec)
Sep 17 19:48:15 paul kernel: [164601.171447] block drbd1: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Sep 17 19:48:17 paul kernel: [164602.851727] block drbd0: conn(
WFBitMapS -> SyncSource )
Sep 17 19:48:17 paul kernel: [164602.851735] block drbd0: Began resync
as SyncSource (will sync 936 KB [234 bits set]).
Sep 17 19:48:35 paul kernel: [164620.454638] block drbd0: read: error=-5
s=3305108992s
Sep 17 19:48:35 paul kernel: [164620.454646] block drbd0: Local IO
failed in drbd_endio_read_sec_final.
Sep 17 19:48:35 paul kernel: [164620.472927] block drbd0: Sending
NegRSDReply. sector 3305108992s.
Sep 17 19:48:43 paul kernel: [164628.574424] block drbd0: read: error=-5
s=3607212544s
Sep 17 19:48:43 paul kernel: [164628.574432] block drbd0: Local IO
failed in drbd_endio_read_sec_final.
Sep 17 19:48:43 paul kernel: [164628.577997] block drbd0: Sending
NegRSDReply. sector 3607212544s.
Sep 17 19:48:55 paul kernel: [164640.764407] block drbd0: read: error=-5
s=3611387392s
Sep 17 19:48:55 paul kernel: [164640.764416] block drbd0: Local IO
failed in drbd_endio_read_sec_final.
Sep 17 19:48:55 paul kernel: [164640.767964] block drbd0: read: error=-5
s=3611387648s
Sep 17 19:48:55 paul kernel: [164640.767970] block drbd0: Local IO
failed in drbd_endio_read_sec_final.
Sep 17 19:48:55 paul kernel: [164640.798838] block drbd0: Sending
NegRSDReply. sector 3611387392s.
Sep 17 19:48:55 paul kernel: [164640.802390] block drbd0: Sending
NegRSDReply. sector 3611387648s.
Sep 17 19:48:55 paul kernel: [164640.847560] block drbd0: Resync done
(total 37 sec; paused 0 sec; 24 K/sec)
Sep 17 19:48:55 paul kernel: [164640.847565] block drbd0:
128 failed blocks
Sep 17 19:48:55 paul kernel: [164640.847570] block drbd0: conn(
SyncSource -> Connected )
Sep 17 19:48:55 paul kernel: [164640.847575] block drbd0: cs:Connected
rs_left=128 > rs_total=0 (rs_failed 0)
Sep 17 19:48:57 paul kernel: [164642.827672] block drbd1: role( Primary
-> Secondary )
Sep 17 19:48:59 paul kernel: [164645.027362] block drbd1: peer(
Secondary -> Primary )
On the secondary side (clock is apparently about 9 seconds behind the
primary)
----------------------------------------------------------------------------
syslog.1:Sep 17 19:47:24 silas kernel: [299848.446753] block drbd0:
peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) pdsk(
UpToDate -> DUnknown )
syslog.1:Sep 17 19:47:24 silas kernel: [299848.446780] block drbd0:
short read expecting header on sock: r=-512
syslog.1:Sep 17 19:47:24 silas kernel: [299848.450321] block drbd0:
asender terminated
syslog.1:Sep 17 19:47:24 silas kernel: [299848.450323] block drbd0:
Terminating drbd0_asender
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460113] block drbd0:
Connection closed
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460129] block drbd0:
conn( Disconnecting -> StandAlone )
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460168] block drbd0:
receiver terminated
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460173] block drbd0:
Terminating drbd0_receiver
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460183] block drbd0:
disk( Inconsistent -> Diskless )
syslog.1:Sep 17 19:47:24 silas kernel: [299848.460229] block drbd0:
Sending state for being diskless failed
syslog.1:Sep 17 19:47:24 silas kernel: [299848.464620] block drbd0:
drbd_bm_resize called with capacity == 0
syslog.1:Sep 17 19:47:24 silas kernel: [299848.469299] block drbd0:
worker terminated
syslog.1:Sep 17 19:47:24 silas kernel: [299848.469306] block drbd0:
Terminating drbd0_worker
syslog.1:Sep 17 19:47:24 silas kernel: [299848.475552] block drbd0:
State change failed: Disk state is lower than outdated
syslog.1:Sep 17 19:47:24 silas kernel: [299848.479034] block drbd0:
state = { cs:StandAlone ro:Secondary/Unknown ds:Diskless/DUnknown r--- }
syslog.1:Sep 17 19:47:24 silas kernel: [299848.482594] block drbd0:
wanted = { cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r--- }
syslog.1:Sep 17 19:47:29 silas kernel: [299853.440677] block drbd1:
role( Primary -> Secondary )
syslog.1:Sep 17 19:47:31 silas kernel: [299855.438726] block drbd1:
peer( Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk(
UpToDate -> DUnknown )
syslog.1:Sep 17 19:47:31 silas kernel: [299855.438738] block drbd1:
short read expecting header on sock: r=-512
syslog.1:Sep 17 19:47:31 silas kernel: [299855.442402] block drbd1: meta
connection shut down by peer.
syslog.1:Sep 17 19:47:31 silas kernel: [299855.445947] block drbd1:
asender terminated
syslog.1:Sep 17 19:47:31 silas kernel: [299855.445950] block drbd1:
Terminating drbd1_asender
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446534] block drbd1:
Connection closed
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446538] block drbd1:
conn( Disconnecting -> StandAlone )
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446548] block drbd1:
receiver terminated
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446549] block drbd1:
Terminating drbd1_receiver
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446580] block drbd1:
disk( UpToDate -> Diskless )
syslog.1:Sep 17 19:47:31 silas kernel: [299855.446586] block drbd1:
Sending state for being diskless failed
syslog.1:Sep 17 19:47:31 silas kernel: [299855.450231] block drbd1:
drbd_bm_resize called with capacity == 0
syslog.1:Sep 17 19:47:31 silas kernel: [299855.450260] block drbd1:
worker terminated
syslog.1:Sep 17 19:47:31 silas kernel: [299855.450262] block drbd1:
Terminating drbd1_worker
syslog.1:Sep 17 19:47:31 silas kernel: [299855.455348] block drbd1:
State change failed: Disk state is lower than outdated
syslog.1:Sep 17 19:47:31 silas kernel: [299855.458900] block drbd1:
state = { cs:StandAlone ro:Secondary/Unknown ds:Diskless/DUnknown r--- }
syslog.1:Sep 17 19:47:31 silas kernel: [299855.462473] block drbd1:
wanted = { cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r--- }
syslog.1:Sep 17 19:48:06 silas kernel: [299890.491436] block drbd0:
Starting worker thread (from kworker/u:3 [281])
syslog.1:Sep 17 19:48:06 silas kernel: [299890.491493] block drbd0:
disk( Diskless -> Attaching )
syslog.1:Sep 17 19:48:06 silas kernel: [299890.501156] block drbd0:
Found 4 transactions (192 active extents) in activity log.
syslog.1:Sep 17 19:48:06 silas kernel: [299890.501167] block drbd0:
Method to ensure write ordering: flush
syslog.1:Sep 17 19:48:06 silas kernel: [299890.501176] block drbd0:
max_segment_size ( = BIO size ) = 65536
syslog.1:Sep 17 19:48:06 silas kernel: [299890.501186] block drbd0:
drbd_bm_resize called with capacity == 3714532576
syslog.1:Sep 17 19:48:06 silas kernel: [299890.533761] block drbd0:
resync bitmap: bits=464316572 words=7254947
syslog.1:Sep 17 19:48:06 silas kernel: [299890.533771] block drbd0: size
= 1771 GB (1857266288 KB)
syslog.1:Sep 17 19:48:07 silas kernel: [299891.047728] block drbd0:
recounting of set bits took additional 2 jiffies
syslog.1:Sep 17 19:48:07 silas kernel: [299891.047732] block drbd0: 512
KB (128 bits) marked out-of-sync by on disk bit-map.
syslog.1:Sep 17 19:48:07 silas kernel: [299891.047738] block drbd0:
disk( Attaching -> Inconsistent )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.061534] block drbd0:
conn( StandAlone -> Unconnected )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.061551] block drbd1:
Starting worker thread (from kworker/u:3 [281])
syslog.1:Sep 17 19:48:07 silas kernel: [299891.061560] block drbd0:
Starting receiver thread (from drbd0_worker [16833])
syslog.1:Sep 17 19:48:07 silas kernel: [299891.065398] block drbd1:
disk( Diskless -> Attaching )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.066439] block drbd0:
receiver (re)started
syslog.1:Sep 17 19:48:07 silas kernel: [299891.066453] block drbd0:
conn( Unconnected -> WFConnection )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.073981] block drbd1:
Found 4 transactions (44 active extents) in activity log.
syslog.1:Sep 17 19:48:07 silas kernel: [299891.073984] block drbd1:
Method to ensure write ordering: flush
syslog.1:Sep 17 19:48:07 silas kernel: [299891.073986] block drbd1:
max_segment_size ( = BIO size ) = 65536
syslog.1:Sep 17 19:48:07 silas kernel: [299891.073989] block drbd1:
drbd_bm_resize called with capacity == 41941688
syslog.1:Sep 17 19:48:07 silas kernel: [299891.074151] block drbd1:
resync bitmap: bits=5242711 words=81918
syslog.1:Sep 17 19:48:07 silas kernel: [299891.074154] block drbd1: size
= 20 GB (20970844 KB)
syslog.1:Sep 17 19:48:07 silas kernel: [299891.093950] block drbd1:
recounting of set bits took additional 0 jiffies
syslog.1:Sep 17 19:48:07 silas kernel: [299891.093954] block drbd1: 0 KB
(0 bits) marked out-of-sync by on disk bit-map.
syslog.1:Sep 17 19:48:07 silas kernel: [299891.093961] block drbd1:
disk( Attaching -> Outdated )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.110146] block drbd1:
conn( StandAlone -> Unconnected )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.110178] block drbd1:
Starting receiver thread (from drbd1_worker [16849])
syslog.1:Sep 17 19:48:07 silas kernel: [299891.110630] block drbd1:
receiver (re)started
syslog.1:Sep 17 19:48:07 silas kernel: [299891.110636] block drbd1:
conn( Unconnected -> WFConnection )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163523] block drbd0:
Handshake successful: Agreed network protocol version 95
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163554] block drbd0:
conn( WFConnection -> WFReportParams )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163611] block drbd0:
Starting asender thread (from drbd0_receiver [16850])
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163747] block drbd0:
data-integrity-alg: <not-used>
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163769] block drbd0:
max_segment_size ( = BIO size ) = 65536
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163781] block drbd0:
drbd_sync_handshake:
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163790] block drbd0: self
E44E88C02198AD7A:0000000000000000:0F8964D1877294D0:D3BAACF0A864D9DF
bits:128 flags:0
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163800] block drbd0: peer
01408D626068BA73:E44E88C02198AD7B:A0415500B675BCC6:D1693A79CE98B05C
bits:234 flags:0
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163808] block drbd0:
uuid_compare()=-1 by rule 50
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163814] block drbd0:
Becoming sync target due to disk states.
syslog.1:Sep 17 19:48:07 silas kernel: [299891.163830] block drbd0:
peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk(
DUnknown -> UpToDate )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.215184] block drbd1:
Handshake successful: Agreed network protocol version 95
syslog.1:Sep 17 19:48:07 silas kernel: [299891.215210] block drbd1:
conn( WFConnection -> WFReportParams )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.215322] block drbd1:
Starting asender thread (from drbd1_receiver [16879])
syslog.1:Sep 17 19:48:07 silas kernel: [299891.216076] block drbd1:
data-integrity-alg: <not-used>
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217486] block drbd1:
max_segment_size ( = BIO size ) = 65536
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217506] block drbd1:
drbd_sync_handshake:
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217514] block drbd1: self
95D5ABF15E518CF6:0000000000000000:7941A513D7A77D8E:809CDAEBF881533E
bits:0 flags:0
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217524] block drbd1: peer
359D13E313FE85EB:95D5ABF15E518CF6:7941A513D7A77D8F:809CDAEBF881533E
bits:33 flags:0
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217540] block drbd1:
uuid_compare()=-1 by rule 50
syslog.1:Sep 17 19:48:07 silas kernel: [299891.217553] block drbd1:
peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) pdsk(
DUnknown -> UpToDate )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.335797] block drbd1:
conn( WFBitMapT -> WFSyncUUID )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.346580] block drbd1:
helper command: /sbin/drbdadm before-resync-target minor-1
syslog.1:Sep 17 19:48:07 silas kernel: [299891.356614] block drbd1:
helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)
syslog.1:Sep 17 19:48:07 silas kernel: [299891.356630] block drbd1:
conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.356645] block drbd1:
Began resync as SyncTarget (will sync 132 KB [33 bits set]).
syslog.1:Sep 17 19:48:07 silas kernel: [299891.632642] block drbd1:
Resync done (total 1 sec; paused 0 sec; 132 K/sec)
syslog.1:Sep 17 19:48:07 silas kernel: [299891.632663] block drbd1:
conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
syslog.1:Sep 17 19:48:07 silas kernel: [299891.636830] block drbd1:
helper command: /sbin/drbdadm after-resync-target minor-1
syslog.1:Sep 17 19:48:07 silas kernel: [299891.644645] block drbd1:
helper command: /sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)
syslog.1:Sep 17 19:48:09 silas kernel: [299893.289450] block drbd0:
conn( WFBitMapT -> WFSyncUUID )
syslog.1:Sep 17 19:48:09 silas kernel: [299893.308154] block drbd0:
helper command: /sbin/drbdadm before-resync-target minor-0
syslog.1:Sep 17 19:48:09 silas kernel: [299893.312519] block drbd0:
helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
syslog.1:Sep 17 19:48:09 silas kernel: [299893.312533] block drbd0:
conn( WFSyncUUID -> SyncTarget )
syslog.1:Sep 17 19:48:09 silas kernel: [299893.312547] block drbd0:
Began resync as SyncTarget (will sync 936 KB [234 bits set]).
syslog.1:Sep 17 19:48:47 silas kernel: [299931.290372] block drbd0:
Resync done (total 37 sec; paused 0 sec; 24 K/sec)
syslog.1:Sep 17 19:48:47 silas kernel: [299931.290381] block
drbd0: 128 failed blocks
syslog.1:Sep 17 19:48:47 silas kernel: [299931.290393] block drbd0:
conn( SyncTarget -> Connected )
syslog.1:Sep 17 19:48:47 silas kernel: [299931.290406] block drbd0:
helper command: /sbin/drbdadm after-resync-target minor-0
syslog.1:Sep 17 19:48:47 silas kernel: [299931.294522] block drbd0:
helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
syslog.1:Sep 17 19:48:47 silas kernel: [299931.294534] block drbd0:
cs:Connected rs_left=128 > rs_total=0 (rs_failed 0)
syslog.1:Sep 17 19:48:49 silas kernel: [299933.299375] block drbd1:
peer( Primary -> Secondary )
syslog.1:Sep 17 19:48:51 silas kernel: [299935.469595] block drbd1:
role( Secondary -> Primary )
syslog.1:Sep 17 19:48:53 silas kernel: [299937.002401] EXT4-fs (drbd1):
mounted filesystem with ordered data mode. Opts: (null)
--
Alan Robertson <alanr at unix.sh> - @OSSAlanR
"Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions." - William Wilberforce