Hello everyone,<div><br></div><div>I am still currently testing DRBD + OCFS2 in both master nodes and notices that if the networks interface is sort of saturated DRBD drop the connection.</div><div><br></div><div>Node A and B both centos 6 - drbd-8.3.12 build from vanilla source kernel 2.6.37.6 with vserver patch vs2.3.0.37-rc5</div>
<div>It has happened before I do not know what the cause but I suspect network related problems. Today I was just running rsync a large directory from A to another host (not B) but using the same interface that drbd is using (that is, I do not have dedicated interface for drbd). The rsync takes long time I I found in dmesg </div>
<div><br></div><div><div>[ 1844.822895] block drbd0: Resync done (total 1 sec; paused 0 sec; 8 K/sec)</div><div>[ 1844.822904] block drbd0: updated UUIDs 56D7AD5A0D1052FC:0000000000000000:34133820BE9332EE:34123820BE9332EF</div>
<div>[ 1844.822910] block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) </div><div>[ 1844.823072] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0</div><div>[ 1844.825128] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)</div>
<div>[ 1844.825584] block drbd0: bitmap WRITE of 22 pages took 0 jiffies</div><div>[ 1844.825592] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>[ 1893.341247] block drbd0: peer( Secondary -> Primary ) </div>
<div>[ 1899.914431] block drbd0: role( Secondary -> Primary ) </div><div>[ 2639.422664] Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-dummy0 instead</div>
<div>[ 2648.421314] dummy0: no IPv6 routers present</div><div>[72370.860513] block drbd0: role( Primary -> Secondary ) </div><div>[72370.860600] block drbd0: bitmap WRITE of 0 pages took 0 jiffies</div><div>[72370.860611] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div>
<div>[72393.260995] block drbd0: Considerable difference in lower level device sizes: 37747512s vs. 6291192s</div><div>[72491.794195] block drbd0: role( Secondary -> Primary ) </div><div>[72566.421735] block drbd0: Considerable difference in lower level device sizes: 37747512s vs. 6291192s</div>
<div>[72664.997280] block drbd0: role( Primary -> Secondary ) </div><div>[72664.997365] block drbd0: bitmap WRITE of 0 pages took 0 jiffies</div><div>[72664.997375] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div>
<div>[72708.334128] block drbd0: Considerable difference in lower level device sizes: 37747512s vs. 6291192s</div><div>[72708.334134] block drbd0: drbd_bm_resize called with capacity == 37747512</div><div>[72708.334238] block drbd0: resync bitmap: bits=4718439 words=73726 pages=144</div>
<div>[72708.334242] block drbd0: size = 18 GB (18873756 KB)</div><div>[72708.334247] block drbd0: Writing the whole bitmap, size changed and md moved</div><div>[72708.335349] block drbd0: bitmap WRITE of 120 pages took 1 jiffies</div>
<div>[72708.335357] block drbd0: 15 GB (3932040 bits) marked out-of-sync by on disk bit-map.</div><div>[72708.335564] block drbd0: Resync of new storage after online grow</div><div>[72708.335573] block drbd0: conn( Connected -> WFSyncUUID ) disk( UpToDate -> Outdated ) </div>
<div>[72708.339931] block drbd0: updated sync uuid 067858B97D1E599C:0000000000000000:34133820BE9332EE:34123820BE9332EF</div><div>[72708.340130] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0</div>
<div>[72708.342435] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)</div><div>[72708.342443] block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) </div>
<div>[72708.342454] block drbd0: Began resync as SyncTarget (will sync 15728160 KB [3932040 bits set]).</div><div>[72742.631740] block drbd0: role( Secondary -> Primary ) </div><div>[75037.775246] block drbd0: Resync done (total 2335 sec; paused 0 sec; 6732 K/sec)</div>
<div>[75037.775254] block drbd0: updated UUIDs 56D7AD5A0D1052FD:0000000000000000:067858B97D1E599D:34133820BE9332EF</div><div>[75037.775262] block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) </div>
<div>[75037.775455] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0</div><div>[75037.777818] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)</div><div>[75037.777837] block drbd0: bitmap WRITE of 0 pages took 0 jiffies</div>
<div>[75037.777846] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>[759301.427734] block drbd0: sock was shut down by peer</div><div>[759301.427743] block drbd0: peer( Primary -> Unknown ) conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown ) </div>
<div>[759301.427751] block drbd0: short read expecting header on sock: r=0</div><div>[759301.427793] block drbd0: meta connection shut down by peer.</div><div>[759301.427812] block drbd0: asender terminated</div><div>[759301.427814] block drbd0: Terminating asender thread</div>
<div>[759301.427848] block drbd0: new current UUID 4E3A8EF514190575:56D7AD5A0D1052FD:067858B97D1E599D:34133820BE9332EF</div><div>[759301.428512] block drbd0: Connection closed</div><div>[759301.428518] block drbd0: conn( BrokenPipe -> Unconnected ) </div>
<div>[759301.428523] block drbd0: receiver terminated</div><div>[759301.428526] block drbd0: Restarting receiver thread</div><div>[759301.428528] block drbd0: receiver (re)started</div><div>[759301.428532] block drbd0: conn( Unconnected -> WFConnection ) </div>
<div>[759302.119164] block drbd0: Handshake successful: Agreed network protocol version 96</div><div>[759302.119192] block drbd0: conn( WFConnection -> WFReportParams ) </div><div>[759302.119342] block drbd0: Starting asender thread (from drbd0_receiver [2247])</div>
<div>[759302.119563] block drbd0: data-integrity-alg: <not-used></div><div>[759302.119671] block drbd0: drbd_sync_handshake:</div><div>[759302.119676] block drbd0: self 4E3A8EF514190575:56D7AD5A0D1052FD:067858B97D1E599D:34133820BE9332EF bits:0 flags:0</div>
<div>[759302.119681] block drbd0: peer B8D842CF6950FAB3:56D7AD5A0D1052FD:067858B97D1E599C:34133820BE9332EF bits:1 flags:0</div><div>[759302.119685] block drbd0: uuid_compare()=100 by rule 90</div><div>[759302.119689] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0</div>
<div>[759302.121991] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)</div><div>[759302.121995] block drbd0: Split-Brain detected but unresolved, dropping connection!</div><div>[759302.122016] block drbd0: helper command: /sbin/drbdadm split-brain minor-0</div>
<div>[759302.124199] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)</div><div>[759302.124205] block drbd0: conn( WFReportParams -> Disconnecting ) </div><div>[759302.124213] block drbd0: error receiving ReportState, l: 4!</div>
<div>[759302.124242] block drbd0: asender terminated</div><div>[759302.124247] block drbd0: Terminating asender thread</div><div>[759302.124288] block drbd0: Connection closed</div><div>[759302.124294] block drbd0: conn( Disconnecting -> StandAlone ) </div>
<div>[759302.124401] block drbd0: receiver terminated</div><div>[759302.124405] block drbd0: Terminating receiver thread</div><div>[842477.436163] block drbd0: role( Primary -> Secondary ) </div><div>[842477.436245] block drbd0: bitmap WRITE of 0 pages took 0 jiffies</div>
<div>[842477.436256] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>[842477.436817] block drbd0: disk( UpToDate -> Failed ) </div><div>[842477.436906] block drbd0: disk( Failed -> Diskless ) </div>
<div>[842477.437531] block drbd0: drbd_bm_resize called with capacity == 0</div><div>[842477.437591] block drbd0: worker terminated</div><div>[842477.437594] block drbd0: Terminating worker thread</div><div>[842477.590493] drbd: module cleanup done.</div>
<div>[842488.657890] drbd: initialized. Version: 8.3.12 (api:88/proto:86-96)</div><div>[842488.657893] drbd: GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by root@cosmos, 2011-12-25 16:56:17</div><div>[842488.657896] drbd: registered as block device major 147</div>
<div>[842488.657898] drbd: minor_table @ 0xffff88041c35e100</div><div>[842488.680349] block drbd0: Starting worker thread (from kworker/u:3 [345])</div><div>[842488.680729] block drbd0: disk( Diskless -> Attaching ) </div>
<div>[842488.694650] block drbd0: No usable activity log found.</div><div>[842488.694655] block drbd0: Method to ensure write ordering: barrier</div><div>[842488.694658] block drbd0: max BIO size = 131072</div><div>[842488.694664] block drbd0: drbd_bm_resize called with capacity == 37747512</div>
<div>[842488.694806] block drbd0: resync bitmap: bits=4718439 words=73726 pages=144</div><div>[842488.694809] block drbd0: size = 18 GB (18873756 KB)</div><div>[842488.703982] block drbd0: bitmap READ of 144 pages took 1 jiffies</div>
<div>[842488.704114] block drbd0: recounting of set bits took additional 0 jiffies</div><div>[842488.704117] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>[842488.704123] block drbd0: disk( Attaching -> UpToDate ) </div>
<div>[842488.704127] block drbd0: attached to UUIDs 4E3A8EF514190575:56D7AD5A0D1052FD:067858B97D1E599D:34133820BE9332EF</div><div>[842488.707024] block drbd0: conn( StandAlone -> Unconnected ) </div><div>[842488.707043] block drbd0: Starting receiver thread (from drbd0_worker [20773])</div>
<div>[842488.707244] block drbd0: receiver (re)started</div><div>[842488.707253] block drbd0: conn( Unconnected -> WFConnection ) </div><div>[842489.201687] block drbd0: Handshake successful: Agreed network protocol version 96</div>
<div>[842489.201713] block drbd0: conn( WFConnection -> WFReportParams ) </div><div>[842489.201905] block drbd0: Starting asender thread (from drbd0_receiver [20782])</div><div>[842489.202075] block drbd0: data-integrity-alg: <not-used></div>
<div>[842489.202090] block drbd0: drbd_sync_handshake:</div><div>[842489.202094] block drbd0: self 4E3A8EF514190574:56D7AD5A0D1052FD:067858B97D1E599D:34133820BE9332EF bits:0 flags:0</div><div>[842489.202099] block drbd0: peer B8D842CF6950FAB2:56D7AD5A0D1052FD:067858B97D1E599C:34133820BE9332EF bits:36737 flags:0</div>
<div>[842489.202102] block drbd0: uuid_compare()=100 by rule 90</div><div>[842489.202107] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0</div><div>[842489.204369] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)</div>
<div>[842489.204374] block drbd0: Split-Brain detected, 0 primaries, automatically solved. Sync from peer node</div><div>[842489.204381] block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate ) </div>
<div>[842489.297122] block drbd0: conn( WFBitMapT -> WFSyncUUID ) </div><div>[842489.307270] block drbd0: updated sync uuid 56D8AD5A0D1052FC:0000000000000000:067858B97D1E599D:34133820BE9332EF</div><div>[842489.307479] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0</div>
<div>[842489.309577] block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)</div><div>[842489.309585] block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent ) </div>
<div>[842489.309595] block drbd0: Began resync as SyncTarget (will sync 146948 KB [36737 bits set]).</div><div>[842508.372386] block drbd0: Resync done (total 19 sec; paused 0 sec; 7732 K/sec)</div><div>[842508.372393] block drbd0: updated UUIDs B8D842CF6950FAB2:0000000000000000:56D8AD5A0D1052FC:56D7AD5A0D1052FD</div>
<div>[842508.372398] block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate ) </div><div>[842508.372485] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0</div><div>[842508.374683] block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)</div>
<div>[842508.375756] block drbd0: bitmap WRITE of 131 pages took 0 jiffies</div><div>[842508.375765] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.</div><div>[842525.184360] block drbd0: peer( Secondary -> Primary ) </div>
<div>[842565.780288] block drbd0: role( Secondary -> Primary ) </div><div>[1871271.922857] block drbd0: sock was shut down by peer</div><div>[1871271.922866] block drbd0: peer( Primary -> Unknown ) conn( Connected -> BrokenPipe ) pdsk( UpToDate -> DUnknown ) </div>
<div>[1871271.922873] block drbd0: short read expecting header on sock: r=0</div><div>[1871271.922900] block drbd0: new current UUID 2EDBDEEE9284642B:B8D842CF6950FAB3:56D8AD5A0D1052FC:56D7AD5A0D1052FD</div><div>[1871271.922908] block drbd0: meta connection shut down by peer.</div>
<div>[1871271.922929] block drbd0: asender terminated</div><div>[1871271.922932] block drbd0: Terminating asender thread</div><div>[1871271.923606] block drbd0: Connection closed</div><div>[1871271.923612] block drbd0: conn( BrokenPipe -> Unconnected ) </div>
<div>[1871271.923618] block drbd0: receiver terminated</div><div>[1871271.923620] block drbd0: Restarting receiver thread</div><div>[1871271.923623] block drbd0: receiver (re)started</div><div>[1871271.923627] block drbd0: conn( Unconnected -> WFConnection ) </div>
<div>[1871272.616257] block drbd0: Handshake successful: Agreed network protocol version 96</div><div>[1871272.616282] block drbd0: conn( WFConnection -> WFReportParams ) </div><div>[1871272.616433] block drbd0: Starting asender thread (from drbd0_receiver [20782])</div>
<div>[1871272.616598] block drbd0: data-integrity-alg: <not-used></div><div>[1871272.616829] block drbd0: drbd_sync_handshake:</div><div>[1871272.616835] block drbd0: self 2EDBDEEE9284642B:B8D842CF6950FAB3:56D8AD5A0D1052FC:56D7AD5A0D1052FD bits:0 flags:0</div>
<div>[1871272.616839] block drbd0: peer C301948203A19D2F:B8D842CF6950FAB3:56D8AD5A0D1052FD:56D7AD5A0D1052FD bits:408 flags:0</div><div>[1871272.616843] block drbd0: uuid_compare()=100 by rule 90</div><div>[1871272.616848] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0</div>
<div>[1871272.619242] block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)</div><div>[1871272.619247] block drbd0: Split-Brain detected but unresolved, dropping connection!</div><div>[1871272.619268] block drbd0: helper command: /sbin/drbdadm split-brain minor-0</div>
<div>[1871272.621532] block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)</div><div>[1871272.621538] block drbd0: conn( WFReportParams -> Disconnecting ) </div><div>[1871272.621547] block drbd0: error receiving ReportState, l: 4!</div>
<div>[1871272.621627] block drbd0: asender terminated</div><div>[1871272.621634] block drbd0: Terminating asender thread</div><div>[1871272.621817] block drbd0: Connection closed</div><div>[1871272.621825] block drbd0: conn( Disconnecting -> StandAlone ) </div>
<div>[1871272.622039] block drbd0: receiver terminated</div><div>[1871272.622043] block drbd0: Terminating receiver thread</div><div><br></div><div><br></div><div>and now I have</div><div><br></div><div><div>cat /proc/drbd </div>
<div>version: 8.3.12 (api:88/proto:86-96)</div><div>GIT-hash: e2a8ef4656be026bbae540305fcb998a5991090f build by root@cosmos, 2011-12-25 16:56:17</div><div> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----</div>
<div> ns:0 nr:107240176 dw:107240176 dr:664 al:0 bm:24 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0</div></div><div><br></div><div><br></div><div>I do not think it is expected? how can I prevent this</div><div><br></div><div>Thanks in advance</div>
<div><br></div><div><br></div>-- <br>Steve Kieu<br>
</div>