<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi,</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I have been working with a 4 node cluster and have noticed some FIXME messages in the logs.</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Maybe the same as <a href="https://lists.linbit.com/pipermail/drbd-user/2019-June/025078.html" id="LPlnk396430">
https://lists.linbit.com/pipermail/drbd-user/2019-June/025078.html</a></div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
drbd90 kernel module version:9.0.22-2(later updated to 9.0.25-1)<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
drbd90-utils:9.12.2-1</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
kernel:3.10.0-1127.18.2.el7.x86_64</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
pacemaker:1.1.21-4</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
corosync-2.4.5-4</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
system is centos:7.6<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
host is VMware ESXi<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
FIXME messages seem to happen during initial sync/resync.<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Example from system log:</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Sep 21 17:02:17 cdc0 kernel: drbd r0 dh-cdc0-n3: Handshake to peer 3 successful: Agreed network protocol version 116
<div>Sep 21 17:02:17 cdc0 kernel: drbd r0 dh-cdc0-n3: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.</div>
<div>Sep 21 17:02:17 cdc0 kernel: drbd r0 dh-cdc0-n3: Starting ack_recv thread (from drbd_r_r0 [16881])</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0: Preparing cluster-wide state change 1184555410 (1->3 499/145)</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0: State change 1184555410: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF1</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0: Committing cluster-wide state change 1184555410 (1365ms)</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0 dh-cdc0-n3: conn( Connecting -> Connected ) peer( Unknown -> Secondary )</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: WRITE_SAME disabled by config</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: drbd_sync_handshake:</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: self 52067E32B742047A:FCB41408CC57E195:0000000000000000:0000000000000000 bits:994 flags:20</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: peer 0000000000000004:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:24</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: uuid_compare()=source-set-bitmap by rule 30</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: Writing the whole bitmap, full sync required after drbd_sync_handshake.</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[18134] op clear, bitmap locked for 'set_n_write from sync_handshake' by drbd_r_r0[16881]</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: pdsk( DUnknown -> Inconsistent ) repl( Off -> WFBitMapS )</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 21(1), total 21; compression: 100.0%</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: helper command: /sbin/drbdadm before-resync-source</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: helper command: /sbin/drbdadm before-resync-source exit code 0</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: repl( WFBitMapS -> SyncSource )</div>
<div>Sep 21 17:02:18 cdc0 kernel: drbd r0/0 drbd0 dh-cdc0-n3: Began resync as SyncSource (will sync 5144576 KB [1286144 bits set]).</div>
<div>Sep 21 17:02:19 cdc0 kernel: drbd r0 dh-cdc0-n2: Preparing remote state change 3124969320</div>
<div>Sep 21 17:02:19 cdc0 kernel: drbd r0 dh-cdc0-n2: Committing remote state change 3124969320 (primary_nodes=2)</div>
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I uninstalled the 9.0.22 kernel module rpm and compiled and installed the 9.0.25-1 version on all 4 nodes and resynced the partition(using 'drbdadm invalidate all' on the 3 secondary nodes) but I still see the FIXME messages:</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0 dh-cdc0-n3: Preparing remote state change 3729689878
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0 dh-cdc0-n3: Committing remote state change 3729689878 (primary_nodes=2)</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: pdsk( UpToDate -> Consistent ) repl( Established -> StartingSyncS )</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: pdsk( Consistent -> Outdated )</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[27396] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: helper command: /sbin/drbdadm before-resync-source</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: helper command: /sbin/drbdadm before-resync-source exit code 0</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: pdsk( Outdated -> Inconsistent ) repl( StartingSyncS -> SyncSource )</div>
<div>Oct 6 12:02:19 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n3: Began resync as SyncSource (will sync 5144576 KB [1286144 bits set]).</div>
<div>Oct 6 12:02:21 dh-cdc0-n1 kernel: drbd r0/0 drbd0: new current UUID: 76D23070780C9DB1 weak: FFFFFFFFFFFFFFE1</div>
<div>Oct 6 12:02:23 dh-cdc0-n1 crmd[24427]: notice: State transition S_IDLE -> S_POLICY_ENGINE</div>
<div>Oct 6 12:02:23 dh-cdc0-n1 pengine[24426]: notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-53.bz2</div>
<div>Oct 6 12:02:23 dh-cdc0-n1 crmd[24427]: notice: Transition 3 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-53.bz2): Complete</div>
<div>Oct 6 12:02:23 dh-cdc0-n1 crmd[24427]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE</div>
<div>Oct 6 12:02:24 dh-cdc0-n1 crmd[24427]: notice: State transition S_IDLE -> S_POLICY_ENGINE</div>
<div>Oct 6 12:02:24 dh-cdc0-n1 pengine[24426]: notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-54.bz2</div>
<div>Oct 6 12:02:24 dh-cdc0-n1 crmd[24427]: notice: Transition 4 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-54.bz2): Complete</div>
<div>Oct 6 12:02:24 dh-cdc0-n1 crmd[24427]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0 dh-cdc0-n4: Preparing remote state change 1158604591</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0 dh-cdc0-n4: Committing remote state change 1158604591 (primary_nodes=2)</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: pdsk( UpToDate -> Consistent ) repl( Established -> StartingSyncS )</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: pdsk( Consistent -> Outdated )</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0: FIXME drbd_a_r0[28404] op clear, bitmap locked for 'set_n_write from StartingSync' by drbd_w_r0[24628]</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0: bitmap page idx 11 changed during IO!</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: helper command: /sbin/drbdadm before-resync-source</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: pdsk( Outdated -> Inconsistent )</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: helper command: /sbin/drbdadm before-resync-source exit code 0</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: repl( StartingSyncS -> SyncSource )</div>
<div>Oct 6 12:02:30 dh-cdc0-n1 kernel: drbd r0/0 drbd0 dh-cdc0-n4: Began resync as SyncSource (will sync 5144576 KB [1286144 bits set]).</div>
<div>Oct 6 12:02:36 dh-cdc0-n1 crmd[24427]: notice: Current ping state: S_IDLE</div>
<div>Oct 6 12:02:40 dh-cdc0-n1 crmd[24427]: notice: State transition S_IDLE -> S_POLICY_ENGINE</div>
<div>Oct 6 12:02:40 dh-cdc0-n1 pengine[24426]: notice: Calculated transition 5, saving inputs in /var/lib/pacemaker/pengine/pe-input-55.bz2</div>
<div>Oct 6 12:02:40 dh-cdc0-n1 crmd[24427]: notice: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-55.bz2): Complete</div>
<div>Oct 6 12:02:40 dh-cdc0-n1 crmd[24427]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE</div>
<div>Oct 6 12:02:41 dh-cdc0-n1 kernel: drbd r0/0 drbd0: new current UUID: 6BB40F1CCF845C01 weak: FFFFFFFFFFFFFFE1</div>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Are these messages a problem or just noise?</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Regards,</div>
<div style="font-family: "Courier New", monospace; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Jeremy<br>
</div>
</body>
</html>