<div dir="ltr">Thanks Mats, this was the reason for me to give up on drbd 9 when I was testing about 2-3 months ago.<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 30, 2016 at 9:50 PM, Mats Ramnefors <span dir="ltr"><<a href="mailto:mats@ramnefors.com" target="_blank">mats@ramnefors.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Patch below.<div><br></div><div>For me it did not work to apply the patch ”as is”. I had to manually edit in the changes. </div><div><br></div><div>Do not understand why, but I am no expert on patching so it may work for you.</div><div><br></div><div>With the changes applied, the RA seems to work OK.</div><div><br></div><div>/Mats<br><div><br></div><div><div>++++++ support-drbd9-ra.patch ++++++</div><div>diff --git a/scripts/drbd.ocf b/scripts/drbd.ocf</div><div>index 632e16e..91990fc 100755</div><div>--- a/scripts/drbd.ocf</div><div>+++ b/scripts/drbd.ocf</div><div>@@ -328,6 +328,23 @@ remove_master_score() {</div><div> do_cmd ${HA_SBIN_DIR}/crm_master -l reboot -D</div><div> }</div><div> </div><div>+_peer_node_process() {</div><div>+ # _since drbd9 support multiple connections</div><div>+ : ${_peer_node_id:=0}</div><div>+ DRBD_PER_NAME[$_peer_node_id]=$_conn_name</div><div>+ DRBD_PER_ID[$_peer_node_id]=$_peer_node_id</div><div>+ DRBD_PER_CSTATE[$_peer_node_id]=$_cstate</div><div>+ DRBD_PER_ROLE_REMOTE[$_peer_node_id]=${_peer:-Unknown}</div><div>+ DRBD_PER_DSTATE_REMOTE[$_peer_node_id]=${_pdsk:-DUnknown}</div><div>+</div><div>+ : == DEBUG == _peer_node_id == ${_peer_node_id} ==</div><div>+ : == DEBUG == DRBD_PER_NAME[_peer_node_id] == ${DRBD_PER_NAME[${_peer_node_id}]} ==</div><div>+ : == DEBUG == DRBD_PER_ID[_peer_node_id] == ${DRBD_PER_ID[${_peer_node_id}]} ==</div><div>+ : == DEBUG == DRBD_PER_CSTATE[_peer_node_id] == ${DRBD_PER_CSTATE[${_peer_node_id}]} ==</div><div>+ : == DEBUG == DRBD_PER_ROLE_REMOTE[_peer_node_id] == ${DRBD_PER_ROLE_REMOTE[${_peer_node_id}]} ==</div><div>+ : == DEBUG == DRBD_PER_DSTATE_REMOTE[_peer_node_id] == ${DRBD_PER_DSTATE_REMOTE[${_peer_node_id}]} ==</div><div>+}</div><div>+</div><div> _sh_status_process() {</div><div> # _volume not present should not happen,</div><div> # but may help make this agent work even if it talks to drbd 8.3.</div><div>@@ -335,11 +352,36 @@ _sh_status_process() {</div><div> # not-yet-created volumes are reported as -1</div><div> (( _volume >= 0 )) || _volume=$[1 << 16]</div><div> DRBD_ROLE_LOCAL[$_volume]=${_role:-Unconfigured}</div><div>- DRBD_ROLE_REMOTE[$_volume]=${_peer:-Unknown}</div><div>- DRBD_CSTATE[$_volume]=$_cstate</div><div> DRBD_DSTATE_LOCAL[$_volume]=${_disk:-Unconfigured}</div><div>- DRBD_DSTATE_REMOTE[$_volume]=${_pdsk:-DUnknown}</div><div>+</div><div>+ if $DRBD_VERSION_9 ; then</div><div>+ #Get from _peer_node_process</div><div>+ DRBD_NAME[$_volume]=${DRBD_PER_NAME[@]}</div><div>+ DRBD_ID[$_volume]=${DRBD_PER_ID[@]}</div><div>+ DRBD_VOLUME[$_volume]=${_volume}</div><div>+ DRBD_CSTATE[$_volume]=${DRBD_PER_CSTATE[@]}</div><div>+ DRBD_ROLE_REMOTE[$_volume]=${DRBD_PER_ROLE_REMOTE[@]}</div><div>+ DRBD_DSTATE_REMOTE[$_volume]=${DRBD_PER_DSTATE_REMOTE[@]}</div><div>+</div><div>+ DRBD_PER_NAME=()</div><div>+ DRBD_PER_ID=()</div><div>+ DRBD_PER_CSTATE=()</div><div>+ DRBD_PER_ROLE_REMOTE=()</div><div>+ DRBD_PER_DSTATE_REMOTE=()</div><div>+</div><div>+ : == DEBUG == _volume == ${_volume} ==</div><div>+ : == DEBUG == DRBD_ROLE_LOCAL == ${DRBD_ROLE_LOCAL[${_volume}]} ==</div><div>+ : == DEBUG == DRBD_DSTATE_LOCAL == ${DRBD_DSTATE_LOCAL[${_volume}]} ==</div><div>+ : == DEBUG == DRBD_CSTATE == ${DRBD_CSTATE[${_volume}]} ==</div><div>+ : == DEBUG == DRBD_ROLE_REMOTE == ${DRBD_ROLE_REMOTE[${_volume}]} ==</div><div>+ : == DEBUG == DRBD_DSTATE_REMOTE == ${DRBD_DSTATE_REMOTE[${_volume}]} ==</div><div>+ else</div><div>+ DRBD_CSTATE[$_volume]=$_cstate</div><div>+ DRBD_ROLE_REMOTE[$_volume]=${_peer:-Unknown}</div><div>+ DRBD_DSTATE_REMOTE[$_volume]=${_pdsk:-DUnknown}</div><div>+ fi</div><div>}</div><div>+</div><div> drbd_set_status_variables() {</div><div> # drbdsetup sh-status prints these values to stdout,</div><div> # and then prints _sh_status_process.</div><div>@@ -352,6 +394,15 @@ drbd_set_status_variables() {</div><div> local _resynced_percent</div><div> local out</div><div> </div><div>+ if $DRBD_VERSION_9 ; then</div><div>+ local _peer_node_id _conn_name</div><div>+ DRBD_PER_NAME=()</div><div>+ DRBD_PER_ID=()</div><div>+ DRBD_PER_CSTATE=()</div><div>+ DRBD_PER_ROLE_REMOTE=()</div><div>+ DRBD_PER_DSTATE_REMOTE=()</div><div>+ fi</div><div>+</div><div> DRBD_ROLE_LOCAL=()</div><div> DRBD_ROLE_REMOTE=()</div><div> DRBD_CSTATE=()</div><div>@@ -369,16 +420,20 @@ drbd_set_status_variables() {</div><div> # if there was no output at all, or a weird output</div><div> # make sure the status arrays won't be empty.</div><div> [[ ${#DRBD_ROLE_LOCAL[@]} != 0 ]] || DRBD_ROLE_LOCAL=(Unconfigured)</div><div>- [[ ${#DRBD_ROLE_REMOTE[@]} != 0 ]] || DRBD_ROLE_REMOTE=(Unknown)</div><div>- [[ ${#DRBD_CSTATE[@]} != 0 ]] || DRBD_CSTATE=(Unconfigured)</div><div> [[ ${#DRBD_DSTATE_LOCAL[@]} != 0 ]] || DRBD_DSTATE_LOCAL=(Unconfigured)</div><div>+ [[ ${#DRBD_CSTATE[@]} != 0 ]] || DRBD_CSTATE=(Unconfigured)</div><div>+ [[ ${#DRBD_ROLE_REMOTE[@]} != 0 ]] || DRBD_ROLE_REMOTE=(Unknown)</div><div> [[ ${#DRBD_DSTATE_REMOTE[@]} != 0 ]] || DRBD_DSTATE_REMOTE=(DUnknown)</div><div> </div><div>-</div><div>+ if $DRBD_VERSION_9 ; then</div><div>+ : == DEBUG == DRBD_NAME == ${DRBD_NAME[@]} ==</div><div>+ : == DEBUG == DRBD_ID == ${DRBD_ID[@]} ==</div><div>+ : == DEBUG == DRBD_VOLUME == ${DRBD_VOLUME[@]} ==</div><div>+ fi</div><div> : == DEBUG == DRBD_ROLE_LOCAL == ${DRBD_ROLE_LOCAL[@]} ==</div><div>- : == DEBUG == DRBD_ROLE_REMOTE == ${DRBD_ROLE_REMOTE[@]} ==</div><div>- : == DEBUG == DRBD_CSTATE == ${DRBD_CSTATE[@]} ==</div><div> : == DEBUG == DRBD_DSTATE_LOCAL == ${DRBD_DSTATE_LOCAL[@]} ==</div><div>+ : == DEBUG == DRBD_CSTATE == ${DRBD_CSTATE[@]} ==</div><div>+ : == DEBUG == DRBD_ROLE_REMOTE == ${DRBD_ROLE_REMOTE[@]} ==</div><div> : == DEBUG == DRBD_DSTATE_REMOTE == ${DRBD_DSTATE_REMOTE[@]} ==</div><div> }</div><div> </div><div>@@ -414,6 +469,9 @@ maybe_outdate_self()</div><div> ocf_is_true $OCF_RESKEY_stop_outdates_secondary || return 1</div><div> </div><div> local host stop_uname</div><div>+ if $DRBD_VERSION_9 ; then</div><div>+ local master temp_nmber outdate_self</div><div>+ fi</div><div> # We ignore $OCF_RESKEY_CRM_meta_notify_promote_uname here</div><div> # because: if demote and promote for a _stacked_ resource</div><div> # (or a "floating" one, where DRBD sits on top of some SAN)</div><div>@@ -437,6 +495,29 @@ maybe_outdate_self()</div><div> return 1</div><div> done</div><div> </div><div>+ if $DRBD_VERSION_9 ; then</div><div>+ temp_name=($DRBD_NAME[@])</div><div>+ temp_dstate=($DRBD_DSTATE_REMOTE[@])</div><div>+ temp_number=${#temp_name[@]}</div><div>+ outdate_self=false</div><div>+</div><div>+ for master in $OCF_RESKEY_CRM_meta_notify_master_uname; do</div><div>+ for i in `seq 0 $((temp_number-1))`; do</div><div>+ if [[ ${temp_name[$i]} == "$master" ]] &&</div><div>+ [[ ${temp_dstate[$i]} == "DUnknown" ]]; then</div><div>+ outdate_self=true</div><div>+ break</div><div>+ fi</div><div>+ done</div><div>+ temp_number=${#temp_name[@]}</div><div>+ done</div><div>+</div><div>+ if ! $outdate_self; then</div><div>+ #The disconnecting node is not in Primary</div><div>+ return 1</div><div>+ fi</div><div>+ fi</div><div>+</div><div> # e.g. post/promote of some other peer.</div><div> # Should not happen, fencing constraints should take care of that.</div><div> # But in case it does, scream out loud.</div><div>@@ -993,6 +1074,7 @@ drbd_validate_all () {</div><div> DRBDADM="drbdadm"</div><div> DRBDSETUP="drbdsetup"</div><div> DRBD_HAS_MULTI_VOLUME=false</div><div>+ DRBD_VERSION_9=false</div><div> </div><div> # these will _exit_ if they don't find the binaries</div><div> check_binary $DRBDADM</div><div>@@ -1015,18 +1097,23 @@ drbd_validate_all () {</div><div> modinfo -F version drbd |</div><div> sed -ne 's/^\([0-9]\+\)\.\([0-9]\+\)\.\([0-9]\+\).*$/\1 \2 \3/p'))</div><div> fi</div><div>- if (( $DRBD_KERNEL_VERSION_CODE >= 0x080400 )); then</div><div>+ if (( $DRBD_KERNEL_VERSION_CODE >= 0x090000 )); then</div><div>+ DRBD_HAS_MULTI_VOLUME=true</div><div>+ DRBD_VERSION_9=true</div><div>+ ocf_log warn "RA for DRBD version 9 is in experiment, do not using multiple primaries in DRBD9.0"</div><div>+ elif (( $DRBD_KERNEL_VERSION_CODE >= 0x080400 )); then</div><div> DRBD_HAS_MULTI_VOLUME=true</div><div>- elif (( $DRBD_KERNEL_VERSION_CODE >= 0x090000 )) ; then</div><div>- ocf_log err "This resource agent does (still) only support DRBD version 8.x"</div><div>- exit $OCF_ERR_INSTALLED</div><div> fi</div><div> check_crm_feature_set</div><div> </div><div> # Check clone and M/S options.</div><div>- meta_expect clone-max -le 2</div><div>+ # Drbd9 support more than two nodes</div><div>+ if ! $DRBD_VERSION_9 ; then</div><div>+ meta_expect clone-max -le 2</div><div>+ fi</div><div> meta_expect clone-node-max = 1</div><div> meta_expect master-node-max = 1</div><div>+ # With current DRBD-9.0 version more than two primaries at the same time is not support.</div><div> meta_expect master-max -le 2</div><div> </div><div> # Rather than returning $OCF_ERR_CONFIGURED, we sometimes return</div><div>@@ -1080,7 +1167,8 @@ drbd_validate_all () {</div><div> # DRBD_DEVICES will be a shell array!</div><div> # FIXME we should double check that we explicitly restrict the set of</div><div> # valid characters in device names...</div><div>- if DRBD_DEVICES=($($DRBDADM --stacked sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div>+ # In DRBD9, no matter stacked or not "$DRBDADM --stacked sh-dev $DRBD_RESOURCE" will return true</div><div>+ if ! $DRBD_VERSION_9 && DRBD_DEVICES=($($DRBDADM --stacked sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div> # apparently a "stacked" resource. Remember for future DRBDADM calls.</div><div> DRBDADM="$DRBDADM -S"</div><div> elif DRBD_DEVICES=($($DRBDADM sh-dev $DRBD_RESOURCE 2>/dev/null)); then</div><div>diff --git a/user/v9/drbdsetup.c b/user/v9/drbdsetup.c</div><div>index 053b9d3..fba72e1 100644</div><div>--- a/user/v9/drbdsetup.c</div><div>+++ b/user/v9/drbdsetup.c</div><div>@@ -251,6 +251,7 @@ static int del_resource_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int show_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int status_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int role_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div>+static int sh_status_9compat_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int cstate_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int dstate_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div> static int check_resize_cmd(struct drbd_cmd *cm, int argc, char **argv);</div><div>@@ -478,6 +479,9 @@ struct drbd_cmd commands[] = {</div><div> {"role", CTX_RESOURCE, 0, NO_PAYLOAD, role_cmd,</div><div> .lockless = true,</div><div> .summary = "Show the current role of a resource." },</div><div>+ {"sh-status", CTX_RESOURCE | CTX_ALL, 0, 0, sh_status_9compat_cmd,</div><div>+ .lockless = true,</div><div>+ .summary = "Show all status of resource." },</div><div> {"cstate", CTX_PEER_NODE, 0, NO_PAYLOAD, cstate_cmd,</div><div> .lockless = true,</div><div> .summary = "Show the current state of a connection." },</div><div>@@ -2576,6 +2580,87 @@ static int role_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div> return 0;</div><div> }</div><div> </div><div>+</div><div>+static int sh_status_9compat_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div>+{</div><div>+</div><div>+ struct resources_list *resources_list, *resource;</div><div>+ char *old_objname = objname;</div><div>+</div><div>+ resources_list = sort_resources(list_resources());</div><div>+</div><div>+ for (resource = resources_list; resource; resource = resource->next) {</div><div>+ struct devices_list *devices, *device;</div><div>+ struct connections_list *connections, *connection;</div><div>+ struct peer_devices_list *peer_devices = NULL, *peer_device;</div><div>+ struct nlattr *nla;</div><div>+</div><div>+ if (strcmp(old_objname, "all") && strcmp(old_objname, resource->name))</div><div>+ continue;</div><div>+</div><div>+ objname = resource->name;</div><div>+ printI("_res_name=%s\n", objname);</div><div>+</div><div>+ nla = nla_find_nested(resource->res_opts, __nla_type(T_node_id));</div><div>+ if (nla)</div><div>+ printI("_node_id=%d\n\n", *(uint32_t *)nla_data(nla));</div><div>+ else</div><div>+ printI("_node_id=\n\n");</div><div>+</div><div>+ devices = list_devices(resource->name);</div><div>+ connections = sort_connections(list_connections(resource->name));</div><div>+ if (devices && connections)</div><div>+ peer_devices = list_peer_devices(resource->name);</div><div>+</div><div>+ link_peer_devices_to_devices(peer_devices, devices);</div><div>+</div><div>+ for (device = devices; device; device = device->next) {</div><div>+ ++indent;</div><div>+ printI("_minor=%d\n", device->minor);</div><div>+ printI("_volume=%d\n", device->ctx.ctx_volume);</div><div>+ //refer to v84</div><div>+ //printI("_known=%s\n", xxx);</div><div>+ printI("_role=%s\n", drbd_role_str(resource->info.res_role));</div><div>+ printI("_disk=%s\n\n", drbd_disk_str(device->info.dev_disk_state));</div><div>+</div><div>+ for (connection = connections; connection; connection = connection->next) {</div><div>+ ++indent;</div><div>+ for (peer_device = peer_devices; peer_device; peer_device = peer_device->next) {</div><div>+ if (connection->ctx.ctx_peer_node_id != peer_device->ctx.ctx_peer_node_id</div><div>+ || device->ctx.ctx_volume != peer_device->ctx.ctx_volume)</div><div>+ continue;</div><div>+ printI("_conn_name=%s\n", connection->ctx.ctx_conn_name);</div><div>+ printI("_peer_node_id=%d\n", connection->ctx.ctx_peer_node_id);</div><div>+ printI("_cstate=%s\n", drbd_conn_str(connection->info.conn_connection_state));</div><div>+ if (connection->info.conn_connection_state == C_CONNECTED) {</div><div>+ printI("_peer=%s\n", drbd_role_str(connection->info.conn_role));</div><div>+ printI("_pdsk=%s\n\n", drbd_disk_str(peer_device->info.peer_disk_state));</div><div>+ } else {</div><div>+ printI("_peer=\n");</div><div>+ printI("_pdsk=\n");</div><div>+ }</div><div>+ wrap_printf(0, "_peer_node_process\n\n");</div><div>+ }</div><div>+ //Dummy</div><div>+ //printI("_flags_susp==%s\n", xxx);</div><div>+ //...</div><div>+ --indent;</div><div>+ }</div><div>+</div><div>+ wrap_printf(0, "_sh_status_process\n\n");</div><div>+ --indent;</div><div>+ }</div><div>+</div><div>+ free_connections(connections);</div><div>+ free_devices(devices);</div><div>+ free_peer_devices(peer_devices);</div><div>+ }</div><div>+</div><div>+ free(resources_list);</div><div>+ objname = old_objname;</div><div>+ return 0;</div><div>+}</div><div>+</div><div> static int cstate_cmd(struct drbd_cmd *cm, int argc, char **argv)</div><div> {</div><div> struct connections_list *connections, *connection;</div><div><div class="h5"><div><br></div><div><br></div><div><br></div><div><br></div><div><blockquote type="cite"><div>30 maj 2016 kl. 13:36 skrev Igor Cicimov <<a href="mailto:icicimov@gmail.com" target="_blank">icicimov@gmail.com</a>>:</div><br><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 17, 2016 at 8:21 AM, Mats Ramnefors <span dir="ltr"><<a href="mailto:mats@ramnefors.com" target="_blank">mats@ramnefors.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am testing a DRBD 9 and 8.4 in simple 2 node active - passive clusters with NFS.<br>
<br>
Copying files form a third server to the NFS share using dd, I typically see an average of 20% CPU load (with v9) on the primary during transfer of larger files, testing with 0,5 and 2 GB.<br>
<br>
At the very end of the transfer DRBD process briefly peaks at 70 - 100% CPU.<br>
<br>
This causes occasional problems with Corosync believing the node is down. Increasing the token time for Corosync to 2000 ms fixes the symptom but I am wondering about the root cause and any possible fixes?<br>
<br>
This is the DRBD configuration.<br>
<br>
resource san_data {<br>
protocol C;<br>
meta-disk internal;<br>
device /dev/drbd1;<br>
disk /dev/nfs/share;<br>
net {<br>
verify-alg sha1;<br>
cram-hmac-alg sha1;<br>
shared-secret ”****************";<br>
after-sb-0pri discard-zero-changes;<br>
after-sb-1pri discard-secondary;<br>
after-sb-2pri disconnect;<br>
}<br>
on san1 {<br>
address <a href="http://192.168.1.86:7789/" rel="noreferrer" target="_blank">192.168.1.86:7789</a>;<br>
}<br>
on san2 {<br>
address <a href="http://192.168.1.87:7789/" rel="noreferrer" target="_blank">192.168.1.87:7789</a>;<br>
}<br>
}<br>
<br>
The nodes are two VM on different ESXi hosts (Dell T620). Hosts are very lightly loaded. Network is 1 Gb at the moment through a Catalyst switch. Network appears not saturated.<br>
<br>
BTW when can we expect a DRBD resource agent for v9? It took me a while to figure out why DRBD9 was not working with Pacemaker and then finding a patch to the agent :)<br>
<br>
Cheers Mats<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a></blockquote><div><br></div><div>Hi Mats,<br><br></div><div>Can you please share the patch if you don't mind?<br><br></div><div>Thanks,<br></div><div>Igor <br></div></div><br></div></div>
</div></blockquote></div><br></div></div></div></div></div></blockquote></div><br></div></div></div>