<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><br><div>Hi Joel,</div><div><br> </div><div>In testing, we found that the problem is consistent with the one that commit "7a9ae1a208" solves.</div><div>commit "7a9ae1a208" comment in w_e_end_rsdata_req() function:<br></div><div> /* DRBD versions without DRBD_FF_RESYNC_DAGTAG lock<br> * 128MiB "resync extents" in the activity log whenever<br> * they make resync requests. Some of these versions<br> * also lock activity lock extents when receiving<br> * P_DATA. In particular, DRBD 9.0 and 9.1. This can<br> * cause a deadlock if we send resync replies in these<br> * extents as follows:<br> * * Node is SyncTarget towards us<br> * * Node locks a resync extent and sends P_RS_DATA_REQUEST<br> * * Node receives P_DATA write in this extent; write<br> * waits for resync extent to be unlocked<br> * * Node receives P_BARRIER (protocol A); receiver<br> * thread blocks waiting for write to complete<br> * * We reply to P_RS_DATA_REQUEST, but it is never<br> * processed because receiver thread is blocked<br> *<br> * Break the deadlock by canceling instead. This is<br> * sent on the control socket so it will be processed. */</div><div><br></div><div>Then we found two suspicious pieces of code (patch content below):</div><div>1. In the w_e_end_rsdata_req() function, there is no lock protection between al_resync_extent_active() and drbd_rs_reply(), and P_DATA may be sent through this gap.</div><div>@@ -180,9 +184,11 @@ struct lc_element *_al_get_nonblock(struct drbd_device *device, unsigned int enr<br> {<br> struct lc_element *al_ext;<br> <br>+ mutex_lock(&device->resync_lock);<br> spin_lock_irq(&device->al_lock);<br> al_ext = is_local ? lc_try_get_local(device->act_log, enr) : lc_try_get(device->act_log, enr);<br> spin_unlock_irq(&device->al_lock);<br>+ mutex_unlock(&device->resync_lock);<br> <br> return al_ext;<br> }<br>@@ -192,9 +198,11 @@ struct lc_element *_al_get(struct drbd_device *device, unsigned int enr, bool is<br> {<br> struct lc_element *al_ext;<br> <br>+ mutex_lock(&device->resync_lock);<br> spin_lock_irq(&device->al_lock);<br> al_ext = is_local ? lc_get_local(device->act_log, enr) : lc_get(device->act_log, enr);<br> spin_unlock_irq(&device->al_lock);<br>+ mutex_unlock(&device->resync_lock);<br> <br> return al_ext;<br> }<br>diff --git a/drbd/drbd_int.h b/drbd/drbd_int.h<br>index e9d2c3914..95cf2bb48 100644<br>--- a/drbd/drbd_int.h<br>+++ b/drbd/drbd_int.h<br>@@ -1588,6 +1588,7 @@ struct drbd_device {<br> <br> int next_barrier_nr;<br> struct drbd_md_io md_io;<br>+ struct mutex resync_lock;<br> spinlock_t al_lock;<br> wait_queue_head_t al_wait;<br> struct lru_cache *act_log; /* activity log */<br>diff --git a/drbd/drbd_main.c b/drbd/drbd_main.c<br>index 02f4ae5d1..f625e2e83 100644<br>--- a/drbd/drbd_main.c<br>+++ b/drbd/drbd_main.c<br>@@ -4092,6 +4092,7 @@ enum drbd_ret_code drbd_create_device(struct drbd_config_context *adm_ctx, unsig<br> #ifdef CONFIG_DRBD_TIMING_STATS<br> spin_lock_init(&device->timing_lock);<br> #endif<br>+ mutex_init(&device->resync_lock);<br> spin_lock_init(&device->al_lock);<br> <br> spin_lock_init(&device->pending_completion_lock);<br>diff --git a/drbd/drbd_req.c b/drbd/drbd_req.c<br>index c4aa23a31..f05e21dd3 100644<br>--- a/drbd/drbd_req.c<br>+++ b/drbd/drbd_req.c<br>@@ -2383,6 +2383,7 @@ static bool prepare_al_transaction_nonblock(struct drbd_device *device,<br> bool made_progress = false;<br> int err;<br> <br>+ mutex_lock(&device->resync_lock);<br> spin_lock_irq(&device->al_lock);<br> <br> /* Don't even try, if someone has it locked right now. */<br>@@ -2418,6 +2419,7 @@ static bool prepare_al_transaction_nonblock(struct drbd_device *device,<br> }<br> out:<br> spin_unlock_irq(&device->al_lock);<br>+ mutex_unlock(&device->resync_lock);<br> return made_progress;<br> }<br> <br>diff --git a/drbd/drbd_sender.c b/drbd/drbd_sender.c<br>index 738be16d5..ddea6230a 100644<br>--- a/drbd/drbd_sender.c<br>+++ b/drbd/drbd_sender.c<br>@@ -2106,6 +2106,7 @@ int w_e_end_rsdata_req(struct drbd_work *w, int cancel)<br> if (peer_device->repl_state[NOW] == L_AHEAD) {<br> err = drbd_send_ack(peer_device, P_RS_CANCEL, peer_req);<br> } else if (likely((peer_req->flags & EE_WAS_ERROR) == 0)) {<br>+ mutex_lock(&peer_device->device->resync_lock);<br> if (unlikely(peer_device->disk_state[NOW] < D_INCONSISTENT)) {<br> if (connection->agreed_features & DRBD_FF_RESYNC_DAGTAG) {<br> drbd_err_ratelimit(peer_device,<br>@@ -2154,6 +2155,7 @@ int w_e_end_rsdata_req(struct drbd_work *w, int cancel)<br> if (expect_ack)<br> peer_req = NULL;<br> }<br>+ mutex_unlock(&peer_device->device->resync_lock);<br> } else {<br> drbd_err_ratelimit(peer_device, "Sending NegRSDReply. sector %llus.\n",<br> (unsigned long long)peer_req->i.sector);<br></div><div><br></div><div><br></div><div>2.The al_resync_extent_active() check does not include changing extent.<br>--- a/drbd/drbd_actlog.c<br>+++ b/drbd/drbd_actlog.c<br>@@ -163,12 +163,16 @@ bool drbd_al_active(struct drbd_device *device, sector_t sector, unsigned int si<br> <br> spin_lock_irq(&device->al_lock);<br> for (enr = first; enr <= last; enr++) {<br>- struct lc_element *al_ext;<br>- al_ext = lc_find(device->act_log, enr);<br>- if (al_ext && al_ext->refcnt > 0) {<br>+ if (lc_is_used(device->act_log, enr)) {<br> active = true;<br> break;<br> }<br>+ // struct lc_element *al_ext;<br>+ // al_ext = lc_find(device->act_log, enr);<br>+ // if (al_ext && al_ext->refcnt > 0) {<br>+ // active = true;<br>+ // break;<br>+ // }<br> }<br> spin_unlock_irq(&device->al_lock);</div><div><br></div><div><br></div><div>When we fixed these two pieces of code, the problem continued to recur.</div><div>Now the reason is that SyncSource send thead is also blocked, so it will not reply to P_RS_CANCEL ack.</div><div><br></div><div>SyncSource send thead:</div><div>[<0>] wait_woken+0x2c/0x60<br>[<0>] sk_stream_wait_memory+0x2bb/0x340<br>[<0>] do_tcp_sendpages+0x258/0x340<br>[<0>] tcp_sendpage_locked+0x44/0x60<br>[<0>] tcp_sendpage+0x37/0x50<br>[<0>] inet_sendpage+0x52/0x90<br>[<0>] dtt_send_page+0x93/0x140 [drbd_transport_tcp]<br>[<0>] flush_send_buffer+0xd0/0x150 [drbd]<br>[<0>] __send_command+0xf8/0x160 [drbd]<br>[<0>] drbd_send_block+0xaa/0x230 [drbd]<br>[<0>] drbd_rs_reply+0x26e/0x300 [drbd]<br>[<0>] w_e_end_rsdata_req+0xd6/0x4b0 [drbd]<br>[<0>] drbd_sender+0x13a/0x3d0 [drbd]<br>[<0>] drbd_thread_setup+0x69/0x190 [drbd]<br>[<0>] kthread+0x10a/0x120<br>[<0>] ret_from_fork+0x1f/0x40<br></div><div><br></div><div>SyncTarget receiver thread:</div><div>[<0>] conn_wait_active_ee_empty_or_disconnect+0x7d/0xb0 [drbd]<br>[<0>] receive_Barrier+0x16b/0x1f0 [drbd]<br>[<0>] drbd_receiver+0x5af/0x7f0 [drbd]<br>[<0>] drbd_thread_setup+0x5c/0x160 [drbd]<br>[<0>] kthread+0x10a/0x120<br>[<0>] ret_from_fork+0x1f/0x40<br><br></div><div>Do you have any good solutions?<br></div><pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Best regards,
zhengbing</pre><div style="position:relative;zoom:1"></div><br>From: Zhengbing <zhengbing.huang@easystack.cn><br>Date: 2024-10-16 20:03:27<br>To: drbd-dev@lists.linbit.com<br>Subject: drbd: resync blocks<br><blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Hi Joel,</div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">I have a problem with resync blocks. <br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">First, I have a 2 node cluster, and node-1 running DRBD 9.1 and node -2 is DRBD 9.2 and protocol C.<br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">and the problem scenario is as follows:</div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">1. node-2 always has application IO<br>2. node-1 network failure with node-2<br>3. the network is restored. node-1 is SyncTarget and node-2 is SyncSource<br>4. then resync process is blocks<br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">You solved the same problem in Commit "7a9ae1a208", but I still have this problem.</div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">So, how do I solve this problem?<br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><br></div><div style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Best regards,
zhengbing<br></pre></div><p></p></div><br></blockquote></div><br>