[Drbd-dev] ordering problem between StateChangeRequest and
Simon.Graham at stratus.com
Sat Feb 2 04:21:59 CET 2008
I have seen a problem where if resync is paused and resumed and then completed quickly, the sync target node correctly transitions to UptoDate but the other ends up Inconsistent. I believe this is a reordering issue caused by an interaction between drbd_send_state_req and drbd_send_state as follows (with the resync starting off in the paused state):
1. We resumed syncing and set aftr_isp->0 -- this schedules a work item to report the change
to the peer.
2. Syncing completes on an other thread, drbd_send_state_req is called inline to send the
new state to the peer with a StateChgRequest, BUT the mdev state is not updated yet.
This code then waits for a response from the peer... when the peer gets this, it updates
it's state immediately (to Consistent) and sends the response.
3. Now the after state change starts running and does a send-state call which grabs
the current state from the device (which is still inconsistent) and sends this. When
the peer gets this, it will change from UpToDate->Inconsistent.
4. Finally, we get the response to the StateChgRequest and change the state to UpToDate on
the sync target - no further state change is sent because transition to UpToDate doesn't
My suggested fix is to use the existing drbd_state_lock - this is already used in drbd_send_state_req, my suggestion is to also grab it in drbd_send_state so that we don't attempt to send ReportState if a state change rq is already in progress.
I've attached a proposed patch that implements this suggestion BUT this is based on an earlier patch of ours that splits drbd_send_state into two routines and acquires the send mutex - the one titled "Ensure that state updated cannot be done out of order whilst running receive_state (914435c)".
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 832 bytes
Url : http://lists.linbit.com/pipermail/drbd-dev/attachments/20080201/e518fe23/0001-Make-sure-we-dont-send-state-if-a-cluster-wide-state.obj
More information about the drbd-dev