[Drbd-dev] ordering problem between StateChangeRequest and send_state

[Graham, Simon]

I have seen a problem where if resync is paused and resumed and then completed quickly, the sync target node correctly transitions to UptoDate but the other ends up Inconsistent. I believe this is a reordering issue caused by an interaction between drbd_send_state_req and drbd_send_state as follows (with the resync starting off in the paused state):

1. We resumed syncing and set aftr_isp->0 -- this schedules a work item to report the change
   to the peer.

2. Syncing completes on an other thread, drbd_send_state_req is called inline to send the
   new state to the peer with a StateChgRequest, BUT the mdev state is not updated yet. 
   This code then waits for a response from the peer... when the peer gets this, it updates
   it's state immediately (to Consistent) and sends the response.

3. Now the after state change starts running and does a send-state call which grabs
   the current state from the device (which is still inconsistent) and sends this. When
   the peer gets this, it will change from UpToDate->Inconsistent.

4. Finally, we get the response to the StateChgRequest and change the state to UpToDate on
   the sync target - no further state change is sent because transition to UpToDate doesn't
   generate one.

My suggested fix is to use the existing drbd_state_lock - this is already used in drbd_send_state_req, my suggestion is to also grab it in drbd_send_state so that we don't attempt to send ReportState if a state change rq is already in progress.

I've attached a proposed patch that implements this suggestion BUT this is based on an earlier patch of ours that splits drbd_send_state into two routines and acquires the send mutex - the one titled "Ensure that state updated cannot be done out of order whilst running receive_state (914435c)".

Simon
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Make-sure-we-dont-send-state-if-a-cluster-wide-state.patch
Type: application/octet-stream
Size: 832 bytes
Desc: 0001-Make-sure-we-dont-send-state-if-a-cluster-wide-state.patch
Url : http://lists.linbit.com/pipermail/drbd-dev/attachments/20080201/e518fe23/0001-Make-sure-we-dont-send-state-if-a-cluster-wide-state.obj