<div class="gmail_quote">On Thu, Apr 16, 2009 at 3:31 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div></div><div>On Wed, Apr 15, 2009 at 03:06:39PM -0400, Gennadiy Nerubayev wrote:<br>
> I've been seeing "Concurrent local write" messages under certain workloads<br>
> and environments - in particular this has been observed with a SCST target<br>
> running on top of DRBD, and benchmarks running on both bare windows<br>
> initiators and windows virtuals through ESX initiator. What does this<br>
> warning actually entail, and why would it happen?<br>
<br>
</div></div>if there is a write request A in flight (submitted, but not yet completed)<br>
to offset a, with size x, and while this is still not completed yet<br>
an other write request B is submitted to offset b with size y,<br>
and these requests do overlap,<br>
<br>
that is a "concurrent local write".<br>
<br>
layers below DRBD may reorder writes.<br>
<br>
which means these workloads violate write ordering constraints.<br>
<br>
problem:<br>
as DRBD replicates the requests, these writes might get reordered on the<br>
other node as well. so the may end up on the lower level device in<br>
different order.<br>
<br>
as they do overlap, the resulting data on the both replicas<br>
may end up being different.<br>
<br>
submitting a new write request overlapping with an in flight write<br>
request is bad practice on any IO subsystem, as it may violate write<br>
ordering, and the result in general is undefined.<br>
<br>
with DRBD in particular, it may even cause data divergence of the<br>
replicas, _if_ the layers below DRBD on both nodes decide to reorder<br>
these two requests. The likeliness of which is difficult to guess.<br>
<br>
in short: DRBD detects that the layer using it is broken.<br>
<br>
most likely it is simply the windos io stack that is broken,<br>
as the initiators and targets involved simply forward the requests<br>
issued by the windows file system and block device layer.<br>
<br>
DRBD cannot help you with that. It simply is the only IO stack paranoid<br>
enough to actually _check_ for that condition, and report it.<br>
because DRBD promises to create _exact_, bitwise identical, replicas.<br>
and this condition in general may cause data divergence.</blockquote><div><br>Thanks for the detailed information. I've been able to confirm that it only happens when blockio mode is used, with both SCST and IET, on either ESX (windows or linux virtuals) or Windows as initiator. I've also observed that it only apparently happens during random io workloads, and not sequential. What I'm still trying to understand is the following:<br>
<br>1. Whether the write concurrency (obviously without warnings) would still happen to the bare disk if you remove the DRBD layer, which seems unlikely due to so many usage cases involving blockio targets, both with and without DRBD<br>
2. What's the worst case scenario (lost write? corrupt data? unknown consistency?) that can result from concurrent writes?<br>3. How would one be able to verify that #2 happened?<br>4. I'd think others would report concurrency warnings as well due to the relatively common usage scenario (and google does show a few hits), but people have yet to actually report an actual problem..<br>
<br>Thanks,<br><br>-Gennadiy<br></div></div>