[DRBD-user] "Concurrent local write detected!"

[Chris Worley]

I really think drbd is being brain-dead here.  Concurrent writes to
the same LBA aren't an issue... just do it!  Note the below is using a
primary/secondary setup on two raw drbd devices; no GFS anywhere.

Let me use an example of two fio invocations as an example, sorry if
you don't know fio.

The first is an example of what I'd normally use, when telling it I
want to run two threads per drive.

fio  --rw=write --bs=1m --rwmixread=0 --iodepth=64
--output=/tmp/fio.out.txt --group_reporting --sync=0 --direct=1
--randrepeat=0 --softrandommap=1 --ioengine=libaio --loops=10000
   --name=test0 --filename=/dev/drbd0 --numjobs=2 --size=16093304832
   --name=tet1 --filename=/dev/drbd1 --numjobs=2 --size=16093304832

In the above case, nearly immediately, the systems starts spewing
"Concurrent local write detected", and as block sizes decrease the
machine-check monitor will eventually do a soft lockup, and the
thumb/boot drive will all of sudden think it's disconnected then
reconnect as a different SD device (leaving the system dead).

If I change the above to assure no two threads write to the same offsets, as in:

fio --rw=write --rwmixread=0 --bs=1m --runtime=600 --iodepth=64
--output=/tmp/fio.out.txt --group_reporting --sync=0 --direct=1
--randrepeat=0 --softrandommap=1 --ioengine=libaio --loops=10000 \
  --name=test0-0 --filename=/dev/drbd0 --offset=0 --numjobs=1
--size=8046652416 \
  --name=test0-1 --filename=/dev/drbd0 --offset=8046652416 --numjobs=1
--size=8046652416 \
  --name=test1-0 --filename=/dev/drbd1 --offset=0 --numjobs=1
--size=8046652416 \
  --name=test1-1 --filename=/dev/drbd1 --offset=8046652416 --numjobs=1
--size=8046652416

... then I see no problems.

Unix semantics has you covered.  If your told to write the same LBA
twice, just write the thing, and don't kill the system.

Thanks,

Chris
On Tue, Dec 28, 2010 at 1:06 PM, Chris Worley <worleys at gmail.com> wrote:
> On Thu, Dec 23, 2010 at 10:48 AM, J. Ryan Earl <oss at jryanearl.us> wrote:
>> On Mon, Dec 20, 2010 at 2:06 PM, Chris Worley <worleys at gmail.com> wrote:
>>>
>>> I'm using RHEL5.5/2.6.18-194.3.1.el5 and IB/SDP.
>>
>> What version of DRBD are you using and what versions have you tried?
>
> 8.3.8-1, using the precompiled binary RPMs.  I've not tried other
> revs.  I have tried other configurations, all of which seem to
> lock-up; in one configuration, the drive devices not associated with
> DRBD get locked-up and the devices go offline.
>
> Thanks,
>
> Chris
>