[DRBD-user] DRBD fsync() seems to return before writing to disk

[Phil Frost]

On 06/20/2012 10:59 AM, Phil Frost wrote:
> On 06/20/2012 10:43 AM, Shaun Thomas wrote:
>> "Unfortunately device mapper (LVM) might not support barriers."
>
> Further, the fsync() latency I observed running ext4 on MD and LVM 
> (but not DRBD devices) empirically suggest that barriers are working 
> for DRBD's underlying block device, at least given my understanding 
> that working barriers are required for a fsync() that really does 
> flush to non-volatile storage.


Just to check my sanity further, I just did a test of a DRBD device 
directly on a partition of a SATA drive. No LVM, no MD; just a plain 
SATA drive and DRBD. I reached the same conclusion, DRBD is not 
observing fsync(), O_SYNC, etc., and it's not for lack of support on the 
underlying device. Can anyone reproduce?





pfrost at storage02:/mnt/synctest$ cat /etc/drbd.d/test.res
resource test {
     device minor 1;
     net {
         shared-secret 
BTawM41lfw8L0RTKBXhOiGK4lWb6dZTqJGNaGJJwF0pCfNhasdfhB5qjBjgZL4O;
     }

     on storage01 {
         disk /dev/sda2;
         meta-disk internal;
         address 10.0.0.7:7790;
     }
     on storage02 {
         disk /dev/sda2;
         meta-disk internal;
         address 10.0.0.9:7790;
     }
}
pfrost at storage02:~$ sudo drbdadm -- --overwrite-data-of-peer primary test
pfrost at storage02:~$ sudo mkfs -t ext4 /dev/drbd/by-res/test
pfrost at storage02:~$ sudo mount /dev/drbd/by-res/test -o barrier=1 -t 
ext4 /mnt/synctest/
pfrost at storage02:~$ cd /mnt/synctest/
pfrost at storage02:/mnt/synctest$ sudo ~/test_fsync -f ./test_fsync
Simple write timing:
         write                    0.002742

Compare fsync times on write() and non-write() descriptor:
If the times are similar, fsync() can sync data written
on a different descriptor.
         write, fsync, close      0.196210
         write, close, fsync      0.175143

Compare one o_sync write to two:
         one 16k o_sync write     0.175284
         two 8k o_sync writes     0.300499

Compare file sync methods with one 8k write:
         (o_dsync unavailable)
         open o_sync, write       0.157953
         write, fdatasync         0.167023
         write, fsync             0.169357

Compare file sync methods with two 8k writes:
         (o_dsync unavailable)
         open o_sync, write       0.300676
         write, fdatasync         0.182892
         write, fsync             0.184106

pfrost at storage02:/mnt/synctest$ cd ..
pfrost at storage02:/mnt$ sudo umount synctest/
pfrost at storage02:/mnt$ sudo drbdadm down test
pfrost at storage02:/mnt$ sudo mkfs -t ext4 /dev/sda2
pfrost at storage02:/mnt$ cd /mnt/synctest/
pfrost at storage02:/mnt/synctest$ sudo ~/test_fsync -f ./test_fsync
Simple write timing:
         write                    0.002765

Compare fsync times on write() and non-write() descriptor:
If the times are similar, fsync() can sync data written
on a different descriptor.
         write, fsync, close      8.561847
         write, close, fsync      8.490528

Compare one o_sync write to two:
         one 16k o_sync write     8.365526
         two 8k o_sync writes    20.089010

Compare file sync methods with one 8k write:
         (o_dsync unavailable)
         open o_sync, write       8.390429
         write, fdatasync         8.407142
         write, fsync             8.540648

Compare file sync methods with two 8k writes:
         (o_dsync unavailable)
         open o_sync, write      16.739327
         write, fdatasync         8.423901
         write, fsync             8.515552