[DRBD-user] Write performance problem after resync done.

[Shuhei Tomita]

Hello,

I found write performance problem after resync done.
Both node are Xeon 2.8GHz, 2 GB RAM, Intel SCSI RAID w/73GB HDD(RAID1),
connected directly via Gigabit Ethernet (Intel PRO/1000 MTU 9000),
kernel 2.4.29 (from source, SMP and HIGHMEM enabled), 
drbd-0.7.7(drbd partion size is about 50GB),filesystem ext3 (and xfs).

node1: primary
node2: secondary

To reproduce,

1. export drbd partition via nfs with sync,no_wdelay option
2. start resync
   execute at node1 like

   /sbin/drbdsetup /dev/drbd0 invalidate_remote

3. start test script from nfs client
4. after resync done, stop the test script from nfs client
   (now there is no writing to drbd partition)

5. test below commands on drbd primary server to drbd partition not via nfs.

   time dd if=/dev/zero of=bigfile count=100000 ; time sync 

   It took 12 seconds or more.
   usually it could be done within 1 second.

I think it is not nfs problem, because 

a) when I disconnect secondary, the problem goes away,
   but I reconnect secondary, the problem comes back again.

b) it does not occur while using drbd-0.7.5,
   both 0.7.7 and 0.7.10 have this problem.

c) via nfs, I could reproduce 100%, not via nfs this problem
   appears sometimes.

To solve the problem, I had to do 
1. make both node secondary
2. drbd stop/start on node1
3. make node1 primary

Here is my config:

[drbd.conf]

resource drbd0 {
  protocol C;

  startup {
        degr-wfc-timeout 120;
   }
  disk {
        on-io-error detach;
  }
  net {
        timeout         60;
        connect-int     10;
        ping-int        10;
        max-buffers     2048;
        max-epoch-size  2048;
        ko-count        4;
        sndbuf-size     4M;
  }
  syncer {
        rate 50M;
        group 1;
        al-extents 257;
  }
 on node1 {
    device      /dev/drbd0;
    disk        /dev/sda6; 
    address     192.168.0.1:7788;
    meta-disk   internal;
  }
  on node2 {
    device      /dev/drbd0;
    disk        /dev/sda6;
    address     192.168.0.2:7788;
    meta-disk   internal;
  }
}

[test script on nfs clinet]

#!/usr/bin/perl

$i = 0;

while(1){
    $TARGETFILE = "${i}.txt";
    open(FILE, ">>$TARGETFILE");
    for($j = 0; $j < 5000; $j++){
        print FILE "abcdefghijklmnopqrstuvwxyz\n";
    }
    close(FILE);
    $i++;
}

If possible, I want to use the newest version of drbd.
(drbd-0.7.5 has a bug that syncer starves)

Any suggestions appreciated. Thanks,

Shuhei Tomita