[DRBD-user] DRBD with 3 nodes, one for DR; FSCK is bad for DR node

[Schaefer, Diane E]

Hello,
  I use DRBD 8.4.4 on SLES 11 SP3 and I'm having difficulties setting up a 3 node DR configuration.  I have a 2-node pacemaker cluster with another stacked resource to the DR site.  My DR node is not part of the cluster.  My DRBD config is as follows:
resource pg {
  on usrv-ffm12 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 172.29.xx.xx:7788;
  meta-disk internal;
}
on usrv-ffm11 {
  device /dev/drbd0;
  disk /dev/sda3;
  address 172.29.xx.xx:7788;
  meta-disk internal;
}
}
resource pg-dr {
  net {
   protocol A;  # async replication
  }

  stacked-on-top-of pg {
   device /dev/drbd10;
   address 192.61.xx.xx:7788;  # floating IP of the primary cluster
  }
# this is the dr site
  on usrv-ffm2 {
   device /dev/drbd10;
   disk /dev/sda3;
   address 192.61.xx.xx:7788; # ip of dr site
   meta-disk internal;
  }
}

I bring up all three nodes and both the stacked and regular resource all get to UpToDate.  When I demote the 2-node cluster master and make the DR site primary the fsck of the file system shows data is bad.  The fsck on the 2 node cluster shows the data is clean.

usrv-ffm2:~ # fsck -n /dev/drbd10
fsck from util-linux 2.19.1
e2fsck 1.41.9 (22-Aug-2009)
fsck.ext3: Group descriptors look bad... trying backup blocks...
fsck.ext3: Bad magic number in super-block when using the backup blocks
fsck.ext3: going back to original superblock
fsck.ext3: Device or resource busy while trying to open /dev/drbd10
Filesystem mounted or opened exclusively by another program?

If I let fsck run, the journal inode 8 is bad.

My setup

1)  Create the meta base resource pg on the first 2 nodes

2)  Connect these two nodes with drbdadm up pg

3)  Clear the bitmap so data is UpToDate for first 2 nodes:

drbdadm -- --clear-bitmap new-current-uuid pg

4)  Make one primary - (called master here)

5)  Calculate the size of the meta needed for the base resource and stacked resource.  Create an ext3 filesystem with the size of disk partition - size of the 2 meta areas.

mkfs.ext3 -b 4096 /dev/drbd0 $size

6)  Make sure the .res files are on the DR system

7)  Create the meta on the master system for pg-dr (stacked res)

8)  drbdadm -S up pg-dr on the master system

9)  On DR site create the meta for pg-dr

10)         On DR site up the pg-dr resource so it can connect.  It's now connected, both are secondary

11)         On the master site, clear the bitmap for pg-dr to get data in UpToDate

12)         On master site make it primary for pg-dr.

13)         Mount the file system to /dev/drbd10 (stacked device) on the Master node

I use the clear-bitmap new-current-uuid command since I start with no data anywhere.  This allows me to get to UpToDate without transporting data.

I'm a bit confused over the creation of the drbd meta area.  Is it done on all 3 nodes for both the base and the stacked resource?  Does it matter if it's done before or after I create the file system (I don't think it matters since I believe the meta is written at the end of the device).

Thanks for any help
Diane Schaefer

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20140918/afeca8bf/attachment.htm>