[DRBD-user] drbd solution for distant sites ? (4MB line)

Romain BOTTAN romain.bottan at celsecat.com
Fri Feb 24 18:06:00 CET 2006

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi,
thank you for all your good suggests!

Before answer to the questions I will explain the solution, now, I would 
consider:
2 servers -
4 IP used
- 10.0.0.1 and 10.0.0.2 for the link "heartbeat sync" in replacement of 
the serial one
- 192.168.1.21 and 192.168.5.232 (it's false examples) for the 
interfaces connected with client PC lan.

The "link" between the two nodes is a VPN which use a closed WAN (used 
to connect to the Intranet of the group)
-- just tested the bandwidth about 500ms of ping command  (glups... ping 
between servers=1.5ms) and ssh file transfer test : 10KO/sec loooool far 
from the 512 of the 2048 line... 200 Gigas -> 230 days to sync... The 
two servers need to be shipped here first !

The configuration of the disk and the "shares" would be done like that:
1. The two nodes are "active" in drbd "A" & "B"
2. They are 2 scsi disk in each server, "A1", "A2" and respectively "B1" 
& "B2"
3. When A & B are online and accessible on address 1.21 and 5.232:
--A1 is mounted and have the files of the first site, B1 is mounted and 
have the files of the second site
--A2 is the RAID 1 mirror of  B1 & B2 the RAID 1 mirror of A1
-- users of first site access to data of the second site by making a 
connection to the distant site (and inverse is true also).
4. IF A burst:
-- B, detecting that A is dead mount disk B2 and users of the fist site 
can access both data using the VPN and the DATA will be as synchronized 
as the drbd process can handle.
5. IF B bust, A detect it.... bla bla bla...

do you think it's a possible implementation of drbd + heartbeat ?
I can't think of a better way to solve my problem and make the use as 
transparent as possible.
We do not have a real transparent cluster where users can write data 
without consider the location where they put it but the best way to have 
"up to date" synchronized backup servers, isn't it ?

2 config questions:
*bcast eth2 is the right param to use my eth2 interface instead of 
serial ? (I've tested and have my answer but "on ne sait jammais" :)*
*
use baud XXX is necessary for me ?*

here : http://linux-ha.org/DataRedundancyByDrbd I read:

 >"But you could use DRBD over long distance links, too. When you have 
the replica several hundred kilometers away in some other data >center 
for Disaster Recovery, your data will survive even a major earthquake at 
your primary location. You want to use protocol A and a >huge 
sndbuf-size here, and probably adjust the timeout, too."

Ok, do you know were I can get info on where and how to modify this 
sndbuf-size  param ? (I've read your advice Philipp, don't matter)

*>Think about privacy!* Since with DRBD the complete disk content goes 
over the wire, if this wire is not a crossover cable but the 
 >(supposedly hostile) Internet, you should route DRBD traffic through 
some virtual private network (VPN).

 >Make sure *no one* other than the partner node can access the DRBD 
ports, or someone might provoke a connection loss, and then race >for 
the first reconnect, to get a full sync of your disks content.

GLUPS... ok... I will use my PKI to do so... loooool

 >You could probably perform the first sync locally.
I will try do make such a thing but I need first a little advice:
IF the 2nd server would be ship to the distant site, and the Scsi disk 
send to me after,  do you think that i can "make a copy" of the first 
partition to the second and do as well to not make drbd to synchronize 
the entire disk when they will plug it in the distant server ?

>I would do it like so....
>
>1. Setup DRBD to sync locally and do your initial sync.
>2. Stop the DRBD/heartbeat daemon on the secondary and make sure it is not
>going to restart under an rc script when it boots the next time.
>3. Stop DRBD/heartbeat on the primary and be sure it too is not loading
>DRBD/heatbeaton boot up.
>4. Ship the secondary to your remote location.

I couldn't ship the server but only the disks.


>5. Update the drbd.conf and ha.cf to have the correct new IP information
>(after testing you have a good connection of course) and pay close attention
>to your timeouts as a little latency in a link may cause heartbeat or DRBD
>to be flopping all over the place. 
Where can I get info on this ?


>Be sure you are using protocol C too. (Though, here is no real reason to
>ever using anything but protocol C)
I've tried A and C and put bandwith (in drbd.conf) to 256K but no difference.


>you are aware that drbd is mainly a failover solution,
>not a "use identical data on both sides at the same time" thing.
>i.e. you can use your data here, and have it mirrored by drbd to your
>other site, as a backup for disaster recovery
Ok, as you can see I think I've (now... lol) really understood that :)

>you could even do a (lvm) snapshot on the remote side,
>and mount that one readonly.
I'm don't know how to do really, make a LVM of B1 on A and present it as a read-only share in samba 
and inform my users they have to connect to the B server to write things on the distant server.
I think it's more simple to force them to mount 2 virtual disks, and when one server is dead change the address of broken disk.

>700km would be what? 4ms? (although 4MB is going to add to that)
>(I suppose double that for the round trip)compare that with disk access times.
>now compare the bandwidths.  much bigger difference ?
I  don't really know, I think I need to compare DRBD time to sync block to the ms that a ping gives me.

>I think if you have a sequence of transactions, say locks you have to take
>or whatever then 10ms is going to start to hurt pretty fast, but I would
>have guessed that a cluster file system would avoid that where possible?

All the question is: 
Is the process used by the protocol (A, b or C) will slows down the process of writing in a node ?

I need an answer: if I put a 100MO file on the first server with a 100MB connection, as drbd ACK each block to the client
system, the writing will make 8sec (12.5 MO/sec) or 6 minutes (256KO/sec -- 2048 sdsl)?

>(mind you I'd also imagine that kind of thing being optimized for SAN-like 
>situations, which is not exactly like 4MB over 700km ;)

>What am I missing ? 

you? nothing... me a lot of things... :)


 >PS: Please report your experiences afterwards, Thanks!
don't matter about it. :)

-- 
=============
Romain BOTTAN




More information about the drbd-user mailing list