Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On 9/5/17 16:31, Steven Maddox wrote:
> Hey,
>
> We use DRBD at work in a very simple way... but since rebuilding it
> we've found it runs slower than before.
When you say slower than before, how fast was it before, and how fast is
it now? Do you have actual specific tests/numbers? Also, what changed
from "before" to "now"?
> Setup is as follows...
>
> Sites A and B both have a Debian 8 VM and they also both have a physical
> SAN each.
>
> These VM's use Open-iSCSI / Multipath to connect to their respective
> SAN's (where 12 real disks are used in raid 6 to make one 30TB virtual
> disk, and that virtual disk just has one partition with ext4 fs).
>
> The VM's replicate that partition between them using DRBD. Site A is
> always primary and Site B always secondary (unless a dragon takes out
> Site A - long story).
>
> Protocol 'A' was chosen as I want writes to happen at Site A immediately
> as the data goes to Site B over a WAN link.
>
> The only reason the below config doesn't look like a WAN link is because
> there's a VPLS between them (think of it like a layer 2 tunnel). Using
> iperf3 I can see general throughput is very good, around 700 to 900 mbps
> region... but we only get about 800 K/sec showing in drbd-overview.
Remember, the replication speed is based more on the latency rather than
the bandwidth. So look at the ping time over the link, compare that to
the ping time on a local ethernet connection..... Most likely, it will
be a *lot* slower.
> Also I've chosen DRBD 8.9.2 that comes with stock Debian 8. I'm willing
> to change that but only if someone points out some massive performance
> hampering bug with it. I prefer to stick to what Debian stable comes
> with unless there's a reason not to.
I would suggest to use at least the latest version from linbit within
the 8.4.x release. Debian won't collect all of the fixes that are
released (because they aren't just security related or major bugs), but
you will definitely solve many bugs and also gain many performance
improvements.
Note: I was in the same camp as you a couple of years ago, but since
then I've recognised that a very small number of packages can/should be
using latest release from upstream. DRBD is one of those for me.
> Also also... you'll notice there's no security on this (e.g. password)
> as it's an entirely private link... I've only put the crc32c in there as
> I'm under the impression you're supposed to have some minimal way (at
> least) of checking the data was sent correctly?
You might want to check CPU consumption at both ends, especially on each
individual CPU. crc32c is unlikely to stress any decent CPU, but it's
worth checking.
Also, given you are using VM's, you might want to check the various
other factors like network performance (offload, zero copy buffers,
etc), CPU and disk overheads. In a VM, there can be a lot of overhead
that you can easily forget about, or ignore. You could also try booting
on raw metal (if possible) at least to get a comparison.
> I've kept the stock config files the way they come with Debian 8, which
> is to say there is nothing in drbd.conf and global_common.conf (as
> everything is commented out by default - except for 'usage-count no')
>
> Just added in my own as drbd.d/sanvirtualdisk.res ...
>
> resource archive1 {
> protocol A;
> device /dev/drbd1;
> meta-disk internal;
> startup {
> wfc-timeout 0;
> degr-wfc-timeout 60;
> become-primary-on site-a;
> }
> net {
> verify-alg crc32c;
> }
> on site-a {
> disk /dev/mapper/sanvirtualdisk-part1;
> address 10.0.0.1:7789;
> }
> on site-b {
> disk /dev/mapper/sanvirtualdisk-part1;
> address 10.0.0.2:7789;
> }
> }
>
> Any suggestions for performance/integrity improvements to this config
> would be greatly appreciated - keep in mind my scenario.
Have you looked at the performance tuning chapter in the documentation?
That is probably the first place to look.
> On last thing as well (something else I've been researching for weeks
> and come up with no concrete answer) I'm wondering how I can tell Site A
> to tell Site B which parts of the filesystem are marked as empty.. that
> way speed up the initial sync. I keep changing my config and
> invalidating Site B to retest sync speeds - but it always seems to want
> to sync the whole 30TB. The virtual disks (on both sides) were
> completely zero'd recently - so in theory DRBD should realise there is
> large amounts of emptiness it could summarise (assuming it uses some
> kind of compression) to help it copy quicker? I've read TRIM would also
> help - but I'm not sure that ability survives when you're using an iSCSI
> disk.
AFAIK, you can't. What you can do is disconnect the secondary, make some
number of changes on the primary, then reconnect. DRBD will only re-sync
the changed blocks.
However, you should keep in mind that the resync performance is not an
indication of the performance of the live system. If you are trying to
improve resync speed, then you will need to use different options, but
is this really your concern, or are you actually trying to improve
overall performance?
First measure what you have, then determine what you need it to be. Then
look at config and/or hardware to get there.
> Thanks for any advice you might have!
>