Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Thanks guys, appreciate the replies!
After trying the on-congestion pull-ahead, which didn't change anything, I
went a few lines up and commented out my data-integrity-alg sha256; line.
Now I'm achieving 200 MB/s, which is close to native disk speed of ~245.
I can only speculate that the SHA256 hashing algorithm in the kernel is too
slow because, as said, then it would limit my write speed to about 90 MB/s.
(Debian wheezy 7.1 / kernel 3.2.0-4-amd64)
When I use 'md5' the speed is 180 MB/s, so only slightly slower than without
data integrity hashing.
I'm on a i5-3570 CPU @ 3.40GHz by the way, so I didn't expect this.
The SHA256 implementation in openssl (so not the kernel implementation that
is actually used.. not sure how to test that), indicates it shouldn't be
that slow...
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
sha256 57214.74k 127829.10k 224316.79k 276672.51k 295370.75k
Anyway, glad I can get into production now :)
Oh one thing, as I mentioned earlier the resync speed was 220MB/s. If the
hashing algo indeed is limited to 100MB/s (which is the unproven theory of
above), does this mean the data integrity algo is unused for a resync?
Regards,
Bram.
J.R. Lillard wrote, on 10-11-2013 20:21:
> Have you tried on-congestion pull-ahead in protocol A?
>
> J.R. Lillard
> System / Network Admin
> Web Programmer
> Golden Heritage Foods
>
> On Nov 10, 2013 1:11 PM, "Bram Matthys" <syzop at vulnscan.org
> <mailto:syzop at vulnscan.org>> wrote:
>
> Hi all,
>
> Any ideas? I'm essentially blocked now.. I can't go into production with
> these write speeds.
>
> Am I the only one who experiences a >60% degradation of write speed when the
> two nodes are connected to each other? With full disk speed only in
> disconnected state?
>
> Regards,
>
> Bram.
>
> Bram Matthys wrote, on 3-11-2013 11:26:
> > Hi,
> >
> > Using a RAID10 array now and (following Arnolds advice) the metadata
> is now
> > on another (SSD) disk.
> > Write speeds are still bad, but are ONLY low when both nodes are
> connected.
> >
> > root at st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd
> > if=/dev/zero of=5G bs=1M count=5000; rm -f 5G
> > 5000+0 records in
> > 5000+0 records out
> > 5242880000 bytes (5.2 GB) copied, 57.1802 s, 91.7 MB/s
> >
> > root at st2:/data# drbdadm disconnect data
> > root at st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd
> > if=/dev/zero of=5G bs=1M count=5000; rm -f 5G
> > 5000+0 records in
> > 5000+0 records out
> > 5242880000 bytes (5.2 GB) copied, 21.4724 s, 244 MB/s
> >
> > When I (re)connect data again, the resync speed is fine:
> > st1# cat /proc/drbd
> > version: 8.3.11 (api:88/proto:86-96)
> > srcversion: F937DCB2E5D83C6CCE4A6C9
> >
> > 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
> > ns:0 nr:5121168 dw:5121168 dr:2010340 al:0 bm:133 lo:0 pe:206 ua:0
> ap:0
> > ep:1 wo:d oos:3130232
> > [======>.............] sync'ed: 38.9% (3056/4996)Mfinish: 0:00:14
> > speed: 220,760 (220,760) want: 256,000 K/sec
> >
> > And, as mentioned earlier, netperf achieves near-10gE speed:
> > # netperf -H 192.168.220.1
> > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.220.1
> > (192.168.220.1) port 0 AF_INET : demo
> > Recv Send Send
> > Socket Socket Message Elapsed
> > Size Size Size Time Throughput
> > bytes bytes bytes secs. 10^6bits/sec
> >
> > 87380 16384 16384 10.00 9897.79
> > (same in both directions)
> >
> > It doesn't matter whether I make server st1 or st2 primary (so either
> > server) and do the tests... with both they achieve around 240 MB/s write
> > speed in disconnected mode and around 80-90 MB/s when connected, so ~
> 1/3rd.
> >
> > Any ideas?
> >
> > # drbdadm dump
> > # /etc/drbd.conf
> > common {
> > protocol C;
> > startup {
> > degr-wfc-timeout 120;
> > wfc-timeout 120;
> > }
> > }
> >
> > # resource data on st2: not ignored, not stacked
> > resource data {
> > on st1 {
> > device /dev/drbd1 minor 1;
> > disk /dev/md4;
> > address ipv4 192.168.220.1:7789
> <http://192.168.220.1:7789>;
> > meta-disk /dev/md3 [0];
> > }
> > on st2 {
> > device /dev/drbd1 minor 1;
> > disk /dev/md4;
> > address ipv4 192.168.220.2:7789
> <http://192.168.220.2:7789>;
> > meta-disk /dev/md3 [0];
> > }
> > net {
> > data-integrity-alg sha256;
> > max-buffers 8000;
> > max-epoch-size 8000;
> > sndbuf-size 512k;
> > }
> > disk {
> > no-disk-barrier;
> > no-disk-flushes;
> > }
> > syncer {
> > csums-alg sha256;
> > rate 250M;
> > }
> > }
> >
> > NOTE: I was using protocol A without no-disk-* before, just using
> > C/no-disk-* to see if it made any significant difference: answer is no.
> >
> > root at st2:/data# uname -a
> > Linux st2 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux
> > root at st2:/data# cat /proc/drbd
> > version: 8.3.11 (api:88/proto:86-96)
> > srcversion: F937DCB2E5D83C6CCE4A6C9
> >
> > 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
> > ns:5121332 nr:0 dw:15358180 dr:5130289 al:3773 bm:377 lo:0 pe:0 ua:0
> > ap:0 ep:1 wo:d oos:0
> >
> > root at st2:/data# dpkg --list|grep drbd
> > ii drbd8-utils 2:8.3.13-2 amd64
> > RAID 1 over tcp/ip for Linux utilities
> >
> > If you need anything else, just let me know.
> >
> > Thanks again,
> >
> > Bram.
> >
> > Arnold Krille wrote, on 7-10-2013 1:37:
> >> Hi,
> >>
> >> On Sun, 06 Oct 2013 18:54:12 +0200 Bram Matthys <syzop at vulnscan.org
> <mailto:syzop at vulnscan.org>>
> >> wrote:
> >>> I'm currently testing DRBD and am having write performance problems.
> >>> On the local raid array I achieve 124MB/s, but with DRBD I get only
> >>> 41MB/s out of it, or if the secondary node is down (to rule out
> >>> network issues) then 53MB/s at best.
> >>> Tried protocol A / B / C, with and without no-disk-barriers and
> >>> no-disk-flushes, but this didn't change much (only +/- 2MB/s
> >>> difference).
> >> <snip>
> >>
> >> Yep, internal meta-disk: write block, seek to end, write log, seek to
> >> front, write block, seek to end, write log...
> >>
> >> Put the meta-disk on a different hd and watch your write rate go up.
> >>
> >> - Arnold
> >>
> >>
> >>
> >> _______________________________________________
> >> drbd-user mailing list
> >> drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
> >> http://lists.linbit.com/mailman/listinfo/drbd-user
> >>
> >
> >
>
>
> --
> Bram Matthys
> Software developer/IT consultant syzop at vulnscan.org
> <mailto:syzop at vulnscan.org>
> Website: www.vulnscan.org
> <http://www.vulnscan.org>
> PGP key: www.vulnscan.org/pubkey.asc
> <http://www.vulnscan.org/pubkey.asc>
> PGP fp: EBCA 8977 FCA6 0AB0 6EDB 04A7 6E67 6D45 7FE1 99A6
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com <mailto:drbd-user at lists.linbit.com>
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
--
Bram Matthys
Software developer/IT consultant syzop at vulnscan.org
Website: www.vulnscan.org
PGP key: www.vulnscan.org/pubkey.asc
PGP fp: EBCA 8977 FCA6 0AB0 6EDB 04A7 6E67 6D45 7FE1 99A6