Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Oct 23, 2006, at 6:02 AM, Poyner, Brandon wrote:
> That does seem a bit odd. I would try running bonnie++ on the drbd
> device to see if the results are consistent with dd. Can you also
> respond with output from federal-b's ifconfig and ethtool on both GigE
> interfaces?
Okay, I figured it out from information I left out of the original
post (sorry).
With mounting /shared "sync,noatime", boy, bonnie++ can take a long
time... and it confirms. 1.9MB/s. With the partition mounted async
(default) the rates are along the lines of ~50MB/s. Thinking about it
now it makes perfect sense. I should have realized that a sync or
format is not affected by mount options and that is the only thing
different in a normal fs write.
Now, my only question is: What are the risks of mounting the FS
async? This is ext3 with journaling. I would think that a likely
crash of federal-a would still leave its peer in a Consistent state,
and thusly shouldn't matter significantly. What considerations am I
missing? Would I be safer with the likes of another filesystem, say
reiser3 or xfs?
Thanks again for the help!
David
### /dev/drbd0 mounted "sync,noatime":
[root at federal-a ~]# bonnie++ -d /shared/temp/ -u nobody
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
federal-a 4G 1036 0 1951 0 4245 0 38340 79 143249
24 329.0 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 324 0 +++++ +++ 203 0 334 0 +++++ ++
+ 101 0
federal-a,4G,1036,0,1951,0,4245,0,38340,79,143249,24,329.0,0,16,324,0,
+++++,+++,203,0,334,0,+++++,+++,101,0
### /dev/drbd0 mounted with no options:
[root at federal-a ~]# bonnie++ -d /shared/temp/ -u nobody
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
federal-a 4G 40983 96 57791 18 37416 10 37281 76 142490
24 356.6 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 3634 99 +++++ +++ +++++ +++ 3716 99 +++++ ++
+ 12071 99
federal-a,4G,
40983,96,57791,18,37416,10,37281,76,142490,24,356.6,0,16,3634,99,++++
+,+++,+++++,+++,3716,99,+++++,+++,12071,99
>
>> -----Original Message-----
>> From: drbd-user-bounces at linbit.com
>> [mailto:drbd-user-bounces at linbit.com] On Behalf Of David Jonas
>> Sent: Friday, October 20, 2006 9:01 PM
>> To: drbd-user at linbit.com
>> Subject: [DRBD-user] Fast sync, slow regular write
>>
>> Hello everyone,
>>
>> I have a problem that google seems to have failed me on. Basically,
>> sync'ing seems to be plenty fast (~50MB/s), but when I simply write
>> to the mirrored partition things move much slower, on the order of
>> 1MB/s. I tried al-extents at 257 and 907, didn't seem to make a
>> difference. A write to a non-drbd partition is blazing fast (~285MB/
>> s). iperf also shows full speed on the link (~984 Mbits/sec, jumbo
>> frames).
>>
>> I'm completely at a loss at this point. Any ideas or information
>> would be greatly appreciated.
>>
>> Thanks,
>>
>> David
>>
>>
>> ########### Vitals ###########
>>
>> OS: Redhat EL4 2.6.9-42.ELsmp
>>
>> DRBD: drbd-0.7.21-1.c4, kernel-module-
>> drbd-2.6.9-42.ELsmp-0.7.21-1.c4 (CentOS4 RPMs)
>>
>> Connection: Intel(R) PRO/1000 Network Connection, e1000 driver,
>> MTU=9000, cat6 crossover.
>>
>> IO: 3ware 9550SX-4LP (2.26.04.010) raid-5, 4 Seagate SATA
>> participating
>>
>> Proc: Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz, MemTotal: 2073840 kB
>> (2GB), MemFree: 1456452 kB
>>
>> df: /dev/drbd0 841G 306M 798G 1% /shared
>>
>>
>> drbd.conf (complete)
>> --------------------------------------------------------------
>> ----------
>> -----------------------
>>
>> resource r0 {
>>
>> protocol C;
>> incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep
>> 60 ; halt -f";
>>
>> startup {
>> degr-wfc-timeout 120;
>> }
>>
>> disk {
>> on-io-error pass_on;
>> }
>>
>> net {
>> on-disconnect reconnect;
>> max-buffers 32768;
>> max-epoch-size 16384;
>> sndbuf-size 1M;
>> unplug-watermark 32768;
>> }
>>
>> syncer {
>> # set this as high as possible for now...
>> rate 500M;
>> group 1;
>> al-extents 907;
>> }
>>
>> on federal-a {
>> device /dev/drbd0;
>> disk /dev/sda9;
>> address 10.2.2.1:7788;
>> meta-disk internal;
>> }
>>
>> on federal-b {
>> device /dev/drbd0;
>> disk /dev/sda9;
>> address 10.2.2.2:7788;
>> meta-disk internal;
>> }
>> }
>> --------------------------------------------------------------
>> ----------
>> -----------------------
>>
>> [root at federal-a ~]# /sbin/ifconfig/eth0
>> eth0 Link encap:Ethernet HWaddr 00:30:48:89:F9:C6
>> inet addr:10.2.2.1 Bcast:10.1.1.255 Mask:255.255.255.0
>> inet6 addr: fe80::230:48ff:fe89:f9c6/64 Scope:Link
>> UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
>> RX packets:5621864 errors:0 dropped:0 overruns:0 frame:0
>> TX packets:7380589 errors:0 dropped:0 overruns:0 carrier:0
>> collisions:0 txqueuelen:1000
>> RX bytes:590376286 (563.0 MiB) TX bytes:340753247 (324.9
>> MiB)
>> Base address:0x5000 Memory:e0300000-e0320000
>>
>>
>> ########### Tests ###########
>>
>> These tests obviously weren't done at the same time. The mirror was
>> "Consistent" at the time of the dd.
>>
>> [djonas at federal-a ~]$ cat /proc/drbd
>> version: 0.7.21 (api:79/proto:74)
>> SVN Revision: 2326 build by buildsvn at build-i386, 2006-10-07 05:12:17
>> 0: cs:SyncSource st:Primary/Secondary ld:Consistent
>> ns:583570960 nr:4 dw:4 dr:583618840 al:0 bm:90251 lo:11903 pe:
>> 814 ua:11970 ap:0
>> [=========>..........] sync'ed: 46.2% (304311/564910)M
>> finish: 1:46:25 speed: 48,748 (52,384) K/sec
>>
>>
>> [root at federal-a ~]# time dd if=/dev/sda of=/shared/100M.bin bs=4096
>> count=25600
>> 25600+0 records in
>> 25600+0 records out
>>
>> real 1m35.336s
>> user 0m0.004s
>> sys 0m0.019s
>>
>>
>> [root at federal-a ~]# scp /shared/100M.bin djonas at 10.2.2.2:
>> djonas at 10.2.2.2's password:
>> 100M.bin
>>
>>
>> 1
>> 00% 100MB 33.3MB/s 00:03
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> drbd-user mailing list
>> drbd-user at lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/drbd-user
>>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>