[DRBD-user] drbd performance with GbE in connected mode

Ross S. W. Walker rwalker at medallion.com
Sun Jan 14 23:15:30 CET 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com 
> [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Ralf Gross
> Sent: Sunday, January 14, 2007 4:21 PM
> To: drbd-user at lists.linbit.com
> Subject: [DRBD-user] drbd performance with GbE in connected mode
> 
> Hi,
> 
> I've two server that are connected by a dedicated GbE connection.
> Benchmarks with netpipe/netio are showing good GbE performance. Even a
> drbd sync shows >100MB/s  (total 19179 sec; paused 8925 sec; 106968
> K/sec).
> 
> NPtcp:
> [...]
> 120: 6291459 bytes      3 times -->    892.94 Mbps in   53754.85 usec
> 121: 8388605 bytes      3 times -->    868.80 Mbps in   73664.51 usec
> 122: 8388608 bytes      3 times -->    881.13 Mbps in   72634.30 usec
> 123: 8388611 bytes      3 times -->    877.56 Mbps in   72929.82 usec
> 
> NETIO - Network Throughput Benchmark, Version 1.26
> (C) 1997-2005 Kai Uwe Rommel
> 
> TCP connection established.
> Packet size  1k bytes:  114306 KByte/s Tx,  114558 KByte/s Rx.
> Packet size  2k bytes:  114575 KByte/s Tx,  114573 KByte/s Rx.
> Packet size  4k bytes:  114608 KByte/s Tx,  114573 KByte/s Rx.
> Packet size  8k bytes:  114612 KByte/s Tx,  114554 KByte/s Rx.
> Packet size 16k bytes:  114608 KByte/s Tx,  114562 KByte/s Rx.
> Packet size 32k bytes:  114585 KByte/s Tx,  114567 KByte/s Rx.
> Done.
> 
> 
> In disconnected mode I can reach >120Mb/s write performance on both
> sides.
> 
> Sequential Reads
> File  Blk   Num                   Avg    Maximum    Lat%    Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s     >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------  ------- ---
> 8000  4096   1  163.66 24.57%     0.118  1734.92  0.00000  0.00000 666
> 
> Random Reads
> File  Blk   Num                   Avg    Maximum    Lat%     Lat% CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      >10s  Eff
> ----- ----- --- ------ ------ --------- --------- ------- -------- ---
> 8000  4096   1    1.65 0.648%    11.861   269.61  0.00000  0.00000 254
> 
> Sequential Writes
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------- ------- ---
> 8000  4096   1  126.34 65.20%     0.143  2579.81  0.00000  0.00000 194
> 
> Random Writes
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------  ------- ---
> 8000  4096   1    3.26 2.255%     5.352   635.71  0.00000  0.00000 145
> 
> 
> 
> Now, when both server are connected, write performance is 
> between 70 and
> 80MB/s.
> 
> 
> Sequential Reads
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------- 
> -------- ---
> 8000  4096    1  161.03 25.72%     0.120 1910.72  0.00000  
> 0.00000  626
> 
> Random Reads
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- --------  
> ------- ---
> 8000  4096    1   1.72 0.571%    11.375   738.84   0.00000  
> 0.00000 300
> 
> Sequential Writes
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------  
> -------- ---
> 8000  4096    1  71.62 42.12%     0.257  17274.23  0.00117  
> 0.00000 170
> 
> Random Writes
> File  Blk   Num                   Avg    Maximum    Lat%     Lat%  CPU
> Size  Size  Thr  Rate  (CPU%)  Latency   Latency    >2s      
> >10s   Eff
> ----- ----- --- ------ ------ --------- --------- -------- 
> -------- ---
> 8000  4096    1   2.58 2.364%     5.107   299.99  0.00000  
> 0.00000  109
> 
> 
> 
> I understand that write performace is limited by GbE performace. But
> shouldn't the write performance be about 90-100MB/s? This is 
> what I can
> get with the drbd benchmark tool.
> 
> ./dm -x -a 0 -s 4g -b 20m -m -y -p -o /mnt/test
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
> RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
> RRRRRRRRRRRRRRRRRRr
> 97.63 MB/sec (4294967296 B / 00:41.952388
> 
> And a simple dd test.
> 
> dd if=/dev/zero of=/mnt/foo bs=1024 count=5000000
> 5000000+0 Datensätze ein
> 5000000+0 Datensätze aus
> 5120000000 Bytes (5,1 GB) kopiert, 61,3171 Sekunden, 83,5 MB/s
> 
> 
> I already tried different setting in drbd.conf.
> 
>  sndbuf-size      1M;
>  max-buffers      20480;
>  max-epoch-size   16384;
> 
> 
> The two systems:
> 
> Debian etch 
> drbd 0.7.21 (debian package)
> 
> server 1: Dual Xeon 2.8 GHz
>           HP cciss Raid 1 for OS
>           easyRAID ext. SATA Raid array with 4 drbd devices on Raid6
> 
> server 2: Core 2 Duo 6600 2.4 GHz
>           Areca ARC-1230 PCI-e int. SATA Raid controller
> 
> The tested ext3 fs is on a 300GB lvm vg.
> 
> Are these the numbers I have to expect? Anything more I could try to
> impove the write throughput?
> 
> Ralf

First of all let me ask which benchmark tool you used for the sequential/random io runs? I like the information it provides and can use a tool like that.

Second, using Prot C, the writes do not return until they are committed to disk on both sides, so cannot the loss of throughput be attributed to the increased latency of writing on host1 while it sends out the write over 1Gbps wire to host2, who then writes it to disk and then sends a successful response back to host1 who then returns?

If the local write and send happened at the same time Copy/Send Latency + Gbps latency + Remote Write Latency + Gbps return latency = increased latency. How much increased latency depends on the speed/effectiveness of the network and the speed of the remote storage versus the speed of the local storage. The slowest storage is always the weakest link, try to get storage systems that are the same for most reliable performance.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.




More information about the drbd-user mailing list