Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
> -----Original Message----- > From: drbd-user-bounces at lists.linbit.com > [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Ralf Gross > Sent: Sunday, January 14, 2007 4:21 PM > To: drbd-user at lists.linbit.com > Subject: [DRBD-user] drbd performance with GbE in connected mode > > Hi, > > I've two server that are connected by a dedicated GbE connection. > Benchmarks with netpipe/netio are showing good GbE performance. Even a > drbd sync shows >100MB/s (total 19179 sec; paused 8925 sec; 106968 > K/sec). > > NPtcp: > [...] > 120: 6291459 bytes 3 times --> 892.94 Mbps in 53754.85 usec > 121: 8388605 bytes 3 times --> 868.80 Mbps in 73664.51 usec > 122: 8388608 bytes 3 times --> 881.13 Mbps in 72634.30 usec > 123: 8388611 bytes 3 times --> 877.56 Mbps in 72929.82 usec > > NETIO - Network Throughput Benchmark, Version 1.26 > (C) 1997-2005 Kai Uwe Rommel > > TCP connection established. > Packet size 1k bytes: 114306 KByte/s Tx, 114558 KByte/s Rx. > Packet size 2k bytes: 114575 KByte/s Tx, 114573 KByte/s Rx. > Packet size 4k bytes: 114608 KByte/s Tx, 114573 KByte/s Rx. > Packet size 8k bytes: 114612 KByte/s Tx, 114554 KByte/s Rx. > Packet size 16k bytes: 114608 KByte/s Tx, 114562 KByte/s Rx. > Packet size 32k bytes: 114585 KByte/s Tx, 114567 KByte/s Rx. > Done. > > > In disconnected mode I can reach >120Mb/s write performance on both > sides. > > Sequential Reads > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff > ----- ----- --- ------ ------ --------- --------- ------- ------- --- > 8000 4096 1 163.66 24.57% 0.118 1734.92 0.00000 0.00000 666 > > Random Reads > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff > ----- ----- --- ------ ------ --------- --------- ------- -------- --- > 8000 4096 1 1.65 0.648% 11.861 269.61 0.00000 0.00000 254 > > Sequential Writes > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- -------- ------- --- > 8000 4096 1 126.34 65.20% 0.143 2579.81 0.00000 0.00000 194 > > Random Writes > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- ------- ------- --- > 8000 4096 1 3.26 2.255% 5.352 635.71 0.00000 0.00000 145 > > > > Now, when both server are connected, write performance is > between 70 and > 80MB/s. > > > Sequential Reads > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- -------- > -------- --- > 8000 4096 1 161.03 25.72% 0.120 1910.72 0.00000 > 0.00000 626 > > Random Reads > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- -------- > ------- --- > 8000 4096 1 1.72 0.571% 11.375 738.84 0.00000 > 0.00000 300 > > Sequential Writes > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- ------- > -------- --- > 8000 4096 1 71.62 42.12% 0.257 17274.23 0.00117 > 0.00000 170 > > Random Writes > File Blk Num Avg Maximum Lat% Lat% CPU > Size Size Thr Rate (CPU%) Latency Latency >2s > >10s Eff > ----- ----- --- ------ ------ --------- --------- -------- > -------- --- > 8000 4096 1 2.58 2.364% 5.107 299.99 0.00000 > 0.00000 109 > > > > I understand that write performace is limited by GbE performace. But > shouldn't the write performance be about 90-100MB/s? This is > what I can > get with the drbd benchmark tool. > > ./dm -x -a 0 -s 4g -b 20m -m -y -p -o /mnt/test > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR > RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR > RRRRRRRRRRRRRRRRRRr > 97.63 MB/sec (4294967296 B / 00:41.952388 > > And a simple dd test. > > dd if=/dev/zero of=/mnt/foo bs=1024 count=5000000 > 5000000+0 Datensätze ein > 5000000+0 Datensätze aus > 5120000000 Bytes (5,1 GB) kopiert, 61,3171 Sekunden, 83,5 MB/s > > > I already tried different setting in drbd.conf. > > sndbuf-size 1M; > max-buffers 20480; > max-epoch-size 16384; > > > The two systems: > > Debian etch > drbd 0.7.21 (debian package) > > server 1: Dual Xeon 2.8 GHz > HP cciss Raid 1 for OS > easyRAID ext. SATA Raid array with 4 drbd devices on Raid6 > > server 2: Core 2 Duo 6600 2.4 GHz > Areca ARC-1230 PCI-e int. SATA Raid controller > > The tested ext3 fs is on a 300GB lvm vg. > > Are these the numbers I have to expect? Anything more I could try to > impove the write throughput? > > Ralf First of all let me ask which benchmark tool you used for the sequential/random io runs? I like the information it provides and can use a tool like that. Second, using Prot C, the writes do not return until they are committed to disk on both sides, so cannot the loss of throughput be attributed to the increased latency of writing on host1 while it sends out the write over 1Gbps wire to host2, who then writes it to disk and then sends a successful response back to host1 who then returns? If the local write and send happened at the same time Copy/Send Latency + Gbps latency + Remote Write Latency + Gbps return latency = increased latency. How much increased latency depends on the speed/effectiveness of the network and the speed of the remote storage versus the speed of the local storage. The slowest storage is always the weakest link, try to get storage systems that are the same for most reliable performance. -Ross ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.