[DRBD-user] drbd performance with GbE in connected mode

Ralf Gross Ralf-Lists at ralfgross.de
Tue Jan 16 12:57:13 CET 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi,

just for the record. Here are some numbers comparing ext3 and xfs on a
300GB lvm on top of drbd. 

sndbuf-size      256k;

tiobench --numruns 3 --threads 1 --threads 2 --threads 4 --threads 8
--block 4096 --size 8000

Primary
HP DL380G4 with attached easyRAID array (Raid 6)
2 x Xeon 2.8GHz
6GB RAM
2x e1000 GbE NIC

Secondary 
custom build server
Areca ARC-1230 RAID 5
Core 2 Duo 2.4GHz
4GB RAM
2x e1000 GbE NIC

The systems are located in different buildings and connected by an
Cisco EtherChannel. Linux bonding mode 2 is used to combine the
interfaces (redundancy).


*xfs disconnected*

Sequential Reads
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1  155.26  26.21%  0.075    1391.34  0.00000  0.00000  592
8000  4096  2  139.51  45.73%  0.165    1192.11  0.00000  0.00000  305
8000  4096  4  141.78  94.20%  0.324    1900.89  0.00000  0.00000  151
8000  4096  8  141.48  188.6%  0.653    4524.14  0.00000  0.00000   75

Random Reads
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1    1.89  0.662%  6.186     273.18  0.00000  0.00000  286
8000  4096  2    3.42  2.073%  6.506     250.17  0.00000  0.00000  165
8000  4096  4    6.05  7.435%  7.393     209.76  0.00000  0.00000   81
8000  4096  8    8.42  12.22%  9.025     270.52  0.00000  0.00000   69

Sequential Writes
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1  143.66  41.24%  0.060     582.16  0.00000  0.00000  348
8000  4096  2  145.34  101.8%  0.107    1311.46  0.00000  0.00000  143
8000  4096  4  156.98  297.3%  0.208    2981.34  0.00000  0.00000   53
8000  4096  8  155.86  784.8%  0.409    4140.28  0.00000  0.00000   20

Random Writes
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1   25.92  22.78%  0.018      0.07  0.00000  0.00000  114
8000  4096  2   29.97  55.75%  0.021      0.25  0.00000  0.00000   54
8000  4096  4   29.76  168.8%  0.032      0.20  0.00000  0.00000   18
8000  4096  8   28.99  425.2%  0.065      0.58  0.00000  0.00000    7



*xfs connected*

Sequential Reads
File  Blk  Num                 Avg     Maximum   Lat%   Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency  Latency   >2s    >10s     Eff
---- ----- --- ------  ------ -------- --------  ------- -------- ----
8000  4096  1  154.68  26.17%  0.075   1239.30   0.00000  0.00000  591
8000  4096  2  145.86  48.72%  0.160   1013.34   0.00000  0.00000  299
8000  4096  4  141.54  94.73%  0.325   2227.22   0.00000  0.00000  149
8000  4096  8  141.33  185.5%  0.647   3515.19   0.00000  0.00000   76

Random Reads
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1    1.93  0.461%  6.063    231.49   0.00000  0.00000  419
8000  4096  2    3.64  1.552%  6.331    240.24   0.00000  0.00000  234
8000  4096  4    6.18  6.541%  7.185    230.83   0.00000  0.00000   95
8000  4096  8   10.13  31.37%  8.596    148.63   0.00000  0.00000   32

Sequential Writes
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1   97.14  41.78%  0.085     674.78  0.00000  0.00000  232
8000  4096  2  100.00  135.8%  0.163    1020.07  0.00000  0.00000   74
8000  4096  4   98.27  409.4%  0.330    2536.07  0.00000  0.00000   24
8000  4096  8   98.58  1067.%  0.651    3276.77  0.00000  0.00000    9

Random Writes
File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
---- ----- ---  ------ ------  -------  -------  -------  ------- ----
8000  4096  1   29.96  33.23%  0.019       0.09  0.00000  0.00000   90
8000  4096  2   16.89  42.66%  0.021       0.21  0.00000  0.00000   40
8000  4096  4   24.96  156.5%  0.032       0.24  0.00000  0.00000   16
8000  4096  8   17.32  331.4%  0.065       1.56  0.00000  0.00000    5



*ext3 disconnected*

Sequential Reads
File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ------ ------  -------  ------- ---------  ------- ---
8000  4096  1  164.83  24.12%  0.070    1146.21   0.00000  0.00000 683
8000  4096  2  142.32  43.10%  0.163    1117.26   0.00000  0.00000 330
8000  4096  4  119.18  76.27%  0.389    2781.03   0.00000  0.00000 156
8000  4096  8  113.66  147.7%  0.816    4131.87   0.00000  0.00000 77

Random Reads
File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ------ ------  -------  -------- --------  ------- ---
8000  4096  1    1.61  0.439%  7.279     298.48   0.00000  0.00000 366
8000  4096  2    3.06  1.854%  7.442     221.55   0.00000  0.00000 165
8000  4096  4    5.28  5.269%  8.230     255.47   0.00000  0.00000 100
8000  4096  8    7.46  18.97%  11.115    168.70   0.00000  0.00000  39

Sequential Writes
File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ------ ------  -------  -------- -------- ------- ----
8000  4096  1  127.97  65.06%  0.086     2639.64  0.00000  0.00000 197
8000  4096  2  129.81  169.4%  0.166     3200.90  0.00000  0.00000  77
8000  4096  4  126.40  596.7%  0.267     4772.29  0.00000  0.00000  21
8000  4096  8  117.99  2937.%  0.546     9165.83  0.00005  0.00000   4

Random Writes
File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ----- ------  --------  -------- -------- ------- ----
8000  4096  1    3.28 2.213%  3.157      282.26   0.00000  0.00000 148
8000  4096  2    3.93 8.518%  5.166      399.06   0.00000  0.00000  46
8000  4096  4    9.59 20.38%  3.217      490.80   0.00000  0.00000  47
8000  4096  8   27.64 421.5%  0.151       57.32   0.00000  0.00000   7


*ext3 connected*

Sequential Reads
File  Blk   Num                  Avg   Maximum   Lat%    Lat%    CPU
Size  Size  Thr   Rate  (CPU%)  Latency Latency   >2s     >10s    Eff
---- ----- --- ------  ------ -------  -------   -------  ------- ----
8000  4096  1  164.33  25.15%  0.071   1310.68   0.00000  0.00000  653
8000  4096  2  148.49  46.06%  0.155    865.10   0.00000  0.00000  322
8000  4096  4  130.86  82.90%  0.352   2085.03   0.00000  0.00000  158
8000  4096  8  121.80  153.7%  0.756   3962.73   0.00000  0.00000   79

Random Reads
File  Blk   Num                 Avg    Maximum    Lat%   Lat%    CPU
Size  Size  Thr   Rate (CPU%) Latency  Latency    >2s    >10s    Eff
---- ----- ---   ----  ------ ------- --------  --------  ------- ----
8000  4096  1    1.73  0.501%   6.778   379.93  0.00000  0.00000  345
8000  4096  2    3.23  2.151%   7.088   208.11  0.00000  0.00000  150
8000  4096  4    5.14  7.724%   8.924   302.45  0.00000  0.00000   67
8000  4096  8    7.47  19.36%  11.497   170.00  0.00000  0.00000   39

Sequential Writes
File  Blk   Num                 Avg     Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ------ ------  -------- --------- ------- ------- ----
8000  4096  1   82.67  47.44%   0.135   10841.54  0.00049  0.00000 174
8000  4096  2   67.73  103.7%   0.269   18573.17  0.00244  0.00010  65
8000  4096  4   65.53  406.6%   0.502   21768.65  0.00390  0.00010  16
8000  4096  8   62.43  1332.%   1.024   18547.28  0.01231  0.00000   5

Random Writes
File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
---- ----- ---  ------ ------  -------  ----------------- ------- ----
8000  4096  1    2.98  7.512%  2.934      267.82  0.00000  0.00000 40
8000  4096  2    1.77  2.649%  6.144     5770.42  0.02500  0.00000 67
8000  4096  4    1.33  4.263%  5.044     7224.07  0.02500  0.00000 31
8000  4096  8    1.11  7.739%  6.045    10170.34  0.02500  0.00000 14





More information about the drbd-user mailing list