[DRBD-user] drbd performance with GbE in connected mode

Weilin Gong wgong at alcatel-lucent.com
Tue Jan 16 17:29:42 CET 2007

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Hi Ralf,

In the data you attached, I see many CPU% being greater than 100% when 
the thread number
increases. Can you explain it? Is this "tiotest" tool problem?

Thanks,
Weilin

Ralf Gross wrote:
> Hi,
>
> just for the record. Here are some numbers comparing ext3 and xfs on a
> 300GB lvm on top of drbd. 
>
> sndbuf-size      256k;
>
> tiobench --numruns 3 --threads 1 --threads 2 --threads 4 --threads 8
> --block 4096 --size 8000
>
> Primary
> HP DL380G4 with attached easyRAID array (Raid 6)
> 2 x Xeon 2.8GHz
> 6GB RAM
> 2x e1000 GbE NIC
>
> Secondary 
> custom build server
> Areca ARC-1230 RAID 5
> Core 2 Duo 2.4GHz
> 4GB RAM
> 2x e1000 GbE NIC
>
> The systems are located in different buildings and connected by an
> Cisco EtherChannel. Linux bonding mode 2 is used to combine the
> interfaces (redundancy).
>
>
> *xfs disconnected*
>
> Sequential Reads
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1  155.26  26.21%  0.075    1391.34  0.00000  0.00000  592
> 8000  4096  2  139.51  45.73%  0.165    1192.11  0.00000  0.00000  305
> 8000  4096  4  141.78  94.20%  0.324    1900.89  0.00000  0.00000  151
> 8000  4096  8  141.48  188.6%  0.653    4524.14  0.00000  0.00000   75
>
> Random Reads
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1    1.89  0.662%  6.186     273.18  0.00000  0.00000  286
> 8000  4096  2    3.42  2.073%  6.506     250.17  0.00000  0.00000  165
> 8000  4096  4    6.05  7.435%  7.393     209.76  0.00000  0.00000   81
> 8000  4096  8    8.42  12.22%  9.025     270.52  0.00000  0.00000   69
>
> Sequential Writes
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1  143.66  41.24%  0.060     582.16  0.00000  0.00000  348
> 8000  4096  2  145.34  101.8%  0.107    1311.46  0.00000  0.00000  143
> 8000  4096  4  156.98  297.3%  0.208    2981.34  0.00000  0.00000   53
> 8000  4096  8  155.86  784.8%  0.409    4140.28  0.00000  0.00000   20
>
> Random Writes
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1   25.92  22.78%  0.018      0.07  0.00000  0.00000  114
> 8000  4096  2   29.97  55.75%  0.021      0.25  0.00000  0.00000   54
> 8000  4096  4   29.76  168.8%  0.032      0.20  0.00000  0.00000   18
> 8000  4096  8   28.99  425.2%  0.065      0.58  0.00000  0.00000    7
>
>
>
> *xfs connected*
>
> Sequential Reads
> File  Blk  Num                 Avg     Maximum   Lat%   Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency  Latency   >2s    >10s     Eff
> ---- ----- --- ------  ------ -------- --------  ------- -------- ----
> 8000  4096  1  154.68  26.17%  0.075   1239.30   0.00000  0.00000  591
> 8000  4096  2  145.86  48.72%  0.160   1013.34   0.00000  0.00000  299
> 8000  4096  4  141.54  94.73%  0.325   2227.22   0.00000  0.00000  149
> 8000  4096  8  141.33  185.5%  0.647   3515.19   0.00000  0.00000   76
>
> Random Reads
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1    1.93  0.461%  6.063    231.49   0.00000  0.00000  419
> 8000  4096  2    3.64  1.552%  6.331    240.24   0.00000  0.00000  234
> 8000  4096  4    6.18  6.541%  7.185    230.83   0.00000  0.00000   95
> 8000  4096  8   10.13  31.37%  8.596    148.63   0.00000  0.00000   32
>
> Sequential Writes
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1   97.14  41.78%  0.085     674.78  0.00000  0.00000  232
> 8000  4096  2  100.00  135.8%  0.163    1020.07  0.00000  0.00000   74
> 8000  4096  4   98.27  409.4%  0.330    2536.07  0.00000  0.00000   24
> 8000  4096  8   98.58  1067.%  0.651    3276.77  0.00000  0.00000    9
>
> Random Writes
> File  Blk  Num                  Avg    Maximum   Lat%    Lat%     CPU
> Size  Size Thr   Rate  (CPU%)  Latency Latency   >2s     >10s     Eff
> ---- ----- ---  ------ ------  -------  -------  -------  ------- ----
> 8000  4096  1   29.96  33.23%  0.019       0.09  0.00000  0.00000   90
> 8000  4096  2   16.89  42.66%  0.021       0.21  0.00000  0.00000   40
> 8000  4096  4   24.96  156.5%  0.032       0.24  0.00000  0.00000   16
> 8000  4096  8   17.32  331.4%  0.065       1.56  0.00000  0.00000    5
>
>
>
> *ext3 disconnected*
>
> Sequential Reads
> File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ------ ------  -------  ------- ---------  ------- ---
> 8000  4096  1  164.83  24.12%  0.070    1146.21   0.00000  0.00000 683
> 8000  4096  2  142.32  43.10%  0.163    1117.26   0.00000  0.00000 330
> 8000  4096  4  119.18  76.27%  0.389    2781.03   0.00000  0.00000 156
> 8000  4096  8  113.66  147.7%  0.816    4131.87   0.00000  0.00000 77
>
> Random Reads
> File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ------ ------  -------  -------- --------  ------- ---
> 8000  4096  1    1.61  0.439%  7.279     298.48   0.00000  0.00000 366
> 8000  4096  2    3.06  1.854%  7.442     221.55   0.00000  0.00000 165
> 8000  4096  4    5.28  5.269%  8.230     255.47   0.00000  0.00000 100
> 8000  4096  8    7.46  18.97%  11.115    168.70   0.00000  0.00000  39
>
> Sequential Writes
> File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ------ ------  -------  -------- -------- ------- ----
> 8000  4096  1  127.97  65.06%  0.086     2639.64  0.00000  0.00000 197
> 8000  4096  2  129.81  169.4%  0.166     3200.90  0.00000  0.00000  77
> 8000  4096  4  126.40  596.7%  0.267     4772.29  0.00000  0.00000  21
> 8000  4096  8  117.99  2937.%  0.546     9165.83  0.00005  0.00000   4
>
> Random Writes
> File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ----- ------  --------  -------- -------- ------- ----
> 8000  4096  1    3.28 2.213%  3.157      282.26   0.00000  0.00000 148
> 8000  4096  2    3.93 8.518%  5.166      399.06   0.00000  0.00000  46
> 8000  4096  4    9.59 20.38%  3.217      490.80   0.00000  0.00000  47
> 8000  4096  8   27.64 421.5%  0.151       57.32   0.00000  0.00000   7
>
>
> *ext3 connected*
>
> Sequential Reads
> File  Blk   Num                  Avg   Maximum   Lat%    Lat%    CPU
> Size  Size  Thr   Rate  (CPU%)  Latency Latency   >2s     >10s    Eff
> ---- ----- --- ------  ------ -------  -------   -------  ------- ----
> 8000  4096  1  164.33  25.15%  0.071   1310.68   0.00000  0.00000  653
> 8000  4096  2  148.49  46.06%  0.155    865.10   0.00000  0.00000  322
> 8000  4096  4  130.86  82.90%  0.352   2085.03   0.00000  0.00000  158
> 8000  4096  8  121.80  153.7%  0.756   3962.73   0.00000  0.00000   79
>
> Random Reads
> File  Blk   Num                 Avg    Maximum    Lat%   Lat%    CPU
> Size  Size  Thr   Rate (CPU%) Latency  Latency    >2s    >10s    Eff
> ---- ----- ---   ----  ------ ------- --------  --------  ------- ----
> 8000  4096  1    1.73  0.501%   6.778   379.93  0.00000  0.00000  345
> 8000  4096  2    3.23  2.151%   7.088   208.11  0.00000  0.00000  150
> 8000  4096  4    5.14  7.724%   8.924   302.45  0.00000  0.00000   67
> 8000  4096  8    7.47  19.36%  11.497   170.00  0.00000  0.00000   39
>
> Sequential Writes
> File  Blk   Num                 Avg     Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ------ ------  -------- --------- ------- ------- ----
> 8000  4096  1   82.67  47.44%   0.135   10841.54  0.00049  0.00000 174
> 8000  4096  2   67.73  103.7%   0.269   18573.17  0.00244  0.00010  65
> 8000  4096  4   65.53  406.6%   0.502   21768.65  0.00390  0.00010  16
> 8000  4096  8   62.43  1332.%   1.024   18547.28  0.01231  0.00000   5
>
> Random Writes
> File  Blk   Num                Avg      Maximum   Lat%     Lat%    CPU
> Size  Size  Thr  Rate  (CPU%)  Latency  Latency   >2s      >10s    Eff
> ---- ----- ---  ------ ------  -------  ----------------- ------- ----
> 8000  4096  1    2.98  7.512%  2.934      267.82  0.00000  0.00000 40
> 8000  4096  2    1.77  2.649%  6.144     5770.42  0.02500  0.00000 67
> 8000  4096  4    1.33  4.263%  5.044     7224.07  0.02500  0.00000 31
> 8000  4096  8    1.11  7.739%  6.045    10170.34  0.02500  0.00000 14
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>   

-------------- next part --------------
A non-text attachment was scrubbed...
Name: wgong.vcf
Type: text/x-vcard
Size: 126 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20070116/08a3236f/attachment.vcf>


More information about the drbd-user mailing list