[DRBD-user] Odd observation when sync'ing

Lee Christie Lee at titaninternet.co.uk
Thu Jul 3 16:01:44 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


Whats even odder is that looking at top we suspected CPU contention (the
boxes have a single CPU each)

so we added another CPU to both boxes. And another pair of RAM sticks.

Now - the box has a lovely spread of write I/O. However the total I/O
has dropped considerably :

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              11.40         0.00         5.70          0         28
sdb            1598.80    189779.20         0.00     948896          0
dm-0            750.80     24025.60         0.00     120128          0
dm-1            752.00     24064.00         0.00     120320          0
dm-2            756.60     24211.20         0.00     121056          0
dm-3            749.60     23987.20         0.00     119936          0
dm-4            755.00     24160.00         0.00     120800          0
dm-5            753.60     24115.20         0.00     120576          0
dm-6            754.00     24128.00         0.00     120640          0
dm-7            712.20     22790.40         0.00     113952          0

We took out the additional CPU :

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              11.00         0.00         5.50          0          5
sdb            5040.00         0.00    191584.00          0     191584
dm-0            765.00         0.00     24480.00          0      24480
dm-1            766.00         0.00     24512.00          0      24512
dm-2            753.00         0.00     24096.00          0      24096
dm-3            706.00         0.00     22592.00          0      22592
dm-4            763.00         0.00     24416.00          0      24416
dm-5            756.00         0.00     24192.00          0      24192
dm-6            761.00         0.00     24352.00          0      24352
dm-7            716.00         0.00     22912.00          0      22912

We then reverted back to 2 x RAM sticks

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              11.00         0.00        11.10          0         55
sdb            3911.40         0.00    153369.60          0     766848
dm-0            608.40         0.00     19468.80          0      97344
dm-1            610.00         0.00     19520.00          0      97600
dm-2            605.40         0.00     19372.80          0      96864
dm-3            570.60         0.00     18259.20          0      91296
dm-4            610.40         0.00     19532.80          0      97664
dm-5            608.80         0.00     19481.60          0      97408
dm-6            603.80         0.00     19321.60          0      96608
dm-7            575.00         0.00     18400.00          0      92000 

what ??!!

try as we might, we simply cannot get back to the figures we saw at the
start, namely 8000+ tps and 309MB/s I/O. Since I put the output into
this email I'm sure we didn't imagine it.

Anwers on a postcard pls.....


> -----Original Message-----
> From: drbd-user-bounces at lists.linbit.com 
> [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lee Christie
> Sent: 03 July 2008 13:25
> To: drbd-user at lists.linbit.com
> Subject: [DRBD-user] Odd observation when sync'ing
> 
> Two identical storage boxes, each with a 6 disk RAID10 carrying 8 drbd
> resources
> 
> Doing a sync :
> 
> Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
> sda              18.96         0.00         9.48          0         47
> sdb            8097.60         0.00    309429.14          0    1550240
> dm-0           1157.29         0.00     37033.13          0     185536
> dm-1           1518.16         0.00     48581.24          0     243392
> dm-2           1683.83         0.00     53882.63          0     269952
> dm-3            180.04         0.00      5761.28          0      28864
> dm-4           1839.32         0.00     58858.28          0     294880
> dm-5           1572.46         0.00     50318.56          0     252096
> dm-6           1466.87         0.00     46939.72          0     235168
> dm-7            217.96         0.00      6974.85          0      34944
> 
> Odd how two of the drbd resources are simply not getting a 
> look in when
> it comes to I/O.
> 
> This is backed up by the output from drbd :
> 
> [root at mail-storage-02 network-scripts]# cat /proc/drbd
> version: 8.2.6 (api:88/proto:86-88)
> GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by
> buildsvn at c5-x8664-build, 2008-06-26 19:33:19
>  0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:27112032 dw:27111648 dr:0 al:0 bm:1655 lo:13 pe:8777 ua:12
> ap:0 oos:65744672
> 	[====>...............] sync'ed: 29.3% (64203/90680)M
> 	finish: 0:19:01 speed: 57,472 (39,808) K/sec
>  1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:32451072 dw:32450560 dr:0 al:0 bm:1982 lo:16 pe:1429 ua:16
> ap:0 oos:60848128
> 	[=====>..............] sync'ed: 34.8% (59422/91112)M
> 	finish: 0:15:50 speed: 63,840 (47,648) K/sec
>  2: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:33048608 dw:33047904 dr:0 al:0 bm:2017 lo:22 pe:1363 ua:22
> ap:0 oos:69241504
> 	[=====>..............] sync'ed: 32.4% (67618/99892)M
> 	finish: 0:17:16 speed: 66,752 (48,528) K/sec
>  3: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:5674112 dw:5674048 dr:0 al:0 bm:348 lo:2 pe:2323 ua:2 ap:0
> oos:83712960
> 	[>...................] sync'ed:  6.4% (81750/87292)M
> 	finish: 4:09:08 speed: 5,280 (8,328) K/sec
>  4: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:36427744 dw:36425376 dr:0 al:0 bm:2228 lo:74 pe:1193 ua:74
> ap:0 oos:29073760
> 	[==========>.........] sync'ed: 55.7% (28392/63964)M
> 	finish: 0:05:56 speed: 81,280 (53,488) K/sec
>  5: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:32996256 dw:32995712 dr:0 al:0 bm:2018 lo:17 pe:1381 ua:17
> ap:0 oos:32958080
> 	[=========>..........] sync'ed: 50.1% (32185/64408)M
> 	finish: 0:07:27 speed: 73,248 (48,448) K/sec
>  6: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:32899328 dw:32898784 dr:0 al:0 bm:2011 lo:17 pe:1351 ua:17
> ap:0 oos:42127648
> 	[=======>............] sync'ed: 43.9% (41140/73268)M
> 	finish: 0:09:48 speed: 71,520 (48,308) K/sec
>  7: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
>     ns:0 nr:5655712 dw:5654816 dr:0 al:0 bm:346 lo:28 pe:2398 
> ua:28 ap:0
> oos:84911840
> 	[>...................] sync'ed:  6.3% (82921/88444)M
> 	finish: 14:44:29 speed: 1,536 (8,300) K/sec
> 
> Odd ?
> 
> --------------------------------------------------------------
> -----------------
> 
> This email may contain legally privileged and/or confidential 
> information. It is solely for and is confidential for use by 
> the addressee. Unauthorised recipients must preserve, observe 
> and respect this confidentiality. If you have received it in 
> error please notify us and delete it from your computer. Do 
> not discuss, distribute or otherwise copy it. 
> 
> Unless expressly stated to the contrary this e-mail is not 
> intended to, and shall not, have any contractually binding 
> effect on the Company and its clients. We accept no liability 
> for any reliance placed on this e-mail other than to the 
> intended recipient. If the content is not about the business 
> of this Company or its clients then the message is neither 
> from nor sanctioned by the Company. 
> 
> We accept no liability or responsibility for any changes made 
> to this e-mail after it was sent or any viruses transmitted 
> through this e-mail or any attachment. It is your 
> responsibility to satisfy yourself that this e-mail or any 
> attachment is free from viruses and can be opened without 
> harm to your systems.
> 
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
> 



More information about the drbd-user mailing list