Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Whats even odder is that looking at top we suspected CPU contention (the boxes have a single CPU each) so we added another CPU to both boxes. And another pair of RAM sticks. Now - the box has a lovely spread of write I/O. However the total I/O has dropped considerably : Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 11.40 0.00 5.70 0 28 sdb 1598.80 189779.20 0.00 948896 0 dm-0 750.80 24025.60 0.00 120128 0 dm-1 752.00 24064.00 0.00 120320 0 dm-2 756.60 24211.20 0.00 121056 0 dm-3 749.60 23987.20 0.00 119936 0 dm-4 755.00 24160.00 0.00 120800 0 dm-5 753.60 24115.20 0.00 120576 0 dm-6 754.00 24128.00 0.00 120640 0 dm-7 712.20 22790.40 0.00 113952 0 We took out the additional CPU : Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 11.00 0.00 5.50 0 5 sdb 5040.00 0.00 191584.00 0 191584 dm-0 765.00 0.00 24480.00 0 24480 dm-1 766.00 0.00 24512.00 0 24512 dm-2 753.00 0.00 24096.00 0 24096 dm-3 706.00 0.00 22592.00 0 22592 dm-4 763.00 0.00 24416.00 0 24416 dm-5 756.00 0.00 24192.00 0 24192 dm-6 761.00 0.00 24352.00 0 24352 dm-7 716.00 0.00 22912.00 0 22912 We then reverted back to 2 x RAM sticks Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 11.00 0.00 11.10 0 55 sdb 3911.40 0.00 153369.60 0 766848 dm-0 608.40 0.00 19468.80 0 97344 dm-1 610.00 0.00 19520.00 0 97600 dm-2 605.40 0.00 19372.80 0 96864 dm-3 570.60 0.00 18259.20 0 91296 dm-4 610.40 0.00 19532.80 0 97664 dm-5 608.80 0.00 19481.60 0 97408 dm-6 603.80 0.00 19321.60 0 96608 dm-7 575.00 0.00 18400.00 0 92000 what ??!! try as we might, we simply cannot get back to the figures we saw at the start, namely 8000+ tps and 309MB/s I/O. Since I put the output into this email I'm sure we didn't imagine it. Anwers on a postcard pls..... > -----Original Message----- > From: drbd-user-bounces at lists.linbit.com > [mailto:drbd-user-bounces at lists.linbit.com] On Behalf Of Lee Christie > Sent: 03 July 2008 13:25 > To: drbd-user at lists.linbit.com > Subject: [DRBD-user] Odd observation when sync'ing > > Two identical storage boxes, each with a 6 disk RAID10 carrying 8 drbd > resources > > Doing a sync : > > Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn > sda 18.96 0.00 9.48 0 47 > sdb 8097.60 0.00 309429.14 0 1550240 > dm-0 1157.29 0.00 37033.13 0 185536 > dm-1 1518.16 0.00 48581.24 0 243392 > dm-2 1683.83 0.00 53882.63 0 269952 > dm-3 180.04 0.00 5761.28 0 28864 > dm-4 1839.32 0.00 58858.28 0 294880 > dm-5 1572.46 0.00 50318.56 0 252096 > dm-6 1466.87 0.00 46939.72 0 235168 > dm-7 217.96 0.00 6974.85 0 34944 > > Odd how two of the drbd resources are simply not getting a > look in when > it comes to I/O. > > This is backed up by the output from drbd : > > [root at mail-storage-02 network-scripts]# cat /proc/drbd > version: 8.2.6 (api:88/proto:86-88) > GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by > buildsvn at c5-x8664-build, 2008-06-26 19:33:19 > 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:27112032 dw:27111648 dr:0 al:0 bm:1655 lo:13 pe:8777 ua:12 > ap:0 oos:65744672 > [====>...............] sync'ed: 29.3% (64203/90680)M > finish: 0:19:01 speed: 57,472 (39,808) K/sec > 1: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:32451072 dw:32450560 dr:0 al:0 bm:1982 lo:16 pe:1429 ua:16 > ap:0 oos:60848128 > [=====>..............] sync'ed: 34.8% (59422/91112)M > finish: 0:15:50 speed: 63,840 (47,648) K/sec > 2: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:33048608 dw:33047904 dr:0 al:0 bm:2017 lo:22 pe:1363 ua:22 > ap:0 oos:69241504 > [=====>..............] sync'ed: 32.4% (67618/99892)M > finish: 0:17:16 speed: 66,752 (48,528) K/sec > 3: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:5674112 dw:5674048 dr:0 al:0 bm:348 lo:2 pe:2323 ua:2 ap:0 > oos:83712960 > [>...................] sync'ed: 6.4% (81750/87292)M > finish: 4:09:08 speed: 5,280 (8,328) K/sec > 4: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:36427744 dw:36425376 dr:0 al:0 bm:2228 lo:74 pe:1193 ua:74 > ap:0 oos:29073760 > [==========>.........] sync'ed: 55.7% (28392/63964)M > finish: 0:05:56 speed: 81,280 (53,488) K/sec > 5: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:32996256 dw:32995712 dr:0 al:0 bm:2018 lo:17 pe:1381 ua:17 > ap:0 oos:32958080 > [=========>..........] sync'ed: 50.1% (32185/64408)M > finish: 0:07:27 speed: 73,248 (48,448) K/sec > 6: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:32899328 dw:32898784 dr:0 al:0 bm:2011 lo:17 pe:1351 ua:17 > ap:0 oos:42127648 > [=======>............] sync'ed: 43.9% (41140/73268)M > finish: 0:09:48 speed: 71,520 (48,308) K/sec > 7: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r--- > ns:0 nr:5655712 dw:5654816 dr:0 al:0 bm:346 lo:28 pe:2398 > ua:28 ap:0 > oos:84911840 > [>...................] sync'ed: 6.3% (82921/88444)M > finish: 14:44:29 speed: 1,536 (8,300) K/sec > > Odd ? > > -------------------------------------------------------------- > ----------------- > > This email may contain legally privileged and/or confidential > information. It is solely for and is confidential for use by > the addressee. Unauthorised recipients must preserve, observe > and respect this confidentiality. If you have received it in > error please notify us and delete it from your computer. Do > not discuss, distribute or otherwise copy it. > > Unless expressly stated to the contrary this e-mail is not > intended to, and shall not, have any contractually binding > effect on the Company and its clients. We accept no liability > for any reliance placed on this e-mail other than to the > intended recipient. If the content is not about the business > of this Company or its clients then the message is neither > from nor sanctioned by the Company. > > We accept no liability or responsibility for any changes made > to this e-mail after it was sent or any viruses transmitted > through this e-mail or any attachment. It is your > responsibility to satisfy yourself that this e-mail or any > attachment is free from viruses and can be opened without > harm to your systems. > > _______________________________________________ > drbd-user mailing list > drbd-user at lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user >