Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Lars,
I think it's time to end this thread since my problems of system locking
are nothing to do with DRBD.
I took your advise and replaced one of my file systems from reiserfs to
ext3. During the process of copying ~100GB of data off and back onto
the partition I took some crude measurements:
+--------------------------------+----------+----------+-----------+------+--------+
| Operation | From | To | Rate MB/s |
Load | iowait |
+--------------------------------+----------+----------+-----------+------+--------+
| Copy before change file system | reiserfs | reiserfs | 20 | 6
| 15% |
| Start copy back after change | reiserfs | ext3 | 25 | 4
| 25% |
| After restart of DRBD | reiserfs | ext3 | 25 | 4
| 20% |
| Disabled cciss acceleration | reiserfs | ext3 | 7 | 3.5
| 20% |
| Enable cciss acceleration | reiserfs | ext3 | 25 | 4
| 20% |
| After reboot of both servers | reiserfs | ext3 | 43 | 3
| 22% |
+--------------------------------+----------+----------+-----------+------+--------+
Therefore:
- ext3 seems a little better than resierfs, by about 20%.
- Disable acceleration makes big difference. Which is odd, a
continuous write of many gig should not be effected.
- Restart DRBD has no effect, nothing wrong there :)
- Huge effect when rebooting my servers.
The only time I got theoretical max 100MB/sec (limited by gigabit
Ethernet) was after write to disk, when DRBD was flushing it's buffers.
Which is inconsistent with disk IO which bonnie+ has measured at
250MB/sec.
Something is holding up my data, but it's not DRBD :)
Thanks for all the advise, I would not have got this far without it.
Ben
Ben Clewett wrote:
>
>
> (Second send, sorry, I used the wrong email again. :)
>
> Lars,
>
> Thanks for understanding my problem and finding the links.
>
> After reading the links, it seems at least very likely that my problem
> is the file system. I have all the symptoms: a single large reiserfs
> file system experiencing high bandwidth, and multiple CPU's.
>
> The kernel patch to correctly order journal commits is in my 2.6.18.8
> kernel. Or at least with the source code SUSE supply with it :)
>
> Have any other members experienced falling DRBD performance on similar
> setup?
>
> Changing a file system is simple. Copy data off, mkfs, back on. As
> long as I can change the file system on a Primary Connected DRBD
> resource without issue?
>
> What do you all recommend for a MySql system, ext2 or ext3? Or any
> others SUSE have given me:
>
> mkfs.cramfs mkfs.ext3 mkfs.minix mkfs.ntfs mkfs.vfat
> mkfs.bfs mkfs.ext2 mkfs.jfs mkfs.msdos mkfs.xfs
>
> ?
>
> I shall let you know...
>
> Ben
>
>
> Lars Ellenberg wrote:
>> On Tue, Jan 08, 2008 at 01:09:36PM +0000, Ben Clewett wrote:
>>>
>>> Hi Lars,
>>>
>>> Lars Ellenberg wrote:
>>>> grrr. stripp of one of the nots, please,
>>>> either "not likely" or "likely not".
>>>> anyways, it is NOT a problem in drbd.
>>>> but you knew what I meant, right?
>>> Many thanks, lots of good information to work with.
>>>
>>> I knew what you meant. I don't believe it is a problem with DRBD.
>>> But it manifests through DRBD. There is a know problem with my NIC,
>>> even at the latest driver:
>>>
>>> http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1199792167647+28353475&threadId=1186722
>>>
>>>
>>> However the suggested fix here doesn't make a difference:
>>>
>>> # ethtool -K eth2 tso off
>>>
>>> I have a known problem which does not respond to the known fix!
>>>
>>> But I still suspect the NIC's... Tonight I am going to move them all
>>> around to see if the problem follows the card.
>>>
>>> --------
>>>
>>> I also know that if I reboot the servers, I'll get a few days of
>>> grace before the locking returns. I don't think that is DRBD related?
>>>
>>> The other think I note is that slowing data input by introducing
>>> small 1 second gaps every half minute or so, actually increases the
>>> rate at which data is accepted. What could cause that affect?
>>
>> don't start me speculating on that :)
>>
>>> --------
>>>
>>> The analysis of /proc/drbd shows data to be moving. But I can see a
>>> pattern:
>>>
>>> With my symmetrical system, on both servers:
>>> # watch -n 0.1 -d cat /proc/drbd
>>>
>>> The DRBD resource which is not 'locked' is moving data continuously.
>>> lo:1 is the normal condition, ns and dw are increasing linearly.
>>>
>>> However the DRBD disk which is locked is only sporadically moved
>>> data. It can spend up to ~2 seconds stuck on:
>>> Primary: bm:0 lo:n pe:0 ua:0 ap:n
>>> Secondary: bm:0 lo:0 pe:0 ua:0 ap:0
>>> Where n is some number between 1 and ~ 50.
>>
>> in that case, it is the _local io subsystem_ on the Primary,
>> that does not return the requests.
>> ap: application bios comming from upper layers (file system),
>> but not yet reported as completed to upper layers
>> lo: submitted to local disk, but not completed from there yet.
>> pe: sent to peer, but not yet acknowledged (pending peer)
>>
>> we can only report completed to upper layers (decrease ap) when
>> both corresponding lo and pe requests have completed. since your pe is
>> 0, but ap and lo are n, your local disk does not respond (quickly
>> enough).
>>
>>> It looks like the traffic flow from one server is blocking the
>>> traffic flow from the another, like it has a higher priority, or is
>>> being traffic shaped?. This might also explain by introducing 1
>>> second gaps helps get data moving, it stops one server letting the
>>> other work. But it might also be coincidence...
>>>
>>> Can you think of anything in the kernel which might effect this?
>>
>> file system waits for these n requests to complete before it sends more,
>> maybe write after write dependency on some journal commit.
>>
>>> ----
>>>
>>> Another idea if it the case my resources are clashing for bandwidth.
>>> I have two resources sharing a gigabit connection. My rate = 100M.
>>> If I set this to 50M, would this ensure that each resource could not
>>> use more than 50% of the connection?
>>
>> the sync rate is only relevant during _resynchronization_,
>> and is a throttle for the resync process to leave bandwidth for the
>> normal, ongoing, _replication_ processs.
>> as long as drbd is cs:Connected, the "syncer rate" is irrelevant.
>>
>>> ----
>>>
>>> You also asked about mounting:
>>>
>>> # cat /proc/mounts | grep drbd
>>>
>>> /dev/drbd1 /dbms-07-02 reiserfs rw 0 0
>>>
>>> Do you know of any options which might help?
>>
>> google for: reiserfs stall
>>
>> there repeatedly have been issues.
>>
>> since you are using suse 10.2, suse kernel 2.6.18-somthing,
>> have a look at
>> http://linux.wordpress.com/2006/09/27/suse-102-ditching-reiserfs-as-it-default-fs/
>>
>> where Jeff Mahony is cited as
>>
>> ReiserFS has serious performance problems with extended attributes and
>> ACLs. (Yes, this one is my own fault, see numerous flamewars on lkml and
>> reiserfs-list for my opinion on this.) xattrs are backed by normal files
>> rooted in a hidden directory structure. This is bad for performance and
>> in rare cases is deadlock prone due to lock inversions between pdflush
>> and the xattr code.
>>
>> also you could try to find out whether this upstream commit
>>
>> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=a3172027148120b8f8797cbecc7d0a0b215736a1
>>
>> [Fix reiserfs latencies caused by data=ordered]
>> is included in your kernel or not.
>> [data=ordered is default, and recommended].
>>
>> also, not this is but speculation,
>> but it may well be that even the latest "fixes" don't fix the cause
>> (some race
>> condition), but make the symptom go away by reducing the likelyhood of
>> the race
>> to trigger. and then introduce drbd replication, and your broken nics,
>> which
>> changes the latency and thus io-timing a lot, and suddenly the same
>> (or some different)
>> race triggers again?
>>
>> is changing file system type an option?
>> or changing kernel version?
>> would you be able to try this on a test cluster?
>>
>> anyways, this starts to become off topic for a _drbd_ list...
>>
>
>
>
>
> *************************************************************************
> This e-mail is confidential and may be legally privileged. It is intended
> solely for the use of the individual(s) to whom it is addressed. Any
> content in this message is not necessarily a view or statement from Road
> Tech Computer Systems Limited but is that of the individual sender. If
> you are not the intended recipient, be advised that you have received
> this e-mail in error and that any use, dissemination, forwarding,
> printing, or copying of this e-mail is strictly prohibited. We use
> reasonable endeavours to virus scan all e-mails leaving the company but
> no warranty is given that this e-mail and any attachments are virus free.
> You should undertake your own virus checking. The right to monitor e-mail
> communications through our networks is reserved by us
>
> Road Tech Computer Systems Ltd. Shenley Hall, Rectory Lane, Shenley,
> Radlett, Hertfordshire, WD7 9AN. - VAT Registration No GB 449 3582 17
> Registered in England No: 02017435, Registered Address: Charter Court,
> Midland Road, Hemel Hempstead, Hertfordshire, HP2 5GE.
> *************************************************************************
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
*************************************************************************
This e-mail is confidential and may be legally privileged. It is intended
solely for the use of the individual(s) to whom it is addressed. Any
content in this message is not necessarily a view or statement from Road
Tech Computer Systems Limited but is that of the individual sender. If
you are not the intended recipient, be advised that you have received
this e-mail in error and that any use, dissemination, forwarding,
printing, or copying of this e-mail is strictly prohibited. We use
reasonable endeavours to virus scan all e-mails leaving the company but
no warranty is given that this e-mail and any attachments are virus free.
You should undertake your own virus checking. The right to monitor e-mail
communications through our networks is reserved by us
Road Tech Computer Systems Ltd. Shenley Hall, Rectory Lane, Shenley,
Radlett, Hertfordshire, WD7 9AN. - VAT Registration No GB 449 3582 17
Registered in England No: 02017435, Registered Address: Charter Court,
Midland Road, Hemel Hempstead, Hertfordshire, HP2 5GE.
*************************************************************************