Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi, Thanks for the quick reply. >Red Hat cluster suite? If so, LACP isn't supported (only Active/Passive is for redundancy). This is aside your question though, of course. I am using RHEL Cluster Suite with minimal configs (node list and fencing only). I have my NICs in bonding mode 4. I am using IPMI fencing (on separate 1GbE NIC). I am using LACP for redundancy not any performance boost. Can you please explain how/why bonding mode makes a difference for RHEL CS? >I'd suggest putting a delay in the second node's fence call. That way, in a true split brain, the primary will have a good head start in calling the fence against the backup node. However, delay to recovery when the primary really does fail will grow by the delay amount. This is my first time using primary/primary, GFS2, and RHEL CS, can you please explain in more detail how and where to do this? Are you talking about DRBD's fencing system, or RHEL CS fencing system, etc? Can DRBD handle this sort of fencing in the case of SB instead of relying on RHEL CS? Also, my nodes are round-robin multipathing. Won't adding a fence delay lead to data corruption? > There is overhead because of the distributed nature of clustered storage. However, I can't say where/why your latency is coming from so I don't have much to recommend at this time. > If you create a simple DRBD resource and test, what is the overhead relative to the bare drives underneath? How does that change when you add simple GFS2? How about if you used CLVMd as a (test) alternative? If the latency is fairly close between GFS2 and clvmd, it's possibly DLM overhead. I've done the following DD tests: 1. Non-replicated DRBD volume with no FS 2. Replicated DRBD volume with no FS 3. Replicated DRBD volume with GFS2 mounted locally 4. Replicated DRBD volume with GFS2 mounted over GNBD 5. Replicated DRBD volume with GFS2 mounted over iSCSI (IET) Results of #4 and #5 are dismal compared to #1,2, and 3. I would think that DLM would apply even to locally mounted GFS2 as I specified lock_dlm when creating FS. Thanks, Michael -----Original Message----- From: Digimer [mailto:linux at alteeve.com] Sent: Wednesday, October 12, 2011 11:26 AM To: Kushnir, Michael (NIH/NLM/LHC) [C] Cc: drbd-user at lists.linbit.com Subject: Re: [DRBD-user] A few configuration questions specific to RHEL 5 primary/primary GFS2 setup On 10/12/2011 11:17 AM, Kushnir, Michael (NIH/NLM/LHC) [C] wrote: > Good morning, > > I am running a two node DRBD 8.3.8 primary/primary cluster with GFS2 on RHEL 5.7. GFS2 is exported via GNBD. My underlying hardware is 2 x Dell C2100 server with LSI 9260-8i RAID controllers. RAID set is RAID10 with 10 x 1TB SATA 7.2k disks. I use 1MB stripe size as well as read and write caching. NIC is dual-port Myri 10GbE (10G-PCIE2-8B2-S2) connected to Cisco M4900. Ports are in LACP group. > My questions: > > 1. In the case of a 2-primary split brain (switch hiccup, etc), I would like server #1 to always remain primary and server #2 to always shut down. I would like this behavior because server #2 can't become secondary because GNBD is not going to release it. What is the best way to accomplish this? I'd suggest putting a delay in the second node's fence call. That way, in a true split brain, the primary will have a good head start in calling the fence against the backup node. However, delay to recovery when the primary really does fail will grow by the delay amount. > 2. I've tried the deadline queue manager as well as CFQ. I've noticed no difference. Can you please elaborate on why deadline is better, and how can I measure any performance difference between the two? I've not used GNDB, so I dare not speculate. > 3. It seems that GNBD is the biggest source of latency in my system. It decreases IOPS by over ~50% (based on DD tests compared to the same DRBD based GFS2 mounted locally). I've also tried Enterprise iSCSI target as an alternative and the results were not much better. The latency on my LAN is ~0.22ms. Can you offer any tuning tips? > > Thanks, > Michael There is overhead because of the distributed nature of clustered storage. However, I can't say where/why your latency is coming from so I don't have much to recommend at this time. If you create a simple DRBD resource and test, what is the overhead relative to the bare drives underneath? How does that change when you add simple GFS2? How about if you used CLVMd as a (test) alternative? If the latency is fairly close between GFS2 and clvmd, it's possibly DLM overhead. -- Digimer E-Mail: digimer at alteeve.com Freenode handle: digimer Papers and Projects: http://alteeve.com Node Assassin: http://nodeassassin.org "At what point did we forget that the Space Shuttle was, essentially, a program that strapped human beings to an explosion and tried to stab through the sky with fire and math?"