Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
It's a 3ware RAID card.
My drbd.conf:
#
# please have a a look at the example configuration file in
# /usr/share/doc/drbd82/drbd.conf
#
common { syncer { rate 100M; al-extents 257; } }
resource r0 {
protocol C;
handlers { pri-on-incon-degr "halt -f"; }
disk { on-io-error detach; }
net { cram-hmac-alg "sha1"; shared-secret "lrtelco"; }
startup { degr-wfc-timeout 15; wfc-timeout 20; }
on zimbra-1 {
address 172.16.0.1:7789;
device /dev/drbd0;
disk /dev/sda6;
meta-disk internal;
}
on zimbra-2 {
address 172.16.0.2:7789;
device /dev/drbd0;
disk /dev/sda6;
meta-disk internal;
}
}
On Fri, Aug 5, 2011 at 11:50 AM, Dan Barker <dbarker at visioncomm.net> wrote:
> You haven’t shared the configuration. There was a ping for more info a
> couple hours ago:****
>
> ** **
>
> “I would suggest looking at the disk hardware. What sort of RAID card are
> you using in the servers? If you are using an HP Smart Array there is some
> tuning to do to get some descent performance out of those as they have a
> rather deep buffer.****
>
> ** **
>
> -Morey”****
>
> ** **
>
> I don’t remember ever seeing your configuration, but if a dd is running
> slow, it’s not a drbd problem.****
>
> ** **
>
> Dan****
>
> ** **
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 12:45 PM
>
> *To:* Dan Barker
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ** **
>
> Ok, then the question becomes, how do I fix the underlying problem?
>
> ****
>
> On Fri, Aug 5, 2011 at 11:42 AM, Dan Barker <dbarker at visioncomm.net>
> wrote:****
>
> Actually, that makes some sense. If the network is way faster than the
> sync, and the dd (which doesn’t even use the network) is the same speed,
> then there is something bad wrong with the underlying device or
> configuration.****
>
> ****
>
> You are not going to be happy until you get that fixed. Then, the dd should
> beat the sync by a good bit.****
>
> ****
>
> You can skip the dd and build your filesystem and use it RIGHT NOW, but
> there is still something wrong and the performance will probably suck. But,
> you can get some work done. ****
>
> ****
>
> If fixing the underlying problem doesn’t wipe your data, then an online
> verify and disconnect/connect (of the secondary node) will get you synced up
> with zero downtime.****
>
> ****
>
> hth****
>
> ****
>
> Dan****
>
> ****
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 12:34 PM****
>
>
> *To:* Dan Barker
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> Ok, looking at this, my guess is that dd and the sync would take roughly
> the same amount of time. Actually, if I am to believe the output of the
> kill command you included, using dd will actually take more time. So I just
> take your previous instructions and omit the dd command to skip it?
>
> I don't care if the sync takes forever, I just want to be able to DO
> something while it's syncing.****
>
> On Fri, Aug 5, 2011 at 11:15 AM, Dan Barker <dbarker at visioncomm.net>
> wrote:****
>
> OK, You can skip the zero, but the devices won’t pass an online verify. It
> won’t hurt anything. All the sectors that need to be synced will be synced.
> The disks are mostly zeros, so an online verify wouldn’t do much, but it’s
> nice to know.****
>
> ****
>
> To find out how long the DD will take, “kill –USR1 taskid” (frightening
> command, but it makes dd tell you how far it’s along and does NOT kill it.
> The 1M blocksize will help a lot. dd defaults to 512).****
>
> ****
>
> If you do skip the dd, the first verify will identify all the non-zero
> sectors, and they’ll sync up 100’s of times faster than a full sync.****
>
> ****
>
> Good Luck!****
>
> ****
>
> Dan****
>
> ****
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 12:10 PM****
>
>
> *To:* Dan Barker
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> Roger that. It's running, but they're 1.8TB a piece so it'll take a
> while. Just wanted to let you know, no need for it to go to the list.
> Thanks for the help thus far, it's been difficult to deal with this and I
> have to get it running ASAP.****
>
> On Fri, Aug 5, 2011 at 11:06 AM, Dan Barker <dbarker at visioncomm.net>
> wrote:****
>
> That’s not the backing device. The backing device is something like
> /dev/sdb. The drbd device is called device in your config. The backing
> device is called disk.****
>
> ****
>
> It should not be mounted.****
>
> ****
>
> Dan****
>
> ****
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 11:44 AM****
>
>
> *To:* Dan Barker
> *Cc:* drbd-user at lists.linbit.com
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> After I down all, it won't let me issue the dd command citing:
>
> dd: opening `/dev/drbd0': Read-only file system****
>
> On Fri, Aug 5, 2011 at 10:37 AM, Dan Barker <dbarker at visioncomm.net>
> wrote:****
>
> We’ve been off-list for a few messages, but this is now interesting enough
> to be public. I apologize for the top-posting. Please read backwards. Dan*
> ***
>
> ****
>
> Well, to be certain, I’d dd the disks to zeros individually, and then start
> with them sync’d. ****
>
> ****
>
> drbdadm down all****
>
> dd if=/dev/zero of=/drbdbackingdevice bs=1M oflag=direct****
>
> ****
>
> on both sides.****
>
> ****
>
> New Blank Disk:****
>
> ===============****
>
> #On both nodes, initialize meta data and configure the device.****
>
> drbdadm -- --force create-md r0****
>
> ****
>
> #They need to do the initial handshake, so they know their sizes.****
>
> drbdadm up r0****
>
> ****
>
> #They are now Connected Secondary/Secondary Inconsistent/Inconsistent.
> Generate a new current-uuid and clear the dirty bitmap.****
>
> drbdadm -- --clear-bitmap new-current-uuid r0****
>
> ****
>
> #They are now Connected Secondary/Secondary UpToDate/UpToDate.****
>
> drbdadm primary r0****
>
> ****
>
> Now, recreate your empty ext3 file system and you are in sync.****
>
> ****
>
> Dan****
>
> ****
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 11:26 AM
> *To:* Dan Barker****
>
>
> *Cc:* drbd-user at lists.linbit.com
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> Well we're the perfect supplements for each other because, as you can see,
> I don't know DRBD from a hole in the wall.
>
> Yes, it's a brand new resource. I don't think I would say it's full of
> zeros, as it has a clean ext3 file system on it.
>
> Yes, I would LOVE to skip the sync, I've been dealing with this for weeks
> now and right when I was about to go live, I tested the failover and it
> didn't work because of a service heartbeat wanted to start wasn't going all
> that well. While troubleshooting, I lost my 100% perfectly live server and
> have to start from scratch. Problem is I only have two days to do it and
> the thing has to sync 1.8TB at 12MB/s. I have no idea where the bottleneck
> could be. Two servers, a cable connecting eth1 to eth1, both are
> auto-negotiated at 1gbps on their own /30 subnet. The only thing there
> would be garbage NIC cards, possible, but not probable, or the cable, more
> likely, but I've never had an issue with it until now. Freshly constructed
> servers, too.
>
> I tried the drbdsetup /dev/drbd0 syncer -r 120M, been running like that for
> about 5 minutes now and it hasn't changed at all.****
>
> On Fri, Aug 5, 2011 at 10:13 AM, Dan Barker <dbarker at visioncomm.net>
> wrote:****
>
> Is this a brand new resource? Why are you doing a full sync? If it’s brand
> new (full of zeros), you can skip the sync. Instructions upon request.****
>
> ****
>
> Btw, I don’t know why you are getting 12% of your requested syncer rate.
> I’m not a hot-shot linux performace analyzer, but there is a bottleneck
> somewhere. I get 25M routinely here on GB nics. I have my Syncer set to 25M.
> It drops to about 14M (each) if 2 are syncing.****
>
> ****
>
> To change sync rate without stop/start drbd: drbdsetup /dev/drbd1 syncer -r
> 120M****
>
> ****
>
> AL Extents seems a bit low. I use 1801 (big prime number that felt about
> right).****
>
> ****
>
> Dan****
>
> ****
>
> ****
>
> ****
>
> ****
>
> *From:* Matt Baer [mailto:mbaer at lrnet1.com]
> *Sent:* Friday, August 05, 2011 11:03 AM
> *To:* Dan Barker****
>
>
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> Ok, revised /etc/drbd.conf and restarted DRBD with the following****
>
> common { syncer { rate 100M; al-extents 257; } }****
>
>
> And I'm getting from /proc/drbd:****
>
> GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by
> buildsvn at c5-x8664-build, 2008-10-03 11:30:17
> 0: cs:SyncTarget st:Secondary/Primary ds:Inconsistent/UpToDate C r---
> ns:0 nr:281056 dw:272864 dr:0 al:0 bm:16 lo:257 pe:1969 ua:256 ap:0
> oos:1308428488
> [>....................] sync'ed: 0.1% (1277762/1278028)M
> finish: 25:57:39 speed: 13,904 (12,400) K/sec****
>
> And I only have one resource, r0. All it's syncing right now is the post
> mkfs.ext3 /dev/drbd0****
>
> On Fri, Aug 5, 2011 at 9:51 AM, Dan Barker <dbarker at visioncomm.net> wrote:
> ****
>
> You are still mixing megabits and megabytes. Your 1000 megabit pipe won’t
> take a 600 megabyte stream, or a 150 megabyte stream. The maximum is about
> 125 MBps.****
>
> ****
>
> DRBD talks (and is documented to talk) bytes. Most everyone else talks
> bits.****
>
> ****
>
> You don’t mention the speed you are getting.****
>
> ****
>
> Also, if you have 3 resources syncing, each will try for the syncer limit.
> So, to use 50% of your capacity to sync 3 resources, you’d specify the rate
> as 21M. Note: you can change the rate on the fly, during a sync.****
>
> ****
>
> Dan ****
>
> ****
>
> *From:* drbd-user-bounces at lists.linbit.com [mailto:
> drbd-user-bounces at lists.linbit.com] *On Behalf Of *Matt Baer
> *Sent:* Friday, August 05, 2011 8:51 AM
> *To:* Caspar Smit
> *Cc:* drbd-user at lists.linbit.com
> *Subject:* Re: [DRBD-user] DRBD won't take 1G syncer rate****
>
> ****
>
> I was playing with the settings yesterday and it let me set it at 600M and
> it didn't make a difference in the sync speed at all. I then tried it with
> your suggestion, dropped it to 150M just to be safe. Still no difference.
> I wonder what the deal is. Could it be that this is the initial sync?****
>
> ****
>
> On Aug 5, 2011 1:29 AM, "Caspar Smit" <c.smit at truebit.nl> wrote:
> > Hi Matt,
> >
> > 1000M means 1000 Mb/s NOT 1000mbps. To reach 1000M you should have at
> least
> > one (probably two) 10gbit interface(s). Since you have two 1gbit
> interfaces
> > (bonded with balance-rr?) a value between 100M and around 170M would be
> more
> > appropiate.
> >
> > Kind regards,
> > Caspar
> > Op 5 aug. 2011 08:21 schreef "Matt Baer" <mbaer at lrnet1.com> het
> volgende:
> >> When setting the syncer rate in drbd.conf to 1G, it won't start, citing
> > that
> >> 1G is invalid. Get the same thing with 1000M. Any clue as to why? It
> >> explicitly states that <mbaer at lrnet1.com>1G is acceptable in the docs.
> > I've
> >> triple checked and both interfaces are auto-negotiated at 1000mbps full
> >> duplex.****
>
>
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
>
>
>
> --
> Thanks,
> Matt Baer
> Network Manager
> Leaf River Telephone Company
> 102 W Second Street
> Leaf River, IL
> 815.738.2211 - Main Office
> 866.738.2211 - Toll Free
> mbaer at lrnet1.com****
>
--
Thanks,
Matt Baer
Network Manager
Leaf River Telephone Company
102 W Second Street
Leaf River, IL
815.738.2211 - Main Office
866.738.2211 - Toll Free
mbaer at lrnet1.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20110805/c64fb66b/attachment.htm>