Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Tue, Jul 22, 2008 at 12:01:48PM +0200, Lars Ellenberg wrote: > On Tue, Jul 22, 2008 at 05:03:23PM +0800, Patrick Coleman wrote: > > Hi, > > > > I've googled around for a while, and I can't find anything definitive > > for or against - what is the maximum volume size supported by DRBD? > > > > I'm running an 11TB DRBD 8.2.6 volume between two nodes, connected by > > 10GE. I've hit some odd issues (OOPSes, continually resyncing data) > > and I'd like to eliminate the volume size as a cause of the issue. > > > DRBD 8.0.x, 8.2.6: > 32bit kernel: > 4 TB hard limit per device > you can have several of them, but you probably run into some > other limit pretty fast. > 64bit kernel: > 4 TB "supported". > (unsupported theoretically) 16 TB hard limit per device, I just tried that myself out of curiosity. it works up to "only" 8 TB per device on 64bit kernel. anything bigger will oops sooner or later. again: maximum 8 TB per device -- or crash is imminent. therefore we will restrict drbd-8.2.7 (8.0.13) to refuse anything bigger than that. > you can have several of them, but you probably run into some > other limit pretty fast. > > you need 32 MB in core RAM per TB storage, > which is pinned by DRBDs bitmap. > that is 512 MB RAM for 16 TB total storage, > 1 GB RAM for 32 TB total storage. > and yes, that bitmap memory just sits there, pinned, > doing nothing during normal operation. sorry for that. as mentioned in the other mail, read that "total storage" as: 8 devices @ 512 GB --> 128 MB RAM usage for drbd bitmap 1 device @ 4 TB --> 128 MB RAM usage for drbd bitmap > the difference between "supported" 4 TB > and the (unsupported) 16TB is that we have had several reports > of instability when going beyond the 4 TB (e.g. it appears to > work on a freshly booted box, but fails if you try again on a > busy box after some uptime). and that we don't have too much > experience with that large devices on "plain" DRBD ourselves, > since we tend to use "DRBD+" then. > > DRBD+ > both 32bit and 64bit > (we recommend 64bit for large storage anyways). > 16 TB per device (currently) > you can have several of them, you should not hit any other limit > but installed physical ram. > you need a support contract with LINBIT for DRBD+. > > we plan to increase the limit on DRBD+ up to 128 TB (or even > "arbitrary") per device "soonish". > once that is done, the current DRBD+ code will be migrated, > which would make 16 TB per device supported in 64bit "plain" > DRBD 8.2.y, where y is several steps larger than 6. this remains true, however. -- : Lars Ellenberg http://www.linbit.com : : DRBD/HA support and consulting sales at linbit.com : : LINBIT Information Technologies GmbH Tel +43-1-8178292-0 : : Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 : __ please don't Cc me, but send to list -- I'm subscribed