[DRBD-user] Update: Definitive answer on >4TB DRBD volumes

Lars Ellenberg lars.ellenberg at linbit.com
Thu Jul 24 10:34:23 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


On Tue, Jul 22, 2008 at 12:01:48PM +0200, Lars Ellenberg wrote:
> On Tue, Jul 22, 2008 at 05:03:23PM +0800, Patrick Coleman wrote:
> > Hi,
> > 
> > I've googled around for a while, and I can't find anything definitive
> > for or against - what is the maximum volume size supported by DRBD?
> > 
> > I'm running an 11TB DRBD 8.2.6 volume between two nodes, connected by
> > 10GE. I've hit some odd issues (OOPSes, continually resyncing data)
> > and I'd like to eliminate the volume size as a cause of the issue.
> 
> 
> DRBD 8.0.x, 8.2.6:
>  32bit kernel:
> 	4 TB hard limit per device
> 	you can have several of them, but you probably run into some
>         other limit pretty fast.
>  64bit kernel:
> 	4 TB "supported".
> 	(unsupported theoretically) 16 TB hard limit per device,

I just tried that myself out of curiosity.
it works up to "only" 8 TB per device on 64bit kernel.
anything bigger will oops sooner or later.

again: maximum 8 TB per device -- or crash is imminent.

therefore we will restrict drbd-8.2.7 (8.0.13)
to refuse anything bigger than that.

> 	you can have several of them, but you probably run into some
>         other limit pretty fast.
> 
> 	you need 32 MB in core RAM per TB storage,
> 	which is pinned by DRBDs bitmap.
> 	that is 512 MB RAM for 16 TB total storage,
> 	          1 GB RAM for 32 TB total storage.
> 	and yes, that bitmap memory just sits there, pinned,
> 	doing nothing during normal operation. sorry for that.

as mentioned in the other mail, read that "total storage" as:
  8 devices @ 512 GB --> 128 MB RAM usage for drbd bitmap
  1 device  @   4 TB --> 128 MB RAM usage for drbd bitmap

> 	the difference between "supported" 4 TB
> 	and the (unsupported) 16TB is that we have had several reports
> 	of instability when going beyond the 4 TB (e.g. it appears to
> 	work on a freshly booted box, but fails if you try again on a
> 	busy box after some uptime).  and that we don't have too much
> 	experience with that large devices on "plain" DRBD ourselves,
> 	since we tend to use "DRBD+" then.
> 
> DRBD+
>   both 32bit and 64bit
>   (we recommend 64bit for large storage anyways).
> 	16 TB per device (currently)
> 	you can have several of them, you should not hit any other limit
> 	but installed physical ram.
> 	you need a support contract with LINBIT for DRBD+.
> 
> we plan to increase the limit on DRBD+ up to 128 TB (or even
> "arbitrary") per device "soonish".
> once that is done, the current DRBD+ code will be migrated,
> which would make 16 TB per device supported in 64bit "plain"
> DRBD 8.2.y, where y is several steps larger than 6.

this remains true, however.

-- 
: Lars Ellenberg                           http://www.linbit.com :
: DRBD/HA support and consulting             sales at linbit.com :
: LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
: Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
__
please don't Cc me, but send to list -- I'm subscribed



More information about the drbd-user mailing list