[DRBD-user] Definitive answer on >4TB DRBD volumes

Lars Ellenberg lars.ellenberg at linbit.com
Tue Jul 22 12:01:48 CEST 2008

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

On Tue, Jul 22, 2008 at 05:03:23PM +0800, Patrick Coleman wrote:
> Hi,
> I've googled around for a while, and I can't find anything definitive
> for or against - what is the maximum volume size supported by DRBD?
> I'm running an 11TB DRBD 8.2.6 volume between two nodes, connected by
> 10GE. I've hit some odd issues (OOPSes, continually resyncing data)
> and I'd like to eliminate the volume size as a cause of the issue.

DRBD 8.0.x, 8.2.6:
 32bit kernel:
	4 TB hard limit per device
	you can have several of them, but you probably run into some
        other limit pretty fast.
 64bit kernel:
	4 TB "supported".
	(unsupported theoretically) 16 TB hard limit per device,
	you can have several of them, but you probably run into some
        other limit pretty fast.

	you need 32 MB in core RAM per TB storage,
	which is pinned by DRBDs bitmap.
	that is 512 MB RAM for 16 TB total storage,
	          1 GB RAM for 32 TB total storage.
	and yes, that bitmap memory just sits there, pinned,
	doing nothing during normal operation. sorry for that.

	the difference between "supported" 4 TB
	and the (unsupported) 16TB is that we have had several reports
	of instability when going beyond the 4 TB (e.g. it appears to
	work on a freshly booted box, but fails if you try again on a
	busy box after some uptime).  and that we don't have too much
	experience with that large devices on "plain" DRBD ourselves,
	since we tend to use "DRBD+" then.

  both 32bit and 64bit
  (we recommend 64bit for large storage anyways).
	16 TB per device (currently)
	you can have several of them, you should not hit any other limit
	but installed physical ram.
	you need a support contract with LINBIT for DRBD+.

we plan to increase the limit on DRBD+ up to 128 TB (or even
"arbitrary") per device "soonish".
once that is done, the current DRBD+ code will be migrated,
which would make 16 TB per device supported in 64bit "plain"
DRBD 8.2.y, where y is several steps larger than 6.

Did that help?

: Lars Ellenberg                           http://www.linbit.com :
: DRBD/HA support and consulting             sales at linbit.com :
: LINBIT Information Technologies GmbH      Tel +43-1-8178292-0  :
: Vivenotgasse 48, A-1120 Vienna/Europe     Fax +43-1-8178292-82 :
please don't Cc me, but send to list -- I'm subscribed

More information about the drbd-user mailing list