[DRBD-user] device size limit <= 2TB for DRBD disk on DRBD <= 0.7.3

Ian Samuel ian at MrZesty.net
Tue Aug 31 22:36:20 CEST 2004

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


One more question - is the 2TB limit for a single drbd device, or is that the 
total for the server?


On August 30, 2004 07:48 pm, Lars Ellenberg wrote:
> / 2004-08-30 16:57:54 -0400
> \ Ian Samuel:
> > Hello,
> >
> > I tried today to set up DRBD over a software-striped disk set (2 x 1.25
> > TB) for a total DRBD drive of 2.5TB.
> >
> > Unfortunatley, although the source code for v0.7.2 mentions a maximum
> > size of 3.8TB - the size reported back by DRBD as only 310 GB instead.
> >
> > I can mount and use the /dev/md5 device with reiserfs (I have enabled
> > >2TB filesystem support in the kernel), it is only when I try to layer
> > DRBD over top that I have a problem with this size of device.
> >
> > I am using another 50GB drbd parition on the same systems and that works
> > fine, and using only 1 disk (1.25GB) also works fine.
> >
> > Kernel is, with DRBD 0.7.2 on Debian sarge.
> Unfortunately I have to admit that we have several pieces in the code
> were we still use "unsigned long" as size, sometimes in sectors and
> sometimes in kilo byte.
> So the mentioned 3.8 TB are a limit by choice of maximum bitmap size.
> the actual limit due to code leftovers from previous times seems to be
> much lower. My current tests (and a quick glance at the code) suggest
> that you can use up to 1.8TB safely, probably just below 2TB, seemingly
> even just below 2.2TB, if meta data is internal (not recommended at that
> size because of seektime for meta data updates) and thus the _effective_
> (user visible) size is back at < 2TB ...
> FYI:
>   ,--- just seen in my test setup, working:
>   |drbd0: Creating state block
>   |drbd0: resync bitmap: bits=503316736 words=15728648
>   |drbd0: size = 2013266944 KB
>   |drbd0: Assuming that all blocks are out of sync (aka FullSync)
>   |drbd0: 2013266944 KB now marked out-of-sync by on disk bit-map.
>   `---
>   this is 0x78000400 kb. added an other 0x0800000 kb,
>   which would end up as 2TB + 1MB, and it just blew up :-(
> Sorry.
> We should at least refuse to setup devices of that size until
> we sort out these issues. I can even imaging weird OOM conditions or
> oopses related to too large devices (if you happen to hit some special
> number of sectors ...)
> > Is it unrealistic to use a single DRBD device this large?   I am hoping
> > it is a simple 32-bit int overflow.
> No, unfortunately it is not that simple. Several unsigned longs need to
> be converted to sector_t, some of them to loff_t; several calculations
> and far dependencies have to be reviewed, and mached up again. General
> code cleanup of leftovers of former times has to follow, and maybe we
> need to forcefully break all 2.4. compatibility.
> We'll think it over and let you know.
> Meanwhile, please use devices of less than 2TB to be on the safe side.
> 	Lars Ellenberg
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

Ian Samuel, B.Sc.
Sr. Linux System Administrator

More information about the drbd-user mailing list