Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Thu, Mar 06, 2008 at 11:04:22AM -0800, Tyler Seaton wrote: > Thanks for your response Lars. > > With some outside assistance we were able to single out the issue > we encountered. > > The problem stemmed from the "secondary" machine not being properly > provisioned. Looking at the two partition tables one can clearly see that there > was a slight size difference between the partitions on the primary and the > partitions on the secondary. This left the partition for /dev/drbd2 on the > secondary machine (nfs2 in the config) *smaller* than the /dev/drbd2 partition > on the primary machine. Upon noticing problematic behavior with our application > we stopped the sync, but shutting down DRBD on the Secondary machine. > > What has caused so much heartache moving forward was that for some reason DRBD > resized the partition for /dev/drbd2: > > Mar 5 14:37:19 nfs2 kernel: drbd2: drbd_bm_resize called with capacity == > 3421310910 unfortunately we had a bug in the "refuse to truncate a live device" code, that triggers in exactly your situation (connecting a smaller device for the first time). this has been fixed with http://git.drbd.org/drbd-8.2.git/?p=drbd-8.2.git;a=commitdiff;h=cd1d57d2710916c332ef57c00545aed9e27d5d03 that fix is in 8.2.5 and 8.0.11 as well. > This led to EXT3 trying to access the data that no longer existed on that > device: > > Mar 5 14:37:34 nfs2 kernel: attempt to access beyond end of device > > If I understand correctly, because the DRBD device beneath LVM was resized, LVM > freaked out. LVM was attempting to map the VG over the DRBD devices, but could > not, because it was the underlying devices were smaller than expected. Hence > this message: > > Mar 5 18:56:26 nfs2 kernel: device-mapper: table: device 147:2 too small for > target > > At this point the gentleman called into to assist us with repair, was able to > dump the MD of the device and resize the "la-size-sect" by hand. yes, that would probably the easiest way to recover from there. > Once the DRBD device was made to match what LVM expected, we were able to bring > up the VG. glad to hear that. -- : Lars Ellenberg http://www.linbit.com : : DRBD/HA support and consulting sales at linbit.com : : LINBIT Information Technologies GmbH Tel +43-1-8178292-0 : : Vivenotgasse 48, A-1120 Vienna/Europe Fax +43-1-8178292-82 : __ please use the "List-Reply" function of your email client.