Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi Lars, On 19/02/15 11:53, Lars Ellenberg wrote: > On Fri, Feb 06, 2015 at 07:45:49PM +0000, Giles Thomas wrote: >> Are we right in thinking that the purpose of using LVM [on the secondary during backups] is purely to >> get a mountable device that you can run the backup from? After all, > *AND* it keeps you from introducing data divergence aka > split-brain, which you would later need to clean up. So just to make sure I understand that correctly -- you're saying it stops us from accidentally writing to the secondary while it's disconnected from the primary. Makes sense, that would obviously be a Bad Thing. [Out of interest, would it be technically feasible for a future version of DRBD to make the disk available on the secondary in a read-only state? I assume it wouldn't be easy (as otherwise it would probably have been done), and I'm not necessarily advocating it as a new feature -- but I am wondering if there are any deep technical barriers that would make it completely impossible.] > >> * drbdadm disconnect the resource on the secondary >> * Use the AWS API to do their kind of snapshot on the underlying >> disks. Wait for these to complete. >> * drbdadm connect the resource secondary >> * Make sure you wait for the primary to have synced fully before doing >> this again! > Should just work. > > You only use DRBD "disconnect" to get "stable data", > which you then can snapshot by other means... An update on this: it looks like it works. As it turns out, AWS's EBS snapshots actually don't even need to complete because they work with a frozen state of the data at the point they're started. So a process that *seems to be* working well for us is: * drbdadm disconnect the resource on the secondary * Use the AWS API to do their kind of snapshot on the underlying disks. Wait for these to *start* -- that is, show up as "pending" on the AWS console. * drbdadm connect the resource on the secondary * Make sure you wait for the primary to have synced fully before doing this again. The reason I say this "seems to be" working is that we're having one problem with the snapshots we take this way. The filesystem is XFS, and when we reconnect the volumes, they cannot be mounted with the project quota flag set. They can be mounted without that flag, and all of the files are there, but if the flag is set, the mount hangs, and running XFS repair just crashes. There are a couple of possibilities there, but it certainly seems to be an XFS thing. Its quota settings aren't journalled, and we suspect that the disconnect means that the filesystem is in an inconsistent state quota-wise when the snapshot is taken. We're considering moving to ext4fs, as that journals its quotas which might make it a better choice for our use case (and in general for backing up volumes while they're live). > Don't forget to excercise *restore* as well. Absolutely! :-) > If you use DRBD internal meta data, it would be backed up, > and restored, as well. Which means a restore would completely confuse > DRBD, to the point where it may require a full resync, or even refuse to > talk to the peer at all ("unrelated data"). > > If you use DRBD external meta data, its even more confusing, > as the meta data would not reflect what is in the data area, > and there would not be a resync of relevant areas, > causing unexpected data divergence. Thanks, that's useful to know. The underlying setup we have is a bunch of EBS volumes RAID-0-ed together into one large volume using mdadm, then LVM on top of that to split it into one smallish metadata volume for DRBD and one large one for the data. So the EBS backup snapshots both the data and the metadata. I suspect in the kind of catastrophic failure we'd need these backups to recover from, we'd wind up restoring the disks from the snapshots and reassembling the RAID array and LVM, but then wiping the metadata volume, connecting the result to a new DRBD primary, and then hooking it up to a secondary which had completely fresh disks (albeit with the whole RAID/LVM thing already set up) -- it takes about 10 hours to do a full sync, and while there would be a risk of a disk failure blowing everything up during that time window, that risk would probably be outweighed by the value of having DRBD having a completely fresh set of metadata. > As long as you do file level backup and restore, > there shold not be an issue. Brilliant, many thanks! All the best, Giles -- Giles Thomas <giles at pythonanywhere.com> PythonAnywhere: Develop and host Python from your browser <https://www.pythonanywhere.com/> A product from PythonAnywhere LLP 17a Clerkenwell Road, London EC1M 5RD, UK VAT No.: GB 893 5643 79 Registered in England and Wales as company number OC378414. Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK