Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On 02/17/2012 11:42 AM, Brian O Mahony wrote: > (Apologies for top posting - working off a different laptop as I broke mine this morning, and outlook is a pain) > > Jake/Maurits/Digimer, Thanks for the replies. > > Jake/Digimer: >> What it sounds like you need is LVM's snapshotting capability. DRBD is designed to be a block-level replication tool, not a backup tool per-se. > We need to back up individual folders, some of which are 80+ GB. I don't think a snapshot allows me to do this? LVM snapshotting works like this; You have an LVM LV, with some space left over on the VG sufficient to hold deltas created for the duration of time it takes to copy the data off. So when you want to make a backup, you create a snapshot LV and mount that somewhere. At the moment of the mount, the snapshot is identical to the source LV. As time passes though and data on the source partition changes, the old data is stored on the smaller snapshot partition, allowing you to still see the data on the snapshot mount as it was at the time the snapshot was made, without making the source partition unavailable. Thus, you need to have a rough idea of how long it takes to copy the data off. With this time in mind, you must then estimate how much data will change in that period of time. Add some headroom and you have the size of the snapshot partition you will need. The result of all this is that, for the life of the snapshot partition, you have an unchanging, point-in-time view of the source LV. Meanwhile, the source partition is never offline, frozen or otherwise unavailable. If you need to stop processes to get the data into a backup-able state, then you can simply; Stop your service -> mount the snapshot -> restart the service and you should have no more than a few seconds of downtime. > Digimer: >> As a secondary aside; Why use cp instead of rsync? > Data on the primary server is on a SAN, with about 8 million files, most tiny. The backup dumps this to fast local disks (there is other things on the SAN also, and all our VMs etc get backed up nightly, meaning sometimes disk IO access can be slowed), this is then compressed locally, before being booted off to local storage on the second server. The department in charge of the data, and the backup process are not that well versed in this, and when I suggested rsync and the delta capabilities, they assumed it is going to take longer to check, and never tested it fully. Preferably, I would like them to rsync to the second machine, and then do the tar/gz there. But with managers involved, ive been told to look for another solution, as they "don't have the time to spend days looking at something that is potentially slower" Sounds like you've got too many cooks in the kitchen. No matter what you explore, there is a chance that it won't work. It may well be the case that rsync isn't viable, but I would suggest that it is a much quicker course to investigate than DRBD. This sounds to me like someone read something, got excited and is now hell-bent on a given solution to the exclusion of other options. Don't get me wrong, DRBD is awesome and ridiculously useful, but I don't think it fits this use-case. >> (inducing a split-brain condition, which is generally something you want to avoid > The second machine does nothing except store a copy of this data (why they need 2 6core procs & 128GB memory baffles me tbh!). It actually stores two copies, one which is a "site sync" in the software, and the other being the tar/gz copy. It is source code, so they are quite finicky about the data. > > I will download the DRBD manual, and have a good read through. As I said im a novice with it. I also haven't used LVM snapshotting much, so will investigate that also. > > Once again, thanks for the replies. > > B Happy to help. -- Digimer E-Mail: digimer at alteeve.com Papers and Projects: https://alteeve.com