Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Wed, 28 May 2008, Iustin Pop wrote: > On Wed, May 28, 2008 at 03:36:33PM +0900, Christian Balzer wrote: >>>> And you really want to use a resilient approach on the lowest level, >>>> in >>>> your case RAID5 over the whole disks (with one spare at least given >>>> that >>>> with 80 drives per node you are bound to have disk failures frequently >>>> enough). In fact I'd question the need for having a DRBD mirror of >>>> archive backups in the first place, but that is your call and money. >>>> ^^ >>> >>> Unfortunately, cost is driving most of my decisions and RAID5 adds >>> 10-20% to the total cost. >> Come again? I was suggesting an overhead of 2 drives, which comes to 2.5% >> with 80 drives. Other than that RAID5 is free (md driver) and you sure >> were not holding back with CPU power in your specs (less, but faster >> cores and most likely Opteron instead of Intel would do better in >> this scenario). Of course I have no idea how many SATA (scsi really) >> drives current kernels can handle and how many drives can be part of a >> software RAID. > > RAID5 might be free (as in md driver), but it surely isn't free as in > performance degradation and I think you don't actually want a raid5 of > 79 drives... No, you'd probably want RAID6 if you're going with md. And since your boxen seem like dedicate disk driving machines, the overhead would be irrelevant. You have more than enough CPU to calculate the Reed-Solomon codes for RAID6 in real-time as fast as the bus will accept the data. Your disks will be able to handle several orders of magnitude more throughput than the bus your controllers will be plugged into. In fact - scratch that - the bottleneck will almost certainly be the network device you will be doing mirroring (DRBD) over, even if you are using multiple bonded Gb ethernet NICs. So the overhead of spending a bit of CPU on RAID6 is certainly not going to be what will be holding you back. > Please read the archives of linux-raid as to what is the recommended > raid5 size (as in number of drives); it's definitely below 20. On _anything_ RAID5 generally makes sense up to about 20 disks, because the failure rate will mean you need more redundancy than that. RAID6 should just about cope with 80 disks, especially if you are looking at mirroring the setup with DRBD (effectively giving you RAID61). More to the point, if you are cost conscious as you say, RAID1 with multiuple mirrors is vastly less economical than RAID6 on a setup that big. You could save an awful lot. > The reason is that two disk failure (out of 79) is quite likely, and > then you lose actually the whole 80 drives and you'll have lot of pain > resynching. Resyncing will involve lots of pain anyway. Have you checked how long it takes to write a TB of data?? RAID6 will keep you going with 2 failed drives, and if you do it so that you have a RAID6 stripe of mirrors (RAID16), with each mirror being a DRBD device, it would give you pretty spectacular redundancy, because you would have to lose three complete mirror sets. Gordan