Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On Mon, Feb 28, 2011 at 11:26:04AM +0100, Walter Haidinger wrote: > Lars, first of all, thanks a _lot_ for this detailed explanation! > I guess this post will help many people in the future. > > Could you add the warning about false positives to the Users Guide at > http://www.drbd.org/users-guide/s-integrity-check.html too? > > > Quoting the DRBD User's Guide: > > Notes on data integrity > > > > This may happen for swap, or for certain append while global sync, or > > truncate/rewrite workloads, and not necessarily poses a problem for the > > integrity of the data. Usually when the initiator of the data transfer > > does this, it already knows that that data block will not be part of an > > on disk data structure, or will be resubmitted with correct data soon > > enough. > > This is new in drbd.conf(5) of 8.3.10. Unfortunately I was only aware > of 8.3.9 which just mentions swap and ReiserFS. Having neither of those > two, I thought I was "free" of false positives... > > > If you want to have DRBD do "end-to-end" data checksums, even if the > > data buffers may be modified while being in flight, and still want it to > > be efficient, sponsor feature development. > > Fair enough! ;-) > > > The Problem: > > http://lwn.net/Articles/429305/ > > http://thread.gmane.org/gmane.linux.kernel/1103571 > > http://thread.gmane.org/gmane.linux.scsi/59259 > > > > And many many more older threads on various ML, > > some of them misleading, some of them mixing > > this issue of in-flight modifications > > with actual (hardware caused) data corruption. > > This and the (outdated) notes of drbd.conf(5) probably got me > on the "wrong" track. It also explains why I was unable to reproduce > any data corruption. > > > Possible Solutions: > [...] > > One last question for clarification, though: > Given the above, even online verify isn't free of false positives, right? > Then some out-of-sync blocks are to be expected. If so, a warning in the > manpage and guide too would probably avoid some gray hair. ;-) Uhm. Well. Online verify does _not_ suffer from _this_ problem. The pages used to read in and compare data are private to DRBD in this case, and we won't modify them while we are calculating checksums. Application IO is locked out while we are reading the data for checksums. So if DRBD Online Verify finds out-of-sync blocks, they have been out of sync at that point in time. They may be unlucky enough to see blocks where data has been modified in flight, and as a result local and remote disk contain differing blocks. Such differences should be very short lived, though, as modification means re-dirtying, and that means the page will be resubmitted by upper layers "soon". Unless these block belong to files that are unlinked before the respective re-dirtied page is written out again (and thus the re-dirtied page is simply discarded, before being re-submitted). Conclusion: out-of-sync blocks found by online verify, or software raid1 "resilvering", any similar procedures, do not necessarily mean broken hardware or memory corruption. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com