Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
On 25/09/17 22:02, Eric Robinson wrote: > Problem: > > > Under high write load, DRBD exhibits data corruption. In repeated tests > over a month-long period, file corruption occurred after 700-900 GB of > data had been written to the DRBD volume. > > Testing Platform: > > 2 x Dell PowerEdge R610 servers > > 32GB RAM > > 6 x Samsung SSD 840 Pro 512GB (latest firmware) > > Dell H200 JBOD Controller > > SUSE Linux Enterprise Server 12 SP2 (kernel 4.4.74-92.32) > > Gigabit network, 900 Mbps throughput, < 1ms latency, 0 packet loss > > Initial Setup: > > Create 2 RAID-0 software arrays using either mdadm or LVM > > On Array 1: sda5 through sdf5, create DRBD replicated > volume (drbd0) with an ext4 filesystem > > On Array 2: sda6 through sdf6, create LVM logical > volume with an ext4 filesystem > > Procedure: > > Download and build the TrimTester SSD burn-in and TRIM > verification tool from Algolia (https://github.com/algolia/trimtester). > Run TrimTester against the filesystem on drbd0, wait > for corruption to occur > > Run TrimTester against the non-drbd backed filesystem, > wait for corruption to occur > > Results: > > In multiple tests over a period of a month, TrimTester would report file > corruption when run against the DRBD volume after 700-900 GB of data had > been written. The error would usually appear within an hour or two. > However, when running it against the non-DRBD volume on the same > physical drives, no corruption would occur. We could let the burn-in run > for 15+ hours and write 20+ TB of data without a problem. Results were > the same with DRBD 8.4 and 9.0. We also tried disabling the TRIM-testing > part of TrimTester and using it as a simple burn-in tool, just to make > sure that SSD TRIM was not a factor. > > Conclusion: > > We are aware of some controversy surrounding the Samsung SSD 8XX series > drives; however, the issues related to that controversy were resolved > and no longer exist as of kernel 4.2. The 840 Pro drives are confirmed > to support RZAT. Also, the data corruption would only occur when writing > through the DRBD layer. It never occurred when bypassing the DRBD layer > and writing directly to the drives, so we must conclude that DRBD has a > data corruption bug under high write load. However, we would be more > than happy to be proved wrong. I think the conclusion you've arrived at is not quite accurate. It could be described more accurately as a possible data corruption issue specific to drbd and/or the kernel, and trim commands as issued by the TrimTester software. It appears TrimTester was written to debug a very specific SSD trim issue when accessing SSDs directly. It would be necessary to look at exactly what TrimTester is doing to determine if this is a real bug or some anomaly due to what exactly the software is doing. If there is an issue it is possible it could be either in in drbd or the kernel. Just because the issue is present when drbd is used does not automatically exclude the problem being in kernel, it may be an issue that only arises in the kernel when drbd is used. What if you would try with a newer kernel such as 4.9, would the same corruption occur? Also, would the specific conditions created by TrimTester ever be reproduced in a real world scenario, or is it something that will only ever show up when running TrimTester?