<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Problem: <o:p></o:p></p>
<p class="MsoNormal"><br>
Under high write load, DRBD exhibits data corruption. In repeated tests over a month-long period, file corruption occurred after 700-900 GB of data had been written to the DRBD volume. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Testing Platform:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2 x Dell PowerEdge R610 servers<o:p></o:p></p>
<p class="MsoNormal">32GB RAM<o:p></o:p></p>
<p class="MsoNormal">6 x Samsung SSD 840 Pro 512GB (latest firmware)<o:p></o:p></p>
<p class="MsoNormal">Dell H200 JBOD Controller<o:p></o:p></p>
<p class="MsoNormal">SUSE Linux Enterprise Server 12 SP2 (kernel 4.4.74-92.32)<o:p></o:p></p>
<p class="MsoNormal">Gigabit network, 900 Mbps throughput, < 1ms latency, 0 packet loss<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Initial Setup:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Create 2 RAID-0 software arrays using either mdadm or LVM<o:p></o:p></p>
<p class="MsoNormal"> On Array 1: sda5 through sdf5, create DRBD replicated volume (drbd0) with an ext4 filesystem<o:p></o:p></p>
<p class="MsoNormal"> On Array 2: sda6 through sdf6, create LVM logical volume with an ext4 filesystem<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Procedure:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Download and build the TrimTester SSD burn-in and TRIM verification tool from Algolia (<a href="https://github.com/algolia/trimtester">https://github.com/algolia/trimtester</a>). <br>
Run TrimTester against the filesystem on drbd0, wait for corruption to occur<o:p></o:p></p>
<p class="MsoNormal"> Run TrimTester against the non-drbd backed filesystem, wait for corruption to occur<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Results:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In multiple tests over a period of a month, TrimTester would report file corruption when run against the DRBD volume after 700-900 GB of data had been written. The error would usually appear within an hour or two. However, when running
it against the non-DRBD volume on the same physical drives, no corruption would occur. We could let the burn-in run for 15+ hours and write 20+ TB of data without a problem. Results were the same with DRBD 8.4 and 9.0. We also tried disabling the TRIM-testing
part of TrimTester and using it as a simple burn-in tool, just to make sure that SSD TRIM was not a factor.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Conclusion:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are aware of some controversy surrounding the Samsung SSD 8XX series drives; however, the issues related to that controversy were resolved and no longer exist as of kernel 4.2. The 840 Pro drives are confirmed to support RZAT. Also,
the data corruption would only occur when writing through the DRBD layer. It never occurred when bypassing the DRBD layer and writing directly to the drives, so we must conclude that DRBD has a data corruption bug under high write load. However, we would be
more than happy to be proved wrong.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Eric Robinson<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="text-indent:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="text-indent:.5in"><o:p> </o:p></p>
<p class="MsoNormal" style="text-indent:.5in"><o:p> </o:p></p>
</div>
</body>
</html>