<html><head><base href="x-msg://208/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Helllo Wiebe,<div><br></div><div>I had that also.</div><div>But when i turned down the cache on the storage, and the other vm clients the error was gone.</div><div>So no cache on the VM's and it worked very good. </div><div><br></div><div><br></div><div>marcel</div><div><br></div><div><br><div><div>On 30 mei 2012, at 17:13, Wiebe Cazemier wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); ">Hi, <br><br>I'm testing setting up a Xen DomU with a DRBD storage for easy failover. Most of the time, immediately after booting the DomU, I get an IO error:<br><br><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.153370] EXT3-fs (xvda2): using internal journal</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.277115] ip_tables: (C) 2000-2006 Netfilter Core Team</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.336014] nf_conntrack version 0.5.0 (3899 buckets, 15596 max)</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.515604] init: failsafe main process (397) killed by TERM signal</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801589] blkfront: barrier: write xvda2 op failed</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801597] blkfront: xvda2: barrier or flush: disabled</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801611] end_request: I/O error, dev xvda2, sector 52171168</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801630] end_request: I/O error, dev xvda2, sector 52171168</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801642] Buffer I/O error on device xvda2, logical block 6521396</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801652] lost page write due to I/O error on xvda2</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.801755] Aborting journal on device xvda2.</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.804415] EXT3-fs (xvda2): error: ext3_journal_start_sb: Detected aborted journal</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.804434] EXT3-fs (xvda2): error: remounting filesystem read-only</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 3.814754] journal commit I/O error</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 6.973831] init: udev-fallback-graphics main process (538) terminated with status 1</span><br style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: arial, helvetica, sans-serif; color: rgb(0, 0, 153); ">[ 6.992267] init: plymouth-splash main process (546) terminated with status 1</span><br><br>The manpage of drbdsetup says that LVM (which I use) doesn't support barriers (better known as "tagged command queuing" or "native command queuing"), so I configured the DRBD device not to use barriers. This can be seen in /proc/drbd (by "wo:f, meaning flush, the next method drbd chooses after barrier):<br><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> 3: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> ns:2160152 nr:520204 dw:2680344 dr:2678107 al:3549 bm:9183 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0</span><br><br>And on the other host:<br><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> 3: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> ns:0 nr:2160152 dw:2160152 dr:0 al:0 bm:8052 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0</span><br><br>I also enabled the option disable_sendpage, as per the DRBD docs:<br><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">cat /sys/module/drbd/parameters/disable_sendpage</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">Y</span><br><br>I also tried adding barriers=0 to fstab as mount option. Still it says:<br><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">[ 58.603896] blkfront: barrier: write xvda2 op failed</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">[ 58.603903] blkfront: xvda2: barrier or flush: disabled</span><br><br>I don't even know if ext3 has a nobarrier option, but it does seem to work. But, because only one of my storage systems is battery backed, it would not be smart.<br><br>Why does it still compain about barriers when I disabled that?<br><br>Both hosts are:<br><br>Debian: 6.0.4<br>uname -a: Linux 2.6.32-5-xen-amd64<br>drbd: 8.3.7<br>Xen: 4.0.1<br><br>Guest:<br><br>Ubuntu 12.04 LTS<br>uname -a: Linux 3.2.0-24-generic pvops<br><br>drbd resource:<br><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">resource drbdvm</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">{</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> meta-disk internal;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> device /dev/drbd3;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> startup</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # The timeout value when the last known state of the other side was available. 0 means infinite.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> wfc-timeout 0;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # Timeout value when the last known state was disconnected. 0 means infinite.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> degr-wfc-timeout 180;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> syncer</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # This is recommended only for low-bandwidth lines, to only send those</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # blocks which really have changed.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> #csums-alg md5;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # Set to about half your net speed</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> rate 60M;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # It seems that this option moved to the 'net' section in drbd 8.4. (later release than Debian has currently)</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> verify-alg md5;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> net</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # The manpage says this is recommended only in pre-production (because of its performance), to determine</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # if your LAN card has a TCP checksum offloading bug.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> #data-integrity-alg md5;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> disk</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # Detach causes the device to work over-the-network-only after the</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # underlying disk fails. Detach is not default for historical reasons, but is</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # recommended by the docs.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # However, the Debian defaults in drbd.conf suggest the machine will reboot in that event...</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> on-io-error detach;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # LVM doesn't support barriers, so disabling it. It will revert to flush. Check wo: in /proc/drbd. If you don't disable it, you get IO errors.</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> no-disk-barrier;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> on host1</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # universe is a VG</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> disk /dev/universe/drbdvm-disk;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> address 10.0.0.1:7792;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> on host2</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> {</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> # universe is a VG</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> disk /dev/universe/drbdvm-disk;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> address 10.0.0.2:7792;</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "> }</span><br style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); "><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; color: rgb(0, 0, 153); ">}</span><br><br><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">In my test setup: the primary host's storage is 9650SE SATA-II RAID PCIe with battery. The secondary is software RAID1.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Isn't DRBD+Xen widely used? With these problems, it's not going to work.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Any help welcome.<br></div><br></div>_______________________________________________<br>drbd-user mailing list<br><a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br><a href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br></div></span></blockquote></div><br></div></body></html>