<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
Hi Joel,<br>
<br>
All drbd devices are in the same volume group on the same physical
disk. The physical disk is a hardware raid-1 device consisting of 2
ssd disk. E.g:<br>
lvdisplay /dev/vg_ssd/test-lv<br>
--- Logical volume ---<br>
LV Path /dev/vg_ssd/test-lv<br>
LV Name test-lv<br>
VG Name vg_ssd<br>
LV UUID HWkc5O-FPBp-hjCb-WHjU-1Z1T-Wb1N-nUtRmH<br>
LV Write Access read/write<br>
LV Creation host, time barentsz, 2019-06-12 09:54:27 +0200<br>
LV Status available<br>
# open 1<br>
LV Size 40.00 GiB<br>
Current LE 10240<br>
Segments 1<br>
Allocation inherit<br>
Read ahead sectors auto<br>
- currently set to 1024<br>
Block device 254:16<br>
and:<br>
lvdisplay /dev/vg_ssd/vm_prace-ldap<br>
--- Logical volume ---<br>
LV Path /dev/vg_ssd/vm_prace-ldap<br>
LV Name vm_prace-ldap<br>
VG Name vg_ssd<br>
LV UUID KbAvaa-F9ER-NNfW-tPmb-C4WF-CHXP-ZheJed<br>
LV Write Access read/write<br>
LV Creation host, time barentsz, 2017-11-02 15:33:06 +0100<br>
LV Status available<br>
# open 2<br>
LV Size 38.00 GiB<br>
Current LE 9728<br>
Segments 2<br>
Allocation inherit<br>
Read ahead sectors auto<br>
- currently set to 1024<br>
Block device 254:9<br>
<br>
The drbd status on the failing R0 device on this host is:<br>
0:r0/0 Connected/Connected Secondary/Primary UpToDate/UpToDate <br>
<br>
Regards,<br>
Rob<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 6/12/19 10:39 AM, Joel Colledge
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGNP_+V5Q9ogc+GsJenqMLs8ccBDCAjHP1a1ZeepZM1CXywnZQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hi Rob,
<div><br>
</div>
<div>This message indicates that mounting with DAX is failing
here:</div>
<div><a
href="https://github.com/openSUSE/kernel/blob/SLE12-SP4/drivers/dax/super.c#L131"
moz-do-not-send="true">https://github.com/openSUSE/kernel/blob/SLE12-SP4/drivers/dax/super.c#L131</a><br>
</div>
<div><br>
</div>
<div>But DRBD performs the corresponding check here:</div>
<div><a
href="https://github.com/LINBIT/drbd-9.0/blob/drbd-9.0.18-1/drbd/drbd_dax_pmem.c#L57"
moz-do-not-send="true">https://github.com/LINBIT/drbd-9.0/blob/drbd-9.0.18-1/drbd/drbd_dax_pmem.c#L57</a><br>
</div>
<div><br>
</div>
<div>I don't know why the check fails for the DAX mount but
succeeds for DRBD. I suggest you set up a DRBD resource using
exactly this vg_ssd/test-lv that you used for the mount test.
It is possible that the LV you are using for DRBD and the test
LV are different in some way, e.g. they have been placed on
different backing devices. If DRBD works successfully on this
LV then we need to find out what the difference between the
LVs is.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Joel</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jun 12, 2019 at 10:00
AM Rob van der Wal <<a
href="mailto:rob.vanderwal@surfsara.nl"
moz-do-not-send="true">rob.vanderwal@surfsara.nl</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> Hi Joel,<br>
<br>
The additional debug setting gives one additional line:<br>
2019-06-12T09:58:27.977088+02:00 barentsz kern warning
kernel - - - [525370.955129] EXT4-fs (dm-16): DAX enabled.
Warning: EXPERIMENTAL, use at your own risk<br>
2019-06-12T09:58:27.977121+02:00 barentsz kern debug kernel
- - - [525370.955135] dm-16: error: dax access failed (-95)<br>
2019-06-12T09:58:27.977122+02:00 barentsz kern err kernel -
- - [525370.955136] EXT4-fs (dm-16): DAX unsupported by
block device. Turning off DAX.<br>
2019-06-12T09:58:27.977124+02:00 barentsz kern info kernel -
- - [525370.957006] EXT4-fs (dm-16): mounted filesystem with
ordered data mode. Opts: dax<br>
<br>
Regards,<br>
Rob<br>
<br>
<br>
<div class="gmail-m_1588587222548306639moz-cite-prefix">On
6/12/19 9:50 AM, Joel Colledge wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Rob,
<div><br>
</div>
<div>This is strange, since the filesystem DAX access
uses essentially the same checks as DRBD does. You can
get more detail about the failure by doing the mount
test again after enabling the relevant debug logging
as follows:</div>
<div>echo "file drivers/dax/super.c +p" >
/sys/kernel/debug/dynamic_debug/control<br>
</div>
<div><br>
</div>
<div>Perhaps there is a subtle difference in the checks
that DRBD does which is relevant for your device on
this kernel.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Joel</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jun 12, 2019
at 8:04 AM Rob van der Wal <<a
href="mailto:rob.vanderwal@surfsara.nl"
target="_blank" moz-do-not-send="true">rob.vanderwal@surfsara.nl</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> Hi Joel,<br>
<br>
Yes, I can mount with "-o dax":<br>
<br>
lvcreate -L 40G -n test-lv vg_ssd<br>
scan_dev_close /dev/sdb2 no DEV_IN_BCACHE set<br>
Logical volume "test-lv" created.<br>
mkfs.ext4 -b 4K /dev/vg_ssd/test-lv<br>
mount -o dax /dev/vg_ssd/test-lv /mnt<br>
df /mnt<br>
Filesystem 1K-blocks Used
Available Use% Mounted on<br>
/dev/mapper/vg_ssd-test--lv 41022688 49176
38859976 1% /mnt<br>
<br>
But it shows a failure in syslog:<br>
[518270.833954] EXT4-fs (dm-16): DAX enabled.
Warning: EXPERIMENTAL, use at your own risk<br>
[518270.833959] EXT4-fs (dm-16): DAX unsupported by
block device. Turning off DAX.<br>
[518270.835599] EXT4-fs (dm-16): mounted filesystem
with ordered data mode. Opts: dax<br>
<br>
Regards,<br>
Rob<br>
<br>
<br>
<div
class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-cite-prefix">On
6/11/19 3:57 PM, Joel Colledge wrote:<br>
</div>
<blockquote type="cite">
<pre class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-quote-pre">Hi Rob,
This is strange. It seems that your LV is reporting that it supports
PMEM/DAX. I suggest that you check that this issue also occurs without
DRBD. For example, create a filesystem and try to mount it with DAX
enabled:
mkfs.ext4 -b 4K /dev/<your-lv>
mount -o dax /dev/<your-lv> <somewhere>
The check the syslog to see whether DAX was successfully enabled. You
can see the expected success and failure output here:
<a class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-txt-link-freetext" href="https://nvdimm.wiki.kernel.org/#filesystems" target="_blank" moz-do-not-send="true">https://nvdimm.wiki.kernel.org/#filesystems</a>
Best regards,
Joel
PS Please "reply all" to keep the mailing list in CC
On Tue, Jun 11, 2019 at 2:53 PM Rob van der Wal
<a class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-txt-link-rfc2396E" href="mailto:rob.vanderwal@surfsara.nl" target="_blank" moz-do-not-send="true"><rob.vanderwal@surfsara.nl></a> wrote:
</pre>
<blockquote type="cite">
<pre class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-quote-pre">Hi Joel,
I've build the module from source (downloaded from <a class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-txt-link-freetext" href="https://www.linbit.com/en/drbd-community/drbd-download/" target="_blank" moz-do-not-send="true">https://www.linbit.com/en/drbd-community/drbd-download/</a>).
I think we don't use external metadata and we are using a ssd disk for storing. This is a part of our drbd.conf:
drbd.conf:
on barentsz {
node-id 1;
device /dev/drbd0 minor 0;
disk /dev/vg_ssd/vm_prace-ldap;
meta-disk internal;
address ipv4 <a href="http://10.0.0.30:7788" target="_blank" moz-do-not-send="true">10.0.0.30:7788</a>;
}
LVM info:
PV /dev/sdb2 VG vg_ssd lvm2 [446.43 GiB / 102.43 GiB free]
ACTIVE '/dev/vg_ssd/vm_prace-ldap' [38.00 GiB] inherit
The requested line in the syslog is:
[1037978.736262] drbd r0/0 drbd0: meta-data IO uses: dax-pmem
But I don't think we are using a PMEM device so this looks not correct.
Regards,
Rob van der Wal
</pre>
</blockquote>
</blockquote>
<br>
<div
class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-signature">--
<br>
<div><font face="Arial">Rob van der Wal</font></div>
<div><font face="Arial"><span
style="font-size:13px">Supercomputing |
SURFsara | Science Park 140 | 1098 XG
Amsterdam | T +31 20 800 1300 | <a
class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-txt-link-abbreviated"
href="mailto:Rob.vanderWal@surfsara.nl"
target="_blank" moz-do-not-send="true">Rob.vanderWal@surfsara.nl</a>
| <a
class="gmail-m_1588587222548306639gmail-m_3095418601436962624moz-txt-link-abbreviated"
href="http://www.surfsara.nl"
target="_blank" moz-do-not-send="true">www.surfsara.nl</a>
|</span></font></div>
<div><font face="Arial"><span
style="font-size:13px"></span></font></div>
<br>
<div
style="display:inline-block;vertical-align:top">
<img
src="cid:part11.66B2B9F2.4F133F48@surfsara.nl"
type="image/jpg" class=""></div>
<div style="display:inline-block">
<div><span
style="font-size:13px;color:rgb(0,85,165)"> We
are ISO 27001 certified and meet the high
requirements for information security. </span><span
style="box-sizing:inherit;color:rgb(0,0,205)"><img
src="cid:part12.DB1F4D0C.FBE99A1D@surfsara.nl"
type="image/png" class=""></span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_1588587222548306639gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr"><font size="2" face="arial,
helvetica, sans-serif"><span
style="color:rgb(0,0,0)">Joel Colledge</span><span
style="color:rgb(0,0,0)"> - </span><span
style="color:rgb(0,0,0)">Software
Developer</span><br
style="color:rgb(0,0,0)">
<a href="tel:+4318178292"
style="color:rgb(0,0,0)" target="_blank"
moz-do-not-send="true">+43-1-817-82-92 x<span>53</span></a><br
style="color:rgb(0,0,0)">
<a href="mailto:joel.colledge@linbit.com"
style="color:rgb(0,0,0)" target="_blank"
moz-do-not-send="true"><span>joel.colledge</span>@linbit.com</a><br
style="color:rgb(0,0,0)">
<br style="color:rgb(0,0,0)">
<a href="http://www.linbit.com/en/"
style="color:rgb(250,133,34)"
target="_blank" moz-do-not-send="true">LIN</a><a
href="http://www.linbit.com/en/"
style="color:rgb(0,0,0)" target="_blank"
moz-do-not-send="true">BIT</a><span
style="color:rgb(0,0,0)"> | Keeping the
Digital World Running</span><br
style="color:rgb(0,0,0)">
<span style="color:rgb(0,0,0)">DRBD HA -
Disaster Recovery - Software-defined
Storage</span><br
style="color:rgb(0,0,0)">
<a href="https://twitter.com/linbit"
title="Twitter"
style="color:rgb(0,172,237)"
target="_blank" moz-do-not-send="true">t</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.facebook.com/pg/linbitdrbd/posts/"
title="Facebook"
style="color:rgb(59,89,152)"
target="_blank" moz-do-not-send="true">f</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.linkedin.com/company/linbit"
title="LinkedIn"
style="color:rgb(0,123,182)"
target="_blank" moz-do-not-send="true">in</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.youtube.com/user/linbit"
title="YouTube"
style="color:rgb(255,0,0)"
target="_blank" moz-do-not-send="true">y</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://plus.google.com/+Linbit/about"
title="Google+"
style="color:rgb(221,75,57)"
target="_blank" moz-do-not-send="true">g+</a><br
style="color:rgb(0,0,0)">
<br style="color:rgb(0,0,0)">
<span style="color:rgb(0,0,0)">DRBD® and
LINBIT® are registered trademarks of
LINBIT, Austria.</span></font><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<div class="gmail-m_1588587222548306639moz-signature">-- <br>
<div><font face="Arial">Rob van der Wal</font></div>
<div><font face="Arial"><span style="font-size:13px">Supercomputing
| SURFsara | Science Park 140 | 1098 XG Amsterdam |
T +31 20 800 1300 | <a
class="gmail-m_1588587222548306639moz-txt-link-abbreviated"
href="mailto:Rob.vanderWal@surfsara.nl"
target="_blank" moz-do-not-send="true">Rob.vanderWal@surfsara.nl</a>
| <a
class="gmail-m_1588587222548306639moz-txt-link-abbreviated"
href="http://www.surfsara.nl" target="_blank"
moz-do-not-send="true">www.surfsara.nl</a> |</span></font></div>
<div><font face="Arial"><span style="font-size:13px"></span></font></div>
<br>
<div style="display:inline-block;vertical-align:top"> <img
type="image/jpg" moz-do-not-send="true"></div>
<div style="display:inline-block">
<div><span style="font-size:13px;color:rgb(0,85,165)"> We
are ISO 27001 certified and meet the high
requirements for information security. </span><span
style="box-sizing:inherit;color:rgb(0,0,205)"><img
type="image/png" moz-do-not-send="true"></span></div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr"><font size="2" face="arial, helvetica,
sans-serif"><span style="color:rgb(0,0,0)">Joel
Colledge</span><span style="color:rgb(0,0,0)"> - </span><span
style="color:rgb(0,0,0)">Software Developer</span><br
style="color:rgb(0,0,0)">
<a href="tel:+4318178292" style="color:rgb(0,0,0)"
target="_blank" moz-do-not-send="true">+43-1-817-82-92
x<span>53</span></a><br style="color:rgb(0,0,0)">
<a href="mailto:joel.colledge@linbit.com"
style="color:rgb(0,0,0)" target="_blank"
moz-do-not-send="true"><span>joel.colledge</span>@linbit.com</a><br
style="color:rgb(0,0,0)">
<br style="color:rgb(0,0,0)">
<a href="http://www.linbit.com/en/"
style="color:rgb(250,133,34)" target="_blank"
moz-do-not-send="true">LIN</a><a
href="http://www.linbit.com/en/"
style="color:rgb(0,0,0)" target="_blank"
moz-do-not-send="true">BIT</a><span
style="color:rgb(0,0,0)"> | Keeping the Digital
World Running</span><br style="color:rgb(0,0,0)">
<span style="color:rgb(0,0,0)">DRBD HA - Disaster
Recovery - Software-defined Storage</span><br
style="color:rgb(0,0,0)">
<a href="https://twitter.com/linbit"
title="Twitter" style="color:rgb(0,172,237)"
target="_blank" moz-do-not-send="true">t</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.facebook.com/pg/linbitdrbd/posts/"
title="Facebook" style="color:rgb(59,89,152)"
target="_blank" moz-do-not-send="true">f</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.linkedin.com/company/linbit"
title="LinkedIn" style="color:rgb(0,123,182)"
target="_blank" moz-do-not-send="true">in</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://www.youtube.com/user/linbit"
title="YouTube" style="color:rgb(255,0,0)"
target="_blank" moz-do-not-send="true">y</a><span
style="color:rgb(0,0,0)"> / </span><a
href="https://plus.google.com/+Linbit/about"
title="Google+" style="color:rgb(221,75,57)"
target="_blank" moz-do-not-send="true">g+</a><br
style="color:rgb(0,0,0)">
<br style="color:rgb(0,0,0)">
<span style="color:rgb(0,0,0)">DRBD® and LINBIT®
are registered trademarks of LINBIT, Austria.</span></font><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<div><font face="Arial">Rob van der Wal</font></div>
<div><font face="Arial"><span style="font-size: 13px;">Supercomputing
| SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31 20
800 1300 | <a class="moz-txt-link-abbreviated" href="mailto:Rob.vanderWal@surfsara.nl">Rob.vanderWal@surfsara.nl</a> | <a class="moz-txt-link-abbreviated" href="http://www.surfsara.nl">www.surfsara.nl</a> |</span></font></div>
<div><font face="Arial"><span style="font-size: 13px;"></span></font></div>
<br>
<div style="display:inline-block;vertical-align:top;">
<img src="cid:part11.66B2B9F2.4F133F48@surfsara.nl"
type="image/jpg"></div>
<div style="display:inline-block;">
<div><span style="font-size: 13px; color: #0055a5;"> We
are ISO 27001 certified and meet the high requirements for
information security. </span><span style="box-sizing:
inherit; color: #0000cd;"><img
src="cid:part12.DB1F4D0C.FBE99A1D@surfsara.nl"
type="image/png"></span></div>
</div>
</div>
</body>
</html>