[DRBD-user] drbd-9.0.18-1 : BUG: unable to handle kernel NULL pointer dereference at 00000000000000b0
Rob van der Wal
rob.vanderwal at surfsara.nl
Wed Jun 12 11:44:08 CEST 2019
Hi Joel,
All drbd devices are in the same volume group on the same physical disk.
The physical disk is a hardware raid-1 device consisting of 2 ssd disk. E.g:
lvdisplay /dev/vg_ssd/test-lv
--- Logical volume ---
LV Path /dev/vg_ssd/test-lv
LV Name test-lv
VG Name vg_ssd
LV UUID HWkc5O-FPBp-hjCb-WHjU-1Z1T-Wb1N-nUtRmH
LV Write Access read/write
LV Creation host, time barentsz, 2019-06-12 09:54:27 +0200
LV Status available
# open 1
LV Size 40.00 GiB
Current LE 10240
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:16
and:
lvdisplay /dev/vg_ssd/vm_prace-ldap
--- Logical volume ---
LV Path /dev/vg_ssd/vm_prace-ldap
LV Name vm_prace-ldap
VG Name vg_ssd
LV UUID KbAvaa-F9ER-NNfW-tPmb-C4WF-CHXP-ZheJed
LV Write Access read/write
LV Creation host, time barentsz, 2017-11-02 15:33:06 +0100
LV Status available
# open 2
LV Size 38.00 GiB
Current LE 9728
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 1024
Block device 254:9
The drbd status on the failing R0 device on this host is:
0:r0/0 Connected/Connected Secondary/Primary UpToDate/UpToDate
Regards,
Rob
On 6/12/19 10:39 AM, Joel Colledge wrote:
> Hi Rob,
>
> This message indicates that mounting with DAX is failing here:
> https://github.com/openSUSE/kernel/blob/SLE12-SP4/drivers/dax/super.c#L131
>
> But DRBD performs the corresponding check here:
> https://github.com/LINBIT/drbd-9.0/blob/drbd-9.0.18-1/drbd/drbd_dax_pmem.c#L57
>
> I don't know why the check fails for the DAX mount but succeeds for
> DRBD. I suggest you set up a DRBD resource using exactly
> this vg_ssd/test-lv that you used for the mount test. It is possible
> that the LV you are using for DRBD and the test LV are different in
> some way, e.g. they have been placed on different backing devices. If
> DRBD works successfully on this LV then we need to find out what the
> difference between the LVs is.
>
> Best regards,
> Joel
>
>
> On Wed, Jun 12, 2019 at 10:00 AM Rob van der Wal
> <rob.vanderwal at surfsara.nl <mailto:rob.vanderwal at surfsara.nl>> wrote:
>
> Hi Joel,
>
> The additional debug setting gives one additional line:
> 2019-06-12T09:58:27.977088+02:00 barentsz kern warning kernel - -
> - [525370.955129] EXT4-fs (dm-16): DAX enabled. Warning:
> EXPERIMENTAL, use at your own risk
> 2019-06-12T09:58:27.977121+02:00 barentsz kern debug kernel - - -
> [525370.955135] dm-16: error: dax access failed (-95)
> 2019-06-12T09:58:27.977122+02:00 barentsz kern err kernel - - -
> [525370.955136] EXT4-fs (dm-16): DAX unsupported by block device.
> Turning off DAX.
> 2019-06-12T09:58:27.977124+02:00 barentsz kern info kernel - - -
> [525370.957006] EXT4-fs (dm-16): mounted filesystem with ordered
> data mode. Opts: dax
>
> Regards,
> Rob
>
>
> On 6/12/19 9:50 AM, Joel Colledge wrote:
>> Hi Rob,
>>
>> This is strange, since the filesystem DAX access uses essentially
>> the same checks as DRBD does. You can get more detail about the
>> failure by doing the mount test again after enabling the relevant
>> debug logging as follows:
>> echo "file drivers/dax/super.c +p" >
>> /sys/kernel/debug/dynamic_debug/control
>>
>> Perhaps there is a subtle difference in the checks that DRBD does
>> which is relevant for your device on this kernel.
>>
>> Best regards,
>> Joel
>>
>>
>> On Wed, Jun 12, 2019 at 8:04 AM Rob van der Wal
>> <rob.vanderwal at surfsara.nl <mailto:rob.vanderwal at surfsara.nl>> wrote:
>>
>> Hi Joel,
>>
>> Yes, I can mount with "-o dax":
>>
>> lvcreate -L 40G -n test-lv vg_ssd
>> scan_dev_close /dev/sdb2 no DEV_IN_BCACHE set
>> Logical volume "test-lv" created.
>> mkfs.ext4 -b 4K /dev/vg_ssd/test-lv
>> mount -o dax /dev/vg_ssd/test-lv /mnt
>> df /mnt
>> Filesystem 1K-blocks Used Available Use%
>> Mounted on
>> /dev/mapper/vg_ssd-test--lv 41022688 49176 38859976 1% /mnt
>>
>> But it shows a failure in syslog:
>> [518270.833954] EXT4-fs (dm-16): DAX enabled. Warning:
>> EXPERIMENTAL, use at your own risk
>> [518270.833959] EXT4-fs (dm-16): DAX unsupported by block
>> device. Turning off DAX.
>> [518270.835599] EXT4-fs (dm-16): mounted filesystem with
>> ordered data mode. Opts: dax
>>
>> Regards,
>> Rob
>>
>>
>> On 6/11/19 3:57 PM, Joel Colledge wrote:
>>> Hi Rob,
>>>
>>> This is strange. It seems that your LV is reporting that it supports
>>> PMEM/DAX. I suggest that you check that this issue also occurs without
>>> DRBD. For example, create a filesystem and try to mount it with DAX
>>> enabled:
>>> mkfs.ext4 -b 4K /dev/<your-lv>
>>> mount -o dax /dev/<your-lv> <somewhere>
>>>
>>> The check the syslog to see whether DAX was successfully enabled. You
>>> can see the expected success and failure output here:
>>> https://nvdimm.wiki.kernel.org/#filesystems
>>>
>>> Best regards,
>>> Joel
>>>
>>> PS Please "reply all" to keep the mailing list in CC
>>>
>>>
>>> On Tue, Jun 11, 2019 at 2:53 PM Rob van der Wal
>>> <rob.vanderwal at surfsara.nl> <mailto:rob.vanderwal at surfsara.nl> wrote:
>>>> Hi Joel,
>>>>
>>>> I've build the module from source (downloaded fromhttps://www.linbit.com/en/drbd-community/drbd-download/).
>>>>
>>>> I think we don't use external metadata and we are using a ssd disk for storing. This is a part of our drbd.conf:
>>>> drbd.conf:
>>>> on barentsz {
>>>> node-id 1;
>>>> device /dev/drbd0 minor 0;
>>>> disk /dev/vg_ssd/vm_prace-ldap;
>>>> meta-disk internal;
>>>> address ipv410.0.0.30:7788 <http://10.0.0.30:7788>;
>>>> }
>>>>
>>>>
>>>> LVM info:
>>>> PV /dev/sdb2 VG vg_ssd lvm2 [446.43 GiB / 102.43 GiB free]
>>>> ACTIVE '/dev/vg_ssd/vm_prace-ldap' [38.00 GiB] inherit
>>>>
>>>> The requested line in the syslog is:
>>>> [1037978.736262] drbd r0/0 drbd0: meta-data IO uses: dax-pmem
>>>>
>>>> But I don't think we are using a PMEM device so this looks not correct.
>>>>
>>>> Regards,
>>>> Rob van der Wal
>>>>
>>
>> --
>> Rob van der Wal
>> Supercomputing | SURFsara | Science Park 140 | 1098 XG
>> Amsterdam | T +31 20 800 1300 | Rob.vanderWal at surfsara.nl
>> <mailto:Rob.vanderWal at surfsara.nl> | www.surfsara.nl
>> <http://www.surfsara.nl> |
>>
>> We are ISO 27001 certified and meet the high
>> requirements for information security.
>>
>>
>>
>> --
>> Joel Colledge - Software Developer
>> +43-1-817-82-92 x53 <tel:+4318178292>
>> joel.colledge at linbit.com <mailto:joel.colledge at linbit.com>
>>
>> LIN <http://www.linbit.com/en/>BIT <http://www.linbit.com/en/> |
>> Keeping the Digital World Running
>> DRBD HA - Disaster Recovery - Software-defined Storage
>> t <https://twitter.com/linbit> / f
>> <https://www.facebook.com/pg/linbitdrbd/posts/> / in
>> <https://www.linkedin.com/company/linbit> / y
>> <https://www.youtube.com/user/linbit> / g+
>> <https://plus.google.com/+Linbit/about>
>>
>> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
>
> --
> Rob van der Wal
> Supercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam |
> T +31 20 800 1300 | Rob.vanderWal at surfsara.nl
> <mailto:Rob.vanderWal at surfsara.nl> | www.surfsara.nl
> <http://www.surfsara.nl> |
>
> We are ISO 27001 certified and meet the high
> requirements for information security.
>
>
>
> --
> Joel Colledge - Software Developer
> +43-1-817-82-92 x53 <tel:+4318178292>
> joel.colledge at linbit.com <mailto:joel.colledge at linbit.com>
>
> LIN <http://www.linbit.com/en/>BIT <http://www.linbit.com/en/> |
> Keeping the Digital World Running
> DRBD HA - Disaster Recovery - Software-defined Storage
> t <https://twitter.com/linbit> / f
> <https://www.facebook.com/pg/linbitdrbd/posts/> / in
> <https://www.linkedin.com/company/linbit> / y
> <https://www.youtube.com/user/linbit> / g+
> <https://plus.google.com/+Linbit/about>
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
--
Rob van der Wal
Supercomputing | SURFsara | Science Park 140 | 1098 XG Amsterdam | T +31
20 800 1300 | Rob.vanderWal at surfsara.nl | www.surfsara.nl |
We are ISO 27001 certified and meet the high requirements
for information security.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20190612/5af60308/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SURFsara
Type: image/png
Size: 2515 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20190612/5af60308/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ISO27001
Type: image/png
Size: 1742 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20190612/5af60308/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3704 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20190612/5af60308/attachment-0001.bin>
More information about the drbd-user
mailing list