[DRBD-user] local WRITE IO error sector 21776+1016 on dm-2

[Roland JARRY]

Hello,

I have an issue mounting drbd 8.4.11-1 resources on a kernel
4.9.0-9-amd64 (debian 9.9). I have this error message : block drbd3:
local WRITE IO error sector 21776+1016 on dm-2

Then, the resource becomes diskless.

Here are the settings of the resource :

root at srv-pg-sav-p:~# cat /etc/drbd.d/vgbackup-lv-back3.res
resource vgbackup-lv-back3 {
  net {
    allow-two-primaries;
  }
  startup {
    wfc-timeout 120;
    degr-wfc-timeout 120;
  }

  volume 0 {
    device    /dev/drbd3;
    #meta-disk internal;
    meta-disk /dev/vgbackup/lv-md-back3;
    disk      /dev/vgbackup/lv-back3;
}

  on srv-pg-sav-p {
    address   192.168.8.221:7803;

  }
  on srv-pg-sav-s {
    address   192.168.8.222:7803;
  }
}

I've changed meta-disk internal to external lv device to have more space
(1GB), but I have the same issue :

root at srv-pg-sav-p:~# lvs
  LV          VG       Attr       LSize  Pool Origin Data%  Meta%  Move
Log Cpy%Sync Convert
  lv-back1    vgbackup -wi-ao----
21.00t                                                   
  lv-back2    vgbackup -wi-ao----
21.00t                                                   
  lv-back3    vgbackup -wi-a-----
21.00t                                                   
  lv-md-back3 vgbackup -wi-a-----  1.00g   

I have 3 resources of same size. 2 works right now and not the 3rd. And
I had the same issue before with 2 first resources.

I notice that the error is on the same sector on each resource and at
each time. Is there a limitation somewhere ?

Here is more log :

Jul 23 10:23:52 srv-pg-sav-p kernel: [1532462.342531] EXT4-fs (drbd3):
mounted filesystem with ordered data mode. Opts: (null)
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.860138] block drbd3: local
WRITE IO error sector 21776+1016 on dm-2
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.860146] block drbd3: disk(
UpToDate -> Failed )
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.860177] block drbd3: Local
IO failed in __req_mod. Detaching...
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.868306] block drbd3:
helper command: /sbin/drbdadm pri-on-incon-degr minor-3
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.868356] block drbd3: IO
ERROR: neither local nor remote data, sector 21776+8
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.876611] block drbd3: IO
ERROR: neither local nor remote data, sector 21784+8
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.881934] block drbd3:
helper command: /sbin/drbdadm pri-on-incon-degr minor-3 exit code 0 (0x0)
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.885020] block drbd3: IO
ERROR: neither local nor remote data, sector 21792+8
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.894204] block drbd3: 21 TB
(5637144528 bits) marked out-of-sync by on disk bit-map.
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.894207] block drbd3: disk(
Failed -> Diskless )
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.894220] block drbd3: IO
ERROR: neither local nor remote data, sector 21800+8
Jul 23 10:23:53 srv-pg-sav-p kernel: [1532463.902118] block drbd3: IO
ERROR: neither local nor remote data, sector 21808+8
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.637796] block drbd3: 122
messages suppressed in /usr/src/modules/drbd/drbd/drbd_req.c:1446.
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.637802] block drbd3: IO
ERROR: neither local nor remote data, sector 22548840448+8
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.648265] Buffer I/O error
on dev drbd3, logical block 2818605056, lost sync page write
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.658056] JBD2: Error -5
detected when updating journal superblock for drbd3-8.
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.668068] Aborting journal
on device drbd3-8.
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.678189] Buffer I/O error
on dev drbd3, logical block 2818605056, lost sync page write
Jul 23 10:23:59 srv-pg-sav-p kernel: [1532469.688556] JBD2: Error -5
detected when updating journal superblock for drbd3-8.
Jul 23 10:45:28 srv-pg-sav-p kernel: [1533756.213355] block drbd3: 1
messages suppressed in /usr/src/modules/drbd/drbd/drbd_req.c:1446.
Jul 23 10:45:28 srv-pg-sav-p kernel: [1533756.213361] block drbd3: IO
ERROR: neither local nor remote data, sector 45097156480+8
Jul 23 10:45:28 srv-pg-sav-p kernel: [1533756.222815] block drbd3: IO
ERROR: neither local nor remote data, sector 45097156592+8
Jul 23 10:45:28 srv-pg-sav-p kernel: [1533756.232250] block drbd3: IO
ERROR: neither local nor remote data, sector 0+8
Jul 23 10:45:28 srv-pg-sav-p kernel: [1533756.241269] block drbd3: IO
ERROR: neither local nor remote data, sector 8+8

Kind regards.