[DRBD-user] fs errors on replacing secondary's disk

george young gry at ll.mit.edu
Tue Jun 15 18:08:37 CEST 2004

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


[drbd 0.6.10, SuSE 8.3 x86 linux, 2.4.20, lsi megaraid scsi 320-2]
nodes are "pig-app" and "pig-db".

One of the disks on node "pig-db" failed, (RAID 5, so no crash), so I
forced pig-db to secondary for both drbd file systems and rebooted pig-db without
drbd.  After replacing the disk, and wiping and reconfiguring the RAID
box(don't ask why...), I need to bring the system back up as drbd secondary.

Starting drbd seems fine:
May  9 15:58:21 pig-db drbd: modprobe -s drbd minor_count=2
May  9 15:58:21 pig-db kernel: drbd: initialised. Version: 0.6.10 (api:64/proto:62)
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb0 disk /dev/sdc1 --disk-size=36708492
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb0 net 10.0.0.114:7788 10.0.0.115:7788 C --sync-nice=-15 --sync-min=10M --sync-max=100M --tl-size=5000 --timeout=60 --connect-int=10 --ping-int=10 --sync-group=1
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb1 disk /dev/sdb1 --disk-size=1060290k
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb1 net 10.0.0.114:7789 10.0.0.115:7789 C --sync-nice=-15 --sync-min=10M --sync-max=100M --tl-size=5000 --timeout=60 --connect-int=10 --ping-int=10 --sync-group=2
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb0 wait_connect -t 0
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb1 wait_connect -t 0
May  9 15:58:21 pig-db kernel: drbd0: Connection established. size=36708492 KB / blksize=4096 B
May  9 15:58:21 pig-db kernel: drbd1: Connection established. size=1060290 KB / blksize=4096 B
May  9 15:58:21 pig-db drbd: 'drbd_home' SyncingQuick, waiting for this to finish
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb0 wait_sync
May  9 15:58:21 pig-db drbd: 'drbd_db' SyncingQuick, waiting for this to finish
May  9 15:58:21 pig-db drbd: drbdsetup /dev/nb1 wait_sync
May  9 15:59:42 pig-db drbd: 'drbd_home' SyncingQuick finished, issue drbdsetup /dev/nb0 secondary_remote
May  9 16:00:14 pig-db drbd: 'drbd_db' SyncingQuick finished, issue drbdsetup /dev/nb1 secondary_remo

I do "datadisk drbd_home start", which switches drbd_home partition to primary and mounts it:
May  9 17:38:46 pig-db datadisk: ===> datadisk drbd_db start <===
May  9 17:38:46 pig-db datadisk: drbdsetup /dev/nb1 primary
May  9 17:38:46 pig-db datadisk: /bin/true /dev/nb1
May  9 17:38:46 pig-db datadisk: mount -v /dev/nb1
May  9 17:38:46 pig-db kernel: drbd1: blksize=1024 B
May  9 17:38:46 pig-db kernel: drbd1: blksize=4096 B
May  9 17:38:46 pig-db kernel: reiserfs: found format "3.6" with standard journal
May  9 17:38:46 pig-db kernel: reiserfs: enabling write barrier flush mode
May  9 17:38:46 pig-db kernel: reiserfs: using ordered data mode
May  9 17:38:46 pig-db kernel: reiserfs: checking transaction log (drbd(43,1)) for (drbd(43,1))
May  9 17:38:46 pig-db kernel: Using r5 hash to sort names
May  9 17:38:46 pig-db datadisk: 'drbd_db' activated

But I get a stream of kernel errors, like:

  is_tree_node: node level 0 does not match to the expected one 1
  vs-5150: search_by_key: invalid format found in block 8831. Fsck?
  zam-7001: io error in reiserfs_find_entry

I double checked the device sizes (that has bitten me before).  Is there
some special initialization I have to do with a blank new disk device?

-- George Young
pig-db:~ # sfdisk -s /dev/sdc1
36716526

[from the OK node]-----------------------------------------------------------
pig-app:~> cat /proc/drbd
version: 0.6.10 (api:64/proto:62)

0: cs:WFConnection st:Primary/Unknown ns:30453144 nr:7689428 dw:42446232 dr:46980364 pe:0 ua:0
1: cs:WFConnection st:Primary/Unknown ns:24495916 nr:4975780 dw:33967100 dr:6315188 pe:0 ua:0

/etc/drbd.conf:-------------------------------------------------------------
resource drbd_home {
  protocol = C
  fsckcmd  = /bin/true
  disk {
    disk-size = 36708492
  }
  net {
     sync-nice  = -15  # if synchronization is high priority for you
    sync-min    = 10M
    sync-max    = 100M    # maximal average syncer bandwidth
    tl-size     = 5000  # transfer log size, ensures strict write ordering
    timeout     = 60    # 0.1 seconds
    connect-int = 10    # seconds
    ping-int    = 10    # seconds
    sync-group = 1
  }
  on pig-app {
    device  = /dev/nb0
    disk    = /dev/rd/c0d0p5
    address = 10.0.0.115
    port    = 7788
  }
  on pig-db {
    device  = /dev/nb0
    disk    = /dev/sdc1
    address = 10.0.0.114
    port    = 7788
  }
}
resource drbd_db {
  protocol = C
  fsckcmd  = /bin/true
  disk {
    disk-size = 1060290k
  }
  net {
     sync-nice  = -15  # if synchronization is high priority for you
    sync-min    = 10M
    sync-max    = 100M    # maximal average syncer bandwidth
    tl-size     = 5000  # transfer log size, ensures strict write ordering
    timeout     = 60    # 0.1 seconds
    connect-int = 10    # seconds
    ping-int    = 10    # seconds
    sync-group = 2
  }
  on pig-app {
    device  = /dev/nb1
    disk    = /dev/rd/c0d0p3
    address = 10.0.0.115
    port    = 7789
  }
  on pig-db {
    device  = /dev/nb1
    disk    = /dev/sdb1
    address = 10.0.0.114
    port    = 7789
  }
}

-- 
"Are the gods not just?"  "Oh no, child.
What would become of us if they were?" (CSL)



More information about the drbd-user mailing list