[DRBD-user] Recover Data from inconsistent nodes

[Manuthu M]

Hi all,

Please assist on recovering data from a DRBD MySQL node that failed to come up after a maintenance window.
Currently we have a setup of 2 mysql nodes (DB01 and DB02) using pacemaker and DRBD.

Earlier today, we needed to put on standby the primary node - DB02 in this case and use DB01. Pacemaker succeeded in promoting the secondary node (DB01) to primary and all other services under it.  After about ~ 45 minutes of usage, we noticed inconsistencies in the data. Further investigations revealed that the data was missing from about 2 weeks ago (07th July). Which means that the sync had somehow stopped between the two nodes. 

Worth noting is that during this time the nodes had started syncing before we noticed.
[root at db01 ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2020-04-05 02:58:18
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:29240140 nr:0 dw:21665384 dr:10156909 al:1027 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:6974196
	[=========>..........] sync'ed: 52.5% (6808/14316)M
	finish: 5:19:38 speed: 360 (1,480) K/sec


We attempted to revert to the previous setup by putting the current node on standby and promoting the secondary to primary but failed to come back online. 

We now have a old copy of the database from 2 weeks ago running on the primary. While we have nightly backups we have lost a significant amount of data from between last night backup to the time the issue arose.

Is it possible to recover this and what is the best approach to doing this. 

Here is the DRBD config file 

[root at db02 ~]# cat /etc/drbd.d/mysql01.res
resource mysql01 {
 protocol C;
 meta-disk internal;
 device /dev/drbd0;
 disk  /dev/centos_db-02/db02;
 handlers {
  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
 }
 net {
  allow-two-primaries no;
  after-sb-0pri discard-zero-changes;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
 }
 disk {
  on-io-error detach;
 }
 syncer {
  verify-alg sha1;
 }
 on db01 {
  address 172.19.5.8:7789;
 }
 on db02 {
  address 172.19.5.9:7789;
 }
}


Thanks in advance,
Mike(null)