[DRBD-user] drbd module crashes at BUG at lru_cache.c:570 when the peer mode goes diskless

Evzen Demcenko demcenko at cldn.cz
Mon May 13 18:43:43 CEST 2019


If drbd node in primary-primary setup looses disk by any reason (faulty 
disk, controller etc., or even manual detach) and there is R/W activity 
on both nodes, the second node eventually crashes with kernel panic 
"kernel BUG at /root/rpmbuild/BUILD/drbd-8.4.11-1/drbd/lru_cache.c:570!"
Before the crash there are a lot of messages in kernel.log on good node 
(with attached disk) like

block drbd1: al_complete_io() called on inactive extent 57
block drbd1: LOGIC BUG for enr=74

Eventually, the "good" node crashes within minutes or hours depending on 
disk activity leaving the cluster without any data.
I tested different versions from 8.4 tree (8.4.6, 8.4.7-1, 8.4.9-1, 
8.4.11-1), always with the same result.
There is also no difference on "real-hardware" and virtualized machines.
kernel is 2.6.32-754.12.1.el6.x86_64 on Centos-6.10 with latest updates. 
Tested also on other kernels with the same outcome.
There is a vmcore-dmesg.txt attached to this email, vmcore itself is 
available as well for every tested version, core files are 35-50Mb, so i 
can't attach them to email, but i'll be glad to share them in some other 
way if needed.

[root at drtest-11 ~]# cat /etc/drbd.d/global_common.conf
global {
         usage-count yes;
}

common {
         protocol C;

         handlers {
                 pri-on-incon-degr 
"/usr/lib/drbd/notify-pri-on-incon-degr.sh; 
/usr/lib/drbd/notify-emergency-reboot.sh";
                 pri-lost-after-sb 
"/usr/lib/drbd/notify-pri-lost-after-sb.sh; 
/usr/lib/drbd/notify-emergency-reboot.sh";
                 local-io-error "/usr/lib/drbd/notify-io-error.sh; 
/usr/lib/drbd/notify-emergency-shutdown.sh";
         }

         startup {
                 # wfc-timeout degr-wfc-timeout outdated-wfc-timeout 
wait-after-sb
         }

         disk {
                 resync-rate 100M;
                 on-io-error detach;
                 al-extents 1447;
                 c-plan-ahead 32;
                 c-max-rate 1000M;
                 c-min-rate 80M;
                 c-fill-target 65536k;
         }

         net {
                 sndbuf-size 4096k;
                 rcvbuf-size 4096k;
                 timeout       100;    #  10 seconds  (unit = 0.1 seconds)
                 connect-int   15;    # 15 seconds  (unit = 1 second)
                 ping-int      15;    # 15 seconds  (unit = 1 second)
                 ping-timeout  50;    # 5000 ms (unit = 0.1 seconds)
                 max-buffers     131072;
                 max-epoch-size  20000;
                 ko-count 0;
                 after-sb-0pri discard-younger-primary;
                 after-sb-1pri consensus;
                 after-sb-2pri disconnect;
                 rr-conflict disconnect;
         }
}

[root at drtest-11 ~]# cat /etc/drbd.d/r1.res
resource r1 {
     net {
         protocol C;
         allow-two-primaries;
         verify-alg crc32c;
         csums-alg crc32c;
     }
     startup {
         become-primary-on both;
     }
   disk {
       disk-timeout 1200;
   }
   on drtest-11.uvt.internal {
         device      /dev/drbd1;
         disk        "/dev/vdb";
         address     10.0.11.201:7790;
         flexible-meta-disk internal;
     }
   on drtest-12.uvt.internal {
         device      /dev/drbd1;
         disk        "/dev/vdb";
         address     10.0.11.202:7790;
         flexible-meta-disk internal;
     }
}

How to reproduce:
After create-md, connect, primary etc.:

On drtest-11:
pvcreate /dev/drbd1
vgcreate test /dev/drbd1
lvcreate -n t1 -L20g test
lvcreate -n t2 -L20g test
mkfs.ext4 /dev/test/t1
mount /dev/test/t1 /mnt/t1
mkdir -m 777 /mnt/t1/test
while true ; do bonnie++ -u nobody -d /mnt/t1/test/ -n 8192  -s8192 ; done

On drtest-12
vgchange -aly
mkfs.ext4 /dev/test/t2
mount /dev/test/t2 /mnt/t2
mkdir -m 777 /mnt/t2/test
while true ; do bonnie++ -u nobody -d /mnt/t2/test/ -n 8192  -s8192 ; done
drbdadm detach r1

After the detach on drtest-12, drtest-11 almost instantly starts 
flooding the log with " al_complete_io() called on inactive extent" and 
"LOGIC BUG for enr=" and crashes within couple of minutes.

Thanks in advance for investigating this issue.
Sincerely,

-- 
Ing. Evzen Demcenko
Senior Linux Administrator
Cluster Design s.r.o.

-------------- next part --------------
<6>Initializing cgroup subsys cpuset
<6>Initializing cgroup subsys cpu
<5>Linux version 2.6.32-754.12.1.el6.x86_64 (mockbuild at x86-01.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Tue Apr 9 14:52:26 UTC 2019
<6>Command line: ro root=UUID=87bf6ea3-cc99-4cc0-b45b-be3eb3f44396 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
<6>KERNEL supported cpus:
<6>  Intel GenuineIntel
<6>  AMD AuthenticAMD
<6>  Centaur CentaurHauls
<6>BIOS-provided physical RAM map:
<6> BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
<6> BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
<6> BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
<6> BIOS-e820: 0000000000100000 - 00000000bfffe000 (usable)
<6> BIOS-e820: 00000000bfffe000 - 00000000c0000000 (reserved)
<6> BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
<6> BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
<6> BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
<4>SMBIOS version 2.4 @ 0xFD9F0
<6>SMBIOS 2.4 present.
<7>DMI: Red Hat KVM, BIOS Bochs 01/01/2011
<6>Hypervisor detected: KVM
<7>e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
<7>e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
<6>last_pfn = 0x140000 max_arch_pfn = 0x400000000
<6>x86 PAT enabled: cpu 0, old 0x70406, new 0x7010600070106
<6>last_pfn = 0xbfffe max_arch_pfn = 0x400000000
<7>initial memory mapped : 0 - 20000000
<6>init_memory_mapping: 0000000000000000-00000000bfffe000
<7> 0000000000 - 00bfe00000 page 2M
<7> 00bfe00000 - 00bfffe000 page 4k
<7>kernel direct mapping tables up to bfffe000 @ 8000-d000
<7>Use unified mapping for non-reserved e820 regions.
<6>init_memory_mapping: 0000000100000000-0000000140000000
<7> 0100000000 - 0140000000 page 2M
<7>kernel direct mapping tables up to 140000000 @ b000-d000
<6>RAMDISK: 369ca000 - 37feff78
<6>ACPI: Deleted _OSI(Windows 2012)
<6>ACPI: Deleted _OSI(Windows 2013)
<4>ACPI: RSDP 00000000000fd7c0 00014 (v00 BOCHS )
<4>ACPI: RSDT 00000000bffffbd6 00030 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
<4>ACPI: FACP 00000000bffff177 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
<4>ACPI: DSDT 00000000bfffe040 01137 (v01   BXPC   BXDSDT 00000001 INTL 20090123)
<4>ACPI: FACS 00000000bfffe000 00040
<4>ACPI: SSDT 00000000bffff1eb 0095B (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
<4>ACPI: APIC 00000000bffffb46 00090 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
<7>ACPI: Local APIC address 0xfee00000
<6>Setting APIC routing to flat.
<6>No NUMA configuration found
<6>Faking a node at 0000000000000000-0000000140000000
<6>Bootmem setup node 0 0000000000000000-0000000140000000
<6>  NODE_DATA [000000000000c000 - 000000000003ffff]
<6>  bootmap [0000000000040000 -  0000000000067fff] pages 28
<6>(8 early reservations) ==> bootmem [0000000000 - 0140000000]
<6>  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
<6>  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
<6>  #2 [0001000000 - 0002052a64]    TEXT DATA BSS ==> [0001000000 - 0002052a64]
<6>  #3 [00369ca000 - 0037feff78]          RAMDISK ==> [00369ca000 - 0037feff78]
<6>  #4 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 - 0000100000]
<6>  #5 [0002053000 - 00020530b9]              BRK ==> [0002053000 - 00020530b9]
<6>  #6 [0000008000 - 000000b000]          PGTABLE ==> [0000008000 - 000000b000]
<6>  #7 [000000b000 - 000000c000]          PGTABLE ==> [000000b000 - 000000c000]
<6>found SMP MP-table at [ffff8800000fda10] fda10
<6>Found 129MB of memory at 48MB for crashkernel auto 
<6>Reserving 129MB of memory at 48MB for crashkernel (System RAM: 5120MB)
<6>kvm-clock: Using msrs 4b564d01 and 4b564d00
<6>kvm-clock: cpu 0, msr 0:1c3db81, boot clock
<7> [ffffea0000000000-ffffea00045fffff] PMD -> [ffff880028600000-ffff88002bdfffff] on node 0
<4>Zone PFN ranges:
<4>  DMA      0x00000001 -> 0x00001000
<4>  DMA32    0x00001000 -> 0x00100000
<4>  Normal   0x00100000 -> 0x00140000
<4>Movable zone start PFN for each node
<4>early_node_map[3] active PFN ranges
<4>    0: 0x00000001 -> 0x0000009f
<4>    0: 0x00000100 -> 0x000bfffe
<4>    0: 0x00100000 -> 0x00140000
<7>On node 0 totalpages: 1048476
<7>  DMA zone: 56 pages used for memmap
<7>  DMA zone: 103 pages reserved
<7>  DMA zone: 3839 pages, LIFO batch:0
<7>  DMA32 zone: 14280 pages used for memmap
<7>  DMA32 zone: 768054 pages, LIFO batch:31
<7>  Normal zone: 3584 pages used for memmap
<7>  Normal zone: 258560 pages, LIFO batch:31
<6>ACPI: PM-Timer IO Port: 0xb008
<7>ACPI: Local APIC address 0xfee00000
<6>Setting APIC routing to flat.
<6>ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
<6>ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
<6>ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
<6>ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
<6>ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
<6>ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
<6>IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
<6>ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
<7>ACPI: IRQ0 used by override.
<7>ACPI: IRQ2 used by override.
<7>ACPI: IRQ5 used by override.
<7>ACPI: IRQ9 used by override.
<7>ACPI: IRQ10 used by override.
<7>ACPI: IRQ11 used by override.
<6>Using ACPI (MADT) for SMP configuration information
<6>SMP: Allowing 4 CPUs, 0 hotplug CPUs
<7>nr_irqs_gsi: 24
<6>PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
<6>PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
<6>PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
<6>PM: Registered nosave memory: 00000000bfffe000 - 00000000c0000000
<6>PM: Registered nosave memory: 00000000c0000000 - 00000000feffc000
<6>PM: Registered nosave memory: 00000000feffc000 - 00000000ff000000
<6>PM: Registered nosave memory: 00000000ff000000 - 00000000fffc0000
<6>PM: Registered nosave memory: 00000000fffc0000 - 0000000100000000
<6>Allocating PCI resources starting at c0000000 (gap: c0000000:3effc000)
<6>Booting paravirtualized kernel on KVM
<6>NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
<6>PERCPU: Embedded 33 pages/cpu @ffff880028200000 s104088 r8192 d22888 u524288
<6>pcpu-alloc: s104088 r8192 d22888 u524288 alloc=1*2097152
<6>pcpu-alloc: [0] 0 1 2 3 
<6>kvm-clock: cpu 0, msr 0:28218b81, primary cpu clock
<6>kvm-stealtime: cpu 0, msr 28212d00
<4>Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1030453
<4>Policy zone: Normal
<5>Kernel command line: ro root=UUID=87bf6ea3-cc99-4cc0-b45b-be3eb3f44396 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M at 48M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
<6>PID hash table entries: 4096 (order: 3, 32768 bytes)
<6>Memory: 3897976k/5242880k available (5540k kernel code, 1048976k absent, 295928k reserved, 6889k data, 1348k init)
<6>Kernel/User page tables isolation: enabled
<6>Hierarchical RCU implementation.
<6>NR_IRQS:33024 nr_irqs:440
<4>Console: colour VGA+ 80x25
<6>console [tty0] enabled
<6>allocated 16777216 bytes of page_cgroup
<6>please try 'cgroup_disable=memory' option if you don't want memory cgroups
<4>Detected 3499.996 MHz processor.
<6>Calibrating delay loop (skipped) preset value.. 6999.99 BogoMIPS (lpj=3499996)
<6>pid_max: default: 32768 minimum: 301
<6>Security Framework initialized
<6>SELinux:  Initializing.
<7>SELinux:  Starting in permissive mode
<6>Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
<6>Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
<4>Mount-cache hash table entries: 256
<6>Initializing cgroup subsys ns
<6>Initializing cgroup subsys cpuacct
<6>Initializing cgroup subsys memory
<6>Initializing cgroup subsys devices
<6>Initializing cgroup subsys freezer
<6>Initializing cgroup subsys net_cls
<6>Initializing cgroup subsys blkio
<6>Initializing cgroup subsys perf_event
<6>Initializing cgroup subsys net_prio
<6>mce: CPU supports 10 MCE banks
<6>Speculative Store Bypass: Vulnerable
<6>FEATURE SPEC_CTRL Not Present
<6>FEATURE IBPB_SUPPORT Not Present
<6>Spectre V2 : Mitigation: Full retpoline
<6>alternatives: switching to unfair spinlock
<6>ACPI: Core revision 20090903
<6>ftrace: converting mcount calls to 0f 1f 44 00 00
<6>ftrace: allocating 22085 entries in 87 pages
<6>Enabling x2apic
<6>Enabled x2apic
<6>APIC routing finalized to physical x2apic.
<6>..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
<6>CPU0: Intel Core 2 Duo P9xxx (Penryn Class Core 2) stepping 03
<6>Performance Events: unsupported p6 CPU model 23 no PMU driver, software events only.
<4>NMI watchdog disabled (cpu0): hardware events not enabled
<6>Booting Node   0, Processors  #1
<6>kvm-clock: cpu 1, msr 0:28298b81, secondary cpu clock
<6>kvm-stealtime: cpu 1, msr 28292d00
<4> #2
<6>kvm-clock: cpu 2, msr 0:28318b81, secondary cpu clock
<6>kvm-stealtime: cpu 2, msr 28312d00
<4> #3 Ok.
<6>kvm-clock: cpu 3, msr 0:28398b81, secondary cpu clock
<6>kvm-stealtime: cpu 3, msr 28392d00
<6>Brought up 4 CPUs
<6>Total of 4 processors activated (27999.96 BogoMIPS).
<7>sizeof(vma)=200 bytes
<7>sizeof(page)=56 bytes
<7>sizeof(inode)=592 bytes
<7>sizeof(dentry)=192 bytes
<7>sizeof(ext3inode)=800 bytes
<7>sizeof(buffer_head)=104 bytes
<7>sizeof(skbuff)=232 bytes
<7>sizeof(task_struct)=2672 bytes
<6>devtmpfs: initialized
<6>regulator: core version 0.5
<6>NET: Registered protocol family 16
<6>ACPI: bus type pci registered
<6>PCI: Using configuration type 1 for base access
<4>bio: create slab <bio-0> at 0
<7>ACPI: EC: Look up EC in DSDT
<6>ACPI: Interpreter enabled
<6>ACPI: (supports S0 S5)
<6>ACPI: Using IOAPIC for interrupt routing
<6>ACPI: No dock devices found.
<6>PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
<6>ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
<6>pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7]
<6>pci_root PNP0A03:00: host bridge window [io  0x0d00-0xffff]
<6>pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff]
<6>pci_root PNP0A03:00: host bridge window [mem 0xc0000000-0xfebfffff]
<6>PCI host bridge to bus 0000:00
<6>pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
<6>pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
<6>pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
<6>pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff]
<7>pci 0000:00:01.1: reg 20: [io  0xc0e0-0xc0ef]
<7>pci 0000:00:01.2: reg 20: [io  0xc080-0xc09f]
<6>pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
<6>pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
<7>pci 0000:00:02.0: reg 10: [mem 0xfc000000-0xfdffffff pref]
<7>pci 0000:00:02.0: reg 14: [mem 0xfebd0000-0xfebd0fff]
<7>pci 0000:00:02.0: reg 30: [mem 0xfebc0000-0xfebcffff pref]
<7>pci 0000:00:03.0: reg 10: [io  0xc0a0-0xc0bf]
<7>pci 0000:00:03.0: reg 14: [mem 0xfebd1000-0xfebd1fff]
<7>pci 0000:00:03.0: reg 30: [mem 0xfeb80000-0xfebbffff pref]
<7>pci 0000:00:04.0: reg 10: [io  0xc000-0xc03f]
<7>pci 0000:00:04.0: reg 14: [mem 0xfebd2000-0xfebd2fff]
<7>pci 0000:00:05.0: reg 10: [io  0xc040-0xc07f]
<7>pci 0000:00:05.0: reg 14: [mem 0xfebd3000-0xfebd3fff]
<7>pci 0000:00:06.0: reg 10: [io  0xc0c0-0xc0df]
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
<6>ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
<6>ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
<6>ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
<6>ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
<6>ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
<6>vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
<6>vgaarb: loaded
<6>vgaarb: bridge control possible 0000:00:02.0
<5>SCSI subsystem initialized
<7>libata version 3.00 loaded.
<6>usbcore: registered new interface driver usbfs
<6>usbcore: registered new interface driver hub
<6>usbcore: registered new device driver usb
<6>PCI: Using ACPI for IRQ routing
<7>PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
<7>PCI: pci_cache_line_size set to 64 bytes
<6>NetLabel: Initializing
<6>NetLabel:  domain hash size = 128
<6>NetLabel:  protocols = UNLABELED CIPSOv4
<6>NetLabel:  unlabeled traffic allowed by default
<6>Switching to clocksource kvm-clock
<6>pnp: PnP ACPI init
<6>ACPI: bus type pnp registered
<7>pnp 00:00: [io  0x0cf8-0x0cff]
<7>pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
<7>pnp 00:01: [io  0x0070-0x0071]
<7>pnp 00:01: [irq 8]
<7>pnp 00:01: [io  0x0072-0x0077]
<7>pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
<7>pnp 00:02: [io  0x0060]
<7>pnp 00:02: [io  0x0064]
<7>pnp 00:02: [irq 1]
<7>pnp 00:02: Plug and Play ACPI device, IDs PNP0303 (active)
<7>pnp 00:03: [irq 12]
<7>pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
<7>pnp 00:04: [io  0x03f2-0x03f5]
<7>pnp 00:04: [io  0x03f7]
<7>pnp 00:04: [irq 6]
<7>pnp 00:04: [dma 2]
<7>pnp 00:04: Plug and Play ACPI device, IDs PNP0700 (active)
<6>pnp: PnP ACPI: found 5 devices
<6>ACPI: ACPI bus type pnp unregistered
<7>pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7]
<7>pci_bus 0000:00: resource 5 [io  0x0d00-0xffff]
<7>pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff]
<7>pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff]
<6>NET: Registered protocol family 2
<6>IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
<6>TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
<6>TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
<6>TCP: Hash tables configured (established 524288 bind 65536)
<6>TCP reno registered
<6>NET: Registered protocol family 1
<6>pci 0000:00:00.0: Limiting direct PCI/PCI transfers
<6>pci 0000:00:01.0: PIIX3: Enabling Passive Release
<6>pci 0000:00:01.0: Activating ISA DMA hang workarounds
<4>ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
<6>pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
<6>pci 0000:00:01.2: PCI INT D disabled
<7>pci 0000:00:02.0: Boot video device
<6>Trying to unpack rootfs image as initramfs...
<6>Freeing initrd memory: 22679k freed
<6>PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
<6>Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
<6>software IO TLB at phys 0x20000000 - 0x24000000
<6>sha256_ssse3: Using SSSE3 optimized SHA-256 implementation
<6>futex hash table entries: 1024 (order: 4, 65536 bytes)
<6>audit: initializing netlink socket (disabled)
<5>type=2000 audit(1557502211.480:1): initialized
<6>HugeTLB registered 2 MB page size, pre-allocated 0 pages
<5>VFS: Disk quotas dquot_6.5.2
<4>Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
<6>msgmni has been set to 7657
<7>SELinux:  Registering netfilter hooks
<4>ksign: Installing public key data
<4>Loading keyring
<4>- Added public key 56C324F9B892E9B7
<4>- User ID: CentOS (Kernel Module GPG key)
<6>Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
<6>io scheduler noop registered
<6>io scheduler anticipatory registered
<6>io scheduler deadline registered
<6>io scheduler cfq registered (default)
<6>pci_hotplug: PCI Hot Plug PCI Core version: 0.5
<6>pciehp: PCI Express Hot Plug Controller Driver version: 0.4
<6>acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
<6>acpiphp: Slot [3] registered
<6>acpiphp: Slot [4] registered
<6>acpiphp: Slot [5] registered
<6>acpiphp: Slot [6] registered
<6>acpiphp: Slot [7] registered
<6>acpiphp: Slot [8] registered
<6>acpiphp: Slot [9] registered
<6>acpiphp: Slot [10] registered
<6>acpiphp: Slot [11] registered
<6>acpiphp: Slot [12] registered
<6>acpiphp: Slot [13] registered
<6>acpiphp: Slot [14] registered
<6>acpiphp: Slot [15] registered
<6>acpiphp: Slot [16] registered
<6>acpiphp: Slot [17] registered
<6>acpiphp: Slot [18] registered
<6>acpiphp: Slot [19] registered
<6>acpiphp: Slot [20] registered
<6>acpiphp: Slot [21] registered
<6>acpiphp: Slot [22] registered
<6>acpiphp: Slot [23] registered
<6>acpiphp: Slot [24] registered
<6>acpiphp: Slot [25] registered
<6>acpiphp: Slot [26] registered
<6>acpiphp: Slot [27] registered
<6>acpiphp: Slot [28] registered
<6>acpiphp: Slot [29] registered
<6>acpiphp: Slot [30] registered
<6>acpiphp: Slot [31] registered
<6>input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
<6>ACPI: Power Button [PWRF]
<7>ACPI: acpi_idle registered with cpuidle
<4>[Firmware Bug]: No valid trip found
<6>GHES: HEST is not enabled!
<6>Non-volatile memory driver v1.3
<6>Linux agpgart interface v0.103
<6>crash memory driver: version 1.1
<6>Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
<6>brd: module loaded
<6>loop: module loaded
<6>input: Macintosh mouse button emulation as /devices/virtual/input/input1
<6>Fixed MDIO Bus: probed
<6>ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<6>ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
<6>uhci_hcd: USB Universal Host Controller Interface driver
<6>uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
<7>uhci_hcd 0000:00:01.2: setting latency timer to 64
<6>uhci_hcd 0000:00:01.2: UHCI Host Controller
<6>uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
<6>uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c080
<6>usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
<6>usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
<6>usb usb1: Product: UHCI Host Controller
<6>usb usb1: Manufacturer: Linux 2.6.32-754.12.1.el6.x86_64 uhci_hcd
<6>usb usb1: SerialNumber: 0000:00:01.2
<6>usb usb1: configuration #1 chosen from 1 choice
<6>hub 1-0:1.0: USB hub found
<6>hub 1-0:1.0: 2 ports detected
<6>PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
<6>serio: i8042 KBD port at 0x60,0x64 irq 1
<6>serio: i8042 AUX port at 0x60,0x64 irq 12
<6>mice: PS/2 mouse device common for all mice
<6>rtc_cmos 00:01: RTC can wake from S4
<6>input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
<6>rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
<6>rtc0: alarms up to one day, 114 bytes nvram
<6>cpuidle: using governor ladder
<6>cpuidle: using governor menu
<6>EFI Variables Facility v0.08 2004-May-17
<6>usbcore: registered new interface driver hiddev
<6>usbcore: registered new interface driver usbhid
<6>usbhid: v2.6:USB HID core driver
<6>GRE over IPv4 demultiplexor driver
<6>TCP cubic registered
<6>Initializing XFRM netlink socket
<6>NET: Registered protocol family 17
<4>registered taskstats version 1
<6>rtc_cmos 00:01: setting system clock to 2019-05-10 15:30:13 UTC (1557502213)
<6>Initalizing network drop monitor service
<6>Freeing unused kernel memory: 1348k freed
<6>Write protecting the kernel read-only data: 10240k
<6>Freeing unused kernel memory: 584k freed
<6>Freeing unused kernel memory: 1524k freed
<6>dracut: dracut-004-411.el6
<6>dracut: rd_NO_LUKS: removing cryptoluks activation
<6>dracut: rd_NO_LVM: removing LVM activation
<6>device-mapper: uevent: version 1.0.3
<6>device-mapper: ioctl: 4.33.1-ioctl (2015-8-18) initialised: dm-devel at redhat.com
<6>udev: starting version 147
<6>dracut: Starting plymouth daemon
<6>dracut: rd_NO_MD: removing MD RAID activation
<7>ata_piix 0000:00:01.1: version 2.13
<7>ata_piix 0000:00:01.1: setting latency timer to 64
<6>scsi0 : ata_piix
<6>scsi1 : ata_piix
<6>ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0e0 irq 14
<6>ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0e8 irq 15
<4>ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
<6>virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
<7>virtio-pci 0000:00:03.0: setting latency timer to 64
<6>virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
<7>virtio-pci 0000:00:04.0: setting latency timer to 64
<4>ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
<6>virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
<7>virtio-pci 0000:00:05.0: setting latency timer to 64
<4>ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
<6>virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11
<7>virtio-pci 0000:00:06.0: setting latency timer to 64
<7>  alloc irq_desc for 24 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:04.0: irq 24 for MSI/MSI-X
<7>  alloc irq_desc for 25 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:04.0: irq 25 for MSI/MSI-X
<6> vda: vda1 vda2
<7>  alloc irq_desc for 26 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X
<7>  alloc irq_desc for 27 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X
<6> vdb: unknown partition table
<6>EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: 
<6>dracut: Mounted root filesystem /dev/vda1
<6>SELinux:  Disabled at runtime.
<7>SELinux:  Unregistering netfilter hooks
<6>input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
<5>type=1404 audit(1557502214.142:2): selinux=0 auid=4294967295 ses=4294967295
<6>dracut: 
<6>dracut: Switching root
<6>udev: starting version 147
<6>Refined TSC clocksource calibration: 3500.088 MHz.
<6>piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
<7>  alloc irq_desc for 28 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:03.0: irq 28 for MSI/MSI-X
<7>  alloc irq_desc for 29 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:03.0: irq 29 for MSI/MSI-X
<7>  alloc irq_desc for 30 on node -1
<7>  alloc kstat_irqs on node -1
<7>virtio-pci 0000:00:03.0: irq 30 for MSI/MSI-X
<6>Adding 4193272k swap on /dev/vda2.  Priority:-1 extents:1 across:4193272k 
<6>NET: Registered protocol family 10
<6>lo: Disabled Privacy Extensions
<6>ip6_tables: (C) 2000-2006 Netfilter Core Team
<4>nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
<5>type=1305 audit(1557502219.162:3): audit_pid=1083 old=0 auid=4294967295 ses=4294967295 res=1
<7>eth0: no IPv6 routers present
<6>drbd: events: mcg drbd: 2
<6>drbd: initialized. Version: 8.4.11-1 (api:1/proto:86-101)
<6>drbd: GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by root at devel610.devel.cluster, 2019-05-09 16:32:15
<6>drbd: registered as block device major 147
<6>drbd r1: Starting worker thread (from drbdsetup-84 [5581])
<6>block drbd1: disk( Diskless -> Attaching ) 
<6>drbd r1: Method to ensure write ordering: flush
<6>block drbd1: max BIO size = 1048576
<7>block drbd1: This kernel is too old, no WRITE_SAME support.
<6>block drbd1: drbd_bm_resize called with capacity == 209708728
<6>block drbd1: resync bitmap: bits=26213591 words=409588 pages=800
<6>block drbd1: size = 100 GB (104854364 KB)
<6>block drbd1: recounting of set bits took additional 2 jiffies
<6>block drbd1: 23 GB (6098114 bits) marked out-of-sync by on disk bit-map.
<6>block drbd1: disk( Attaching -> UpToDate ) 
<6>block drbd1: attached to UUIDs EDB8F225EEF5DF6E:E80839F0EF48A4F7:6E89AEC5219FF5DB:6E88AEC5219FF5DB
<6>drbd r1: conn( StandAlone -> Unconnected ) 
<6>drbd r1: Starting receiver thread (from drbd_w_r1 [5583])
<6>drbd r1: receiver (re)started
<6>drbd r1: conn( Unconnected -> WFConnection ) 
<6>drbd r1: Handshake successful: Agreed network protocol version 101
<6>drbd r1: Feature flags enabled on protocol level: 0xf TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES.
<6>drbd r1: conn( WFConnection -> WFReportParams ) 
<6>drbd r1: Starting ack_recv thread (from drbd_r_r1 [5589])
<7>block drbd1: This kernel is too old, no WRITE_SAME support.
<6>block drbd1: drbd_sync_handshake:
<6>block drbd1: self EDB8F225EEF5DF6E:E80839F0EF48A4F7:6E89AEC5219FF5DB:6E88AEC5219FF5DB bits:6098114 flags:0
<6>block drbd1: peer E80839F0EF48A4F6:0000000000000000:6E89AEC5219FF5DB:6E88AEC5219FF5DB bits:118784 flags:2
<6>block drbd1: uuid_compare()=1 by rule 70
<6>block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent ) 
<6>block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 318(1), total 318; compression: 100.0%
<6>block drbd1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 321(1), total 321; compression: 100.0%
<6>block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1
<6>block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)
<6>block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent ) 
<6>block drbd1: Began resync as SyncSource (will sync 24410320 KB [6102580 bits set]).
<6>block drbd1: updated sync UUID EDB8F225EEF5DF6E:E80939F0EF48A4F7:E80839F0EF48A4F7:6E89AEC5219FF5DB
<6>block drbd1: peer( Secondary -> Primary ) 
<6>block drbd1: role( Secondary -> Primary ) 
<6>block drbd1: Resync done (total 142 sec; paused 0 sec; 171900 K/sec)
<6>block drbd1: 0 % had equal checksums, eliminated: 51980K; transferred 24358340K total 24410320K
<6>block drbd1: updated UUIDs EDB8F225EEF5DF6F:0000000000000000:E80939F0EF48A4F7:E80839F0EF48A4F7
<6>block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) 
<6>EXT4-fs (dm-0): recovery complete
<6>EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
<6>block drbd1: pdsk( UpToDate -> Failed ) 
<6>block drbd1: new current UUID FD890D158CCE2589:EDB8F225EEF5DF6F:E80939F0EF48A4F7:E80839F0EF48A4F7
<6>block drbd1: pdsk( Failed -> Diskless ) 
<6>block drbd1: LOGIC BUG for enr=51
<6>block drbd1: LOGIC BUG for enr=51
<6>block drbd1: LOGIC BUG for enr=51
<6>block drbd1: LOGIC BUG for enr=51
<3>block drbd1: al_complete_io() called on inactive extent 51
<3>block drbd1: al_complete_io() called on inactive extent 51
<3>block drbd1: al_complete_io() called on inactive extent 51
<3>block drbd1: al_complete_io() called on inactive extent 51
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<6>block drbd1: LOGIC BUG for enr=57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<3>block drbd1: al_complete_io() called on inactive extent 57
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<6>block drbd1: LOGIC BUG for enr=74
<4>------------[ cut here ]------------
<2>kernel BUG at /root/rpmbuild/BUILD/drbd-8.4.11-1/drbd/lru_cache.c:570!
<4>invalid opcode: 0000 [#1] SMP 
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 0 
<4>Modules linked in: drbd(U) libcrc32c ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 virtio_balloon virtio_net i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 5589, comm: drbd_r_r1 Not tainted 2.6.32-754.12.1.el6.x86_64 #1 Red Hat KVM
<4>RIP: 0010:[<ffffffffa020c33b>]  [<ffffffffa020c33b>] lc_put+0xab/0xe0 [drbd]
<4>RSP: 0018:ffff880028203b88  EFLAGS: 00010046
<4>RAX: 0000000000000000 RBX: ffff880138794180 RCX: 0000000000000000
<4>RDX: ffff880139600000 RSI: ffff8801394ea978 RDI: ffff880138794180
<4>RBP: ffff880028203b98 R08: 00000000000005a7 R09: 00000000018ffcd7
<4>R10: 0000000000000001 R11: 00000000000005a7 R12: ffff8801394ea978
<4>R13: 000000000000004a R14: ffff88013ce73cb8 R15: 0000000000000046
<4>FS:  0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 0000000080050033
<4>CR2: 0000000000404040 CR3: 0000000133428000 CR4: 00000000000006f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process drbd_r_r1 (pid: 5589, threadinfo ffff88013ac90000, task ffff880137bfb520)
<4>Stack:
<4> ffff88013ce73800 000000000000004a ffff880028203bd8 ffffffffa02085d7
<4><d> ffff88006d6c98c0 ffff88005d94d14c ffff88005d94d078 ffff88013ce73800
<4><d> 0000000000008806 ffff88005d94d0a0 ffff880028203c28 ffffffffa0204861
<4>Call Trace:
<4> <IRQ> 
<4> [<ffffffffa02085d7>] drbd_al_complete_io+0xe7/0x130 [drbd]
<4> [<ffffffffa0204861>] drbd_req_destroy+0x231/0x340 [drbd]
<4> [<ffffffffa0204630>] ? drbd_req_destroy+0x0/0x340 [drbd]
<4> [<ffffffff812a7d89>] kref_put+0x39/0x70
<4> [<ffffffffa02041b1>] mod_rq_state+0x301/0x720 [drbd]
<4> [<ffffffffa0204c35>] __req_mod+0x105/0x9f0 [drbd]
<4> [<ffffffffa01eee2b>] drbd_request_endio+0xab/0x380 [drbd]
<4> [<ffffffff811dc4d0>] bio_endio+0x20/0x40
<4> [<ffffffff8128528b>] req_bio_endio+0x9b/0xe0
<4> [<ffffffff81289c87>] blk_update_request+0x117/0x500
<4> [<ffffffff8128a097>] blk_update_bidi_request+0x27/0xa0
<4> [<ffffffff8128a13e>] __blk_end_request_all+0x2e/0x60
<4> [<ffffffffa006123a>] blk_done+0x4a/0x110 [virtio_blk]
<4> [<ffffffff810fdef4>] ? __rcu_process_callbacks+0x54/0x350
<4> [<ffffffffa005438f>] vring_interrupt+0x3f/0xe0 [virtio_ring]
<4> [<ffffffff810f8316>] handle_IRQ_event+0x66/0x180
<4> [<ffffffff81089784>] ? __do_softirq+0x154/0x240
<4> [<ffffffff810fad3e>] handle_edge_irq+0xde/0x180
<4> [<ffffffff8100e56a>] handle_irq+0x6a/0x100
<4> [<ffffffff810893e2>] ? irq_enter+0x22/0x80
<4> [<ffffffff8156671c>] do_IRQ+0x6c/0xf0
<4> [<ffffffff81564e93>] ret_from_intr+0x0/0x11
<4> <EOI> 
<4> [<ffffffff812b0c2d>] ? copy_user_generic_string+0x2d/0x40
<4> [<ffffffff81480085>] ? memcpy_toiovec+0x55/0x80
<4> [<ffffffff81480489>] skb_copy_datagram_iovec+0x159/0x2c0
<4> [<ffffffff814d6098>] tcp_recvmsg+0xea8/0x1100
<4> [<ffffffff814f7760>] inet_recvmsg+0x60/0xa0
<4> [<ffffffff8155c6ab>] ? _spin_unlock_bh+0x1b/0x20
<4> [<ffffffff81473ba9>] sock_recvmsg+0x139/0x170
<4> [<ffffffff81480489>] ? skb_copy_datagram_iovec+0x159/0x2c0
<4> [<ffffffff810aaeb0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81473ba9>] ? sock_recvmsg+0x139/0x170
<4> [<ffffffff81473c24>] kernel_recvmsg+0x44/0x60
<4> [<ffffffffa01f447d>] drbd_recv_short+0x4d/0x50 [drbd]
<4> [<ffffffff81136119>] ? mempool_alloc+0x69/0x150
<4> [<ffffffffa01f44b5>] drbd_recv+0x35/0x1a0 [drbd]
<4> [<ffffffff810aaeb0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff810aaeb0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffffa01f4636>] drbd_recv_all+0x16/0x40 [drbd]
<4> [<ffffffffa01f4684>] drbd_recv_all_warn+0x24/0x70 [drbd]
<4> [<ffffffffa01fdedb>] read_in_block+0x4fb/0x620 [drbd]
<4> [<ffffffffa02007cb>] receive_Data+0x8b/0x1030 [drbd]
<4> [<ffffffffa01f447d>] ? drbd_recv_short+0x4d/0x50 [drbd]
<4> [<ffffffffa01f4636>] ? drbd_recv_all+0x16/0x40 [drbd]
<4> [<ffffffffa01ff084>] drbd_receiver+0x104/0x2e0 [drbd]
<4> [<ffffffffa0211620>] ? drbd_thread_setup+0x0/0x110 [drbd]
<4> [<ffffffffa0211653>] drbd_thread_setup+0x33/0x110 [drbd]
<4> [<ffffffffa0211620>] ? drbd_thread_setup+0x0/0x110 [drbd]
<4> [<ffffffff810aaa10>] kthread+0xa0/0xc0
<4> [<ffffffff815657d0>] child_rip+0x20/0x30
<4> [<ffffffff810aa970>] ? kthread+0x0/0xc0
<4> [<ffffffff815657b0>] ? child_rip+0x0/0x30
<4>Code: 10 48 8b 13 e8 67 c4 0a e1 83 6b 64 01 f0 80 a3 90 00 00 00 f7 f0 80 a3 90 00 00 00 fe 5b 41 8b 44 24 20 41 5c c9 c3 0f 0b eb fe <0f> 0b eb fe 0f 0b eb fe 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 
<1>RIP  [<ffffffffa020c33b>] lc_put+0xab/0xe0 [drbd]
<4> RSP <ffff880028203b88>


More information about the drbd-user mailing list