Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hi DRBD users,
On DRBD 8.4.0 running on CentOS 6.0 64 bit, I have just run into a non-oops-non-crash-infloop problem of some sort. Below is a stack of possibly relevant information. Sadly, this is the second DRBD project that has failed for me - the first was unsatisfactorily plagued by kernel panics on real genuine RHEL around 3 years back.
I hope there's sufficient information here to help you find the bug, while I approach my project in a completely different manner. If you would like to blindly try this at home with a similar work load, you will need a stack of mysql data, and a sync- and o_direct-free version of mysql or the LD_PRELOAD that I used.
lock up and backtrace::::::::::::::::::::::::::::::::::::::::
INFO: task drbd_w_r1:4599 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
drbd_w_r1 D ffff880818639900 0 4599 2 0x00000080
ffff88080dd93c70 0000000000000046 ffff88081a9b8af8 ffff8800282569f0
ffff88080dd93bf0 ffffffff81056720 ffff88080dd93c40 000000000000056c
ffff88081a9b8638 ffff88080dd93fd8 0000000000010518 ffff88081a9b8638
Call Trace:
[<ffffffff81056720>] ? __dequeue_entity+0x30/0x50
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02098e5>] bm_page_io_async+0xe5/0x370 [drbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa020b8c2>] bm_rw+0x1a2/0x680 [drbd]
[<ffffffffa0202056>] ? crc32c+0x56/0x7c [libcrc32c]
[<ffffffffa020bdba>] drbd_bm_write_hinted+0x1a/0x20 [drbd]
[<ffffffffa0224602>] _al_write_transaction+0x2c2/0x6a0 [drbd]
[<ffffffffa0224d42>] w_al_write_transaction+0x22/0x50 [drbd]
[<ffffffffa020e85e>] drbd_worker+0x10e/0x480 [drbd]
[<ffffffffa022aa19>] drbd_thread_setup+0xa9/0x160 [drbd]
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffffa022a970>] ? drbd_thread_setup+0x0/0x160 [drbd]
[<ffffffff810141c0>] ? child_rip+0x0/0x20
Work load at the time of crashing:
Mysqld, running with O_DIRECT and O_SYNC disabled for the import exercise, via LD_PRELOAD
Around 22Gb of innodb .idb data written over 5 hours of mysql inserts and alter-table create index on large innodb tables
DRBD state at the time of crash/oops/infloop: disconnected administratively, so effectively stand-alone:
# cat /proc/drbd
version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root at mojosqlha1, 2011-09-30 07:02:41
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-----
ns:27125456 nr:0 dw:187074924 dr:606245 al:1018399 bm:1 lo:5 pe:0 ua:0 ap:3 ep:1 wo:b oos:39310620
Hardware:
Motherboard: Supermicro X8SIL
Disks: 2 x SATA in LVM on MD-raid1 for DRBD data (ext3) and root filesystem, 1 x SATA in LVM for DRBD meta data
CPU: 4 x Intel(R) Xeon(R) CPU X3430 @ 2.40GHz
Operating system: CentOS 6.0
DRBD: 8.4.0
Load just before crash: (zombie processes are an artefact of the import script that wraps zcat)
2011-10-26 22:08:05 # top -b -n 1
top - 22:08:05 up 13:32, 2 users, load average: 6.93, 6.21, 5.52
Tasks: 134 total, 1 running, 129 sleeping, 0 stopped, 4 zombie
Cpu(s): 2.1%us, 0.6%sy, 0.0%ni, 80.8%id, 16.3%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 33007192k total, 32744712k used, 262480k free, 31704k buffers
Swap: 32764472k total, 70876k used, 32693596k free, 14517424k cached
Here are the queries it got stuck on (it's still stuck 35000 seconds later):
+-----+------+-----------+---------------------+---------+-------+--------+-----------------------------
| Id | User | Host | db | Command | Time | State | Info
+-----+------+-----------+---------------------+---------+-------+--------+-----------------------------
| 734 | root | localhost | busihost_busibillv2 | Query | 35432 | update | INSERT INTO `xxtes` VALUES (
| 778 | root | localhost | busihost_busibillv2 | Query | 35416 | update | INSERT INTO `xxarchable_note
| 782 | root | localhost | busihost_busibillv2 | Query | 35432 | update | INSERT INTO `xxs` VALUES (49
| 789 | root | localhost | busihost_busibillv2 | Query | 35432 | update | INSERT INTO `xxgqueue` VALUE
| 895 | root | localhost | NULL | Query | 0 | NULL | show processlist
+-----+------+-----------+---------------------+---------+-------+--------+-----------------------------
drbd resource configuration::::::::::::::::::::::::::::::::::::::
resource r1 {
protocol A; # There are A, B and C protocols. Stick with C.
on bussqlha1 {
device /dev/drbd0; # The name of our drbd device.
disk /dev/Store/r1; # Partition we wish drbd to use.
address 192.168.60.81:7788; # Centos5a IP address and port number.
meta-disk /dev/Meta/r1meta; # Stores meta-data in lower portion of md2.
#meta-disk internal;
}
on bussqlha2 {
device /dev/drbd0; # Our drbd device, must match node0.
disk /dev/Store/r1; # Partition drbd should use.
address 192.168.60.82:7788; # IP address of Centos5b, and port number.
meta-disk internal; #Stores meta-data in lower portion of sdb1.
}
disk {
on-io-error detach; # What to do when the lower level device errors.
}
net {
max-buffers 2048; #datablock buffers used before writing to disk.
ko-count 4; # Peer is dead if this count is exceeded.
#on-disconnect reconnect; # Peer disconnected, try to reconnect.
cram-hmac-alg "sha1";
shared-secret "somethingverysecret";
}
syncer {
rate 90M; # Theoretical is 125M
#group 1; # Used for grouping resources, parallel sync.
al-extents 257; # Must be prime, number of active sets.
}
startup {
wfc-timeout 0; # drbd init script will wait infinitely on resources.
degr-wfc-timeout 120; # 2 minutes.
}
}
dmesg ::::::::::::::::::::::::::::::::::::::::
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild at c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
Command line: ro root=UUID=fd290d33-42cc-4a46-aee6-34c5a416ccd6 rd_MD_UUID=71a7d9b2:839ff9f1:7b431ec2:49f508a7 rd_MD_UUID=cf36d9bf:596e2a2b:c63eac40:1ae7fcb6 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009ec00 (usable)
BIOS-e820: 000000000009ec00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bf790000 (usable)
BIOS-e820: 00000000bf79e000 - 00000000bf7a0000 type 9
BIOS-e820: 00000000bf7a0000 - 00000000bf7ae000 (ACPI data)
BIOS-e820: 00000000bf7ae000 - 00000000bf7d0000 (ACPI NVS)
BIOS-e820: 00000000bf7d0000 - 00000000bf7e0000 (reserved)
BIOS-e820: 00000000bf7ed000 - 00000000c0000000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fed20000 - 00000000fed40000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 0000000100000000 - 0000000840000000 (usable)
DMI present.
AMI BIOS detected: BIOS may corrupt low RAM, working around it.
e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x840000 max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR fixed ranges enabled:
00000-9FFFF write-back
A0000-BFFFF uncachable
C0000-CBFFF write-protect
CC000-DFFFF uncachable
E0000-E3FFF write-protect
E4000-EFFFF write-through
F0000-FFFFF write-protect
MTRR variable ranges enabled:
0 base 000000000 mask 800000000 write-back
1 base 800000000 mask FC0000000 write-back
2 base 0C0000000 mask FC0000000 uncachable
3 disabled
4 disabled
5 disabled
6 disabled
7 disabled
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
original variable MTRRs
reg 0, base: 0GB, range: 32GB, type WB
reg 1, base: 32GB, range: 1GB, type WB
reg 2, base: 3GB, range: 1GB, type UC
total RAM covered: 32768M
Found optimal setting for mtrr clean up
gran_size: 64K chunk_size: 64K num_reg: 6 lose cover RAM: 0G
New variable MTRRs
reg 0, base: 0GB, range: 2GB, type WB
reg 1, base: 2GB, range: 1GB, type WB
reg 2, base: 4GB, range: 4GB, type WB
reg 3, base: 8GB, range: 8GB, type WB
reg 4, base: 16GB, range: 16GB, type WB
reg 5, base: 32GB, range: 1GB, type WB
e820 update range: 00000000c0000000 - 0000000100000000 (usable) ==> (reserved)
last_pfn = 0xbf790 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-00000000bf790000
0000000000 - 00bf600000 page 2M
00bf600000 - 00bf790000 page 4k
kernel direct mapping tables up to bf790000 @ 10000-15000
init_memory_mapping: 0000000100000000-0000000840000000
0100000000 - 0840000000 page 2M
kernel direct mapping tables up to 840000000 @ 13000-35000
RAMDISK: 3748b000 - 37fefba3
ACPI: RSDP 00000000000fa280 00024 (v02 ACPIAM)
ACPI: XSDT 00000000bf7a0100 00084 (v01 SMCI 20100527 MSFT 00000097)
ACPI: FACP 00000000bf7a0290 000F4 (v03 052710 FACP1450 20100527 MSFT 00000097)
ACPI: DSDT 00000000bf7a05f0 06C7B (v01 10605 10605000 00000000 INTL 20051117)
ACPI: FACS 00000000bf7ae000 00040
ACPI: APIC 00000000bf7a0390 00092 (v01 052710 APIC1450 20100527 MSFT 00000097)
ACPI: MCFG 00000000bf7a0430 0003C (v01 052710 OEMMCFG 20100527 MSFT 00000097)
ACPI: OEMB 00000000bf7ae040 00073 (v01 052710 OEMB1450 20100527 MSFT 00000097)
ACPI: HPET 00000000bf7aa5f0 00038 (v01 052710 OEMHPET 20100527 MSFT 00000097)
ACPI: GSCI 00000000bf7ae0c0 02024 (v01 052710 GMCHSCI 20100527 MSFT 00000097)
ACPI: DMAR 00000000bf7b00f0 00090 (v01 AMI OEMDMAR 00000001 MSFT 00000097)
ACPI: SSDT 00000000bf7b1580 00363 (v01 DpgPmm CpuPm 00000012 INTL 20051117)
ACPI: EINJ 00000000bf7aa630 00130 (v01 AMIER AMI_EINJ 20100527 MSFT 00000097)
ACPI: BERT 00000000bf7aa7c0 00030 (v01 AMIER AMI_BERT 20100527 MSFT 00000097)
ACPI: ERST 00000000bf7aa7f0 001B0 (v01 AMIER AMI_ERST 20100527 MSFT 00000097)
ACPI: HEST 00000000bf7aa9a0 000A8 (v01 AMIER ABC_HEST 20100527 MSFT 00000097)
ACPI: Local APIC address 0xfee00000
No NUMA configuration found
Faking a node at 0000000000000000-0000000840000000
Bootmem setup node 0 0000000000000000-0000000840000000
NODE_DATA [0000000000030000 - 0000000000063fff]
bootmap [0000000000100000 - 0000000000207fff] pages 108
(8 early reservations) ==> bootmem [0000000000 - 0840000000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#2 [0001000000 - 0001ca0ff8] TEXT DATA BSS ==> [0001000000 - 0001ca0ff8]
#3 [003748b000 - 0037fefba3] RAMDISK ==> [003748b000 - 0037fefba3]
#4 [000009ec00 - 0000100000] BIOS reserved ==> [000009ec00 - 0000100000]
#5 [0001ca1000 - 0001ca1217] BRK ==> [0001ca1000 - 0001ca1217]
#6 [0000010000 - 0000013000] PGTABLE ==> [0000010000 - 0000013000]
#7 [0000013000 - 0000030000] PGTABLE ==> [0000013000 - 0000030000]
found SMP MP-table at [ffff8800000ff780] ff780
Reserving 129MB of memory at 32MB for crashkernel (System RAM: 33792MB)
[ffffea0000000000-ffffea000fbfffff] PMD -> [ffff880028600000-ffff8800373fffff] on node 0
[ffffea000fc00000-ffffea001cdfffff] PMD -> [ffff880038000000-ffff8800451fffff] on node 0
Zone PFN ranges:
DMA 0x00000010 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00840000
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
0: 0x00000010 -> 0x0000009e
0: 0x00000100 -> 0x000bf790
0: 0x00100000 -> 0x00840000
On node 0 totalpages: 8386334
DMA zone: 56 pages used for memmap
DMA zone: 132 pages reserved
DMA zone: 3794 pages, LIFO batch:0
DMA32 zone: 14280 pages used for memmap
DMA32 zone: 765896 pages, LIFO batch:31
Normal zone: 103936 pages used for memmap
Normal zone: 7498240 pages, LIFO batch:31
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x84] disabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x85] disabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x86] disabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x87] disabled)
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x07] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 7, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a701 base: 0xfed00000
SMP: Allowing 8 CPUs, 4 hotplug CPUs
nr_irqs_gsi: 24
PM: Registered nosave memory: 000000000009e000 - 000000000009f000
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
PM: Registered nosave memory: 00000000bf790000 - 00000000bf79e000
PM: Registered nosave memory: 00000000bf79e000 - 00000000bf7a0000
PM: Registered nosave memory: 00000000bf7a0000 - 00000000bf7ae000
PM: Registered nosave memory: 00000000bf7ae000 - 00000000bf7d0000
PM: Registered nosave memory: 00000000bf7d0000 - 00000000bf7e0000
PM: Registered nosave memory: 00000000bf7e0000 - 00000000bf7ed000
PM: Registered nosave memory: 00000000bf7ed000 - 00000000c0000000
PM: Registered nosave memory: 00000000c0000000 - 00000000e0000000
PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000
PM: Registered nosave memory: 00000000f0000000 - 00000000fed20000
PM: Registered nosave memory: 00000000fed20000 - 00000000fed40000
PM: Registered nosave memory: 00000000fed40000 - 00000000fee00000
PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
PM: Registered nosave memory: 00000000fee01000 - 0000000100000000
Allocating PCI resources starting at c0000000 (gap: c0000000:20000000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880028200000 s95064 r8192 d23720 u262144
pcpu-alloc: s95064 r8192 d23720 u262144 alloc=1*2097152
pcpu-alloc: [0] 0 1 2 3 4 5 6 7
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 8267930
Policy zone: Normal
Kernel command line: ro root=UUID=fd290d33-42cc-4a46-aee6-34c5a416ccd6 rd_MD_UUID=71a7d9b2:839ff9f1:7b431ec2:49f508a7 rd_MD_UUID=cf36d9bf:596e2a2b:c63eac40:1ae7fcb6 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=129M at 0M rhgb quiet
PID hash table entries: 4096 (order: 3, 32768 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
software IO TLB at phys 0x20000000 - 0x24000000
Memory: 32862208k/34603008k available (4939k kernel code, 1057672k absent, 683128k reserved, 3931k data, 1220k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:472
Extended CMOS year: 2000
Console: colour VGA+ 80x25
console [tty0] enabled
allocated 335544320 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
hpet clockevent registered
alloc irq_desc for 24 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 25 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 26 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 27 on node 0
alloc kstat_irqs on node 0
alloc irq_desc for 28 on node 0
alloc kstat_irqs on node 0
HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Fast TSC calibration using PIT
Detected 2399.866 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 4799.73 BogoMIPS (lpj=2399866)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 9 MCE banks
CPU0: Thermal monitoring enabled (TM1)
using mwait in idle threads.
Performance Events: PEBS fmt1+, Nehalem events, Intel PMU driver.
... version: 3
... bit width: 48
... generic registers: 4
... value mask: 0000ffffffffffff
... max period: 000000007fffffff
... fixed-purpose events: 3
... event mask: 000000070000000f
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20293 entries in 80 pages
DMAR: Host address width 36
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
IOMMU fed90000: ver 1:0 cap c90780106f0462 ecap f020e3
DMAR: RMRR base: 0x000000000ed000 end: 0x000000000effff
DMAR: RMRR base: 0x000000bf7ed000 end: 0x000000bf7fffff
DMAR: No ATSR found
Setting APIC routing to flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz stepping 05
Booting Node 0, Processors #1 #2 #3
Brought up 4 CPUs
Total of 4 processors activated (19197.43 BogoMIPS).
Testing NMI watchdog ... OK.
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2600 bytes
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
PCI: MCFG area at e0000000 reserved in E820
PCI: Using MMCONFIG at e0000000 - efffffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: EC: Look up EC in DSDT
ACPI Warning for \_SB_._OSC: Return type mismatch - found Integer, expected Buffer (20090903/nspredef-1006)
\_SB_:_OSC evaluation returned wrong type
_OSC request data:1 7
ACPI: Executed 1 blocks of module-level executable AML code
ACPI: Interpreter enabled
ACPI: (supports S0 S1 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI Warning: Incorrect checksum in table [OEMB] - A4, should be A1 (20090903/tbutils-314)
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
pci 0000:00:03.0: PME# disabled
pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
pci 0000:00:05.0: PME# disabled
pci 0000:00:1a.0: reg 10 32bit mmio: [0xfb4fc000-0xfb4fc3ff]
pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1a.0: PME# disabled
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: PME# disabled
pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.4: PME# disabled
pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.5: PME# disabled
pci 0000:00:1d.0: reg 10 32bit mmio: [0xfb4fa000-0xfb4fa3ff]
pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.0: PME# disabled
pci 0000:00:1f.2: reg 10 io port: [0xcc00-0xcc07]
pci 0000:00:1f.2: reg 14 io port: [0xc880-0xc883]
pci 0000:00:1f.2: reg 18 io port: [0xc800-0xc807]
pci 0000:00:1f.2: reg 1c io port: [0xc480-0xc483]
pci 0000:00:1f.2: reg 20 io port: [0xc400-0xc40f]
pci 0000:00:1f.2: reg 24 io port: [0xc080-0xc08f]
pci 0000:00:1f.3: reg 10 64bit mmio: [0xfb4f8000-0xfb4f80ff]
pci 0000:00:1f.3: reg 20 io port: [0x400-0x41f]
pci 0000:00:1f.5: reg 10 io port: [0xbc00-0xbc07]
pci 0000:00:1f.5: reg 14 io port: [0xb880-0xb883]
pci 0000:00:1f.5: reg 18 io port: [0xb800-0xb807]
pci 0000:00:1f.5: reg 1c io port: [0xb480-0xb483]
pci 0000:00:1f.5: reg 20 io port: [0xb400-0xb40f]
pci 0000:00:1f.5: reg 24 io port: [0xb080-0xb08f]
pci 0000:04:00.0: reg 10 32bit mmio: [0xfb5e0000-0xfb5fffff]
pci 0000:04:00.0: reg 18 io port: [0xdc00-0xdc1f]
pci 0000:04:00.0: reg 1c 32bit mmio: [0xfb5dc000-0xfb5dffff]
pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
pci 0000:04:00.0: PME# disabled
pci 0000:00:1c.4: bridge io port: [0xd000-0xdfff]
pci 0000:00:1c.4: bridge 32bit mmio: [0xfb500000-0xfb5fffff]
pci 0000:05:00.0: reg 10 32bit mmio: [0xfb6e0000-0xfb6fffff]
pci 0000:05:00.0: reg 18 io port: [0xec00-0xec1f]
pci 0000:05:00.0: reg 1c 32bit mmio: [0xfb6dc000-0xfb6dffff]
pci 0000:05:00.0: PME# supported from D0 D3hot D3cold
pci 0000:05:00.0: PME# disabled
pci 0000:00:1c.5: bridge io port: [0xe000-0xefff]
pci 0000:00:1c.5: bridge 32bit mmio: [0xfb600000-0xfb6fffff]
pci 0000:06:03.0: reg 10 32bit mmio pref: [0xfa000000-0xfaffffff]
pci 0000:06:03.0: reg 14 32bit mmio: [0xfb7fc000-0xfb7fffff]
pci 0000:06:03.0: reg 18 32bit mmio: [0xfb800000-0xfbffffff]
pci 0000:00:1e.0: transparent bridge
pci 0000:00:1e.0: bridge 32bit mmio: [0xfb700000-0xfbffffff]
pci 0000:00:1e.0: bridge 64bit mmio pref: [0xfa000000-0xfaffffff]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P5._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR1E._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR20._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR24._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR25._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 6 7 10 11 12 *14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 6 *7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 6 7 10 11 12 14 *15)
vgaarb: device added: PCI:0000:06:03.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
PCI: pci_cache_line_size set to 64 bytes
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 24, 25, 26, 27, 28, 0
hpet0: 8 comparators, 64-bit 14.318180 MHz counter
hpet: hpet2 irq 24 for MSI
hpet: hpet3 irq 25 for MSI
hpet: hpet4 irq 26 for MSI
hpet: hpet5 irq 27 for MSI
Switching to clocksource tsc
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 16 devices
ACPI: ACPI bus type pnp unregistered
system 00:01: iomem range 0xfc000000-0xfcffffff has been reserved
system 00:01: iomem range 0xfd000000-0xfdffffff has been reserved
system 00:01: iomem range 0xfe000000-0xfebfffff has been reserved
system 00:01: iomem range 0xfed14000-0xfed19fff has been reserved
system 00:09: ioport range 0x164e-0x164f has been reserved
system 00:0a: ioport range 0xa00-0xa0f has been reserved
system 00:0a: ioport range 0xa10-0xa1f has been reserved
system 00:0b: ioport range 0xca2-0xca3 has been reserved
system 00:0b: ioport range 0xcf8-0xcff could not be reserved
system 00:0b: ioport range 0x4d0-0x4d1 has been reserved
system 00:0b: ioport range 0x800-0x87f has been reserved
system 00:0b: ioport range 0x500-0x57f has been reserved
system 00:0b: iomem range 0xfed1c000-0xfed1ffff has been reserved
system 00:0b: iomem range 0xfed20000-0xfed3ffff has been reserved
system 00:0b: iomem range 0xfed40000-0xfed8ffff has been reserved
system 00:0d: iomem range 0xfec00000-0xfec00fff could not be reserved
system 00:0d: iomem range 0xfee00000-0xfee00fff has been reserved
system 00:0e: iomem range 0xe0000000-0xefffffff has been reserved
system 00:0f: iomem range 0xc0000-0xcffff has been reserved
system 00:0f: iomem range 0xe0000-0xfffff could not be reserved
system 00:0f: iomem range 0xfed90000-0xffffffff could not be reserved
pci 0000:00:03.0: PCI bridge to [bus 01-01]
pci 0000:00:03.0: PCI bridge, secondary bus 0000:01
pci 0000:00:03.0: bridge window [io disabled]
pci 0000:00:03.0: bridge window [mem disabled]
pci 0000:00:03.0: bridge window [mem pref disabled]
pci 0000:00:05.0: PCI bridge to [bus 02-02]
pci 0000:00:05.0: PCI bridge, secondary bus 0000:02
pci 0000:00:05.0: bridge window [io disabled]
pci 0000:00:05.0: bridge window [mem disabled]
pci 0000:00:05.0: bridge window [mem pref disabled]
pci 0000:00:1c.0: PCI bridge to [bus 03-03]
pci 0000:00:1c.0: PCI bridge, secondary bus 0000:03
pci 0000:00:1c.0: bridge window [0x2000-0x2fff]
pci 0000:00:1c.0: bridge window [0xc0000000-0xc01fffff]
pci 0000:00:1c.0: bridge window [0xc0200000-0xc03fffff]
pci 0000:00:1c.4: PCI bridge to [bus 04-04]
pci 0000:00:1c.4: PCI bridge, secondary bus 0000:04
pci 0000:00:1c.4: bridge window [0xd000-0xdfff]
pci 0000:00:1c.4: bridge window [0xfb500000-0xfb5fffff]
pci 0000:00:1c.4: bridge window [0xc0400000-0xc05fffff]
pci 0000:00:1c.5: PCI bridge to [bus 05-05]
pci 0000:00:1c.5: PCI bridge, secondary bus 0000:05
pci 0000:00:1c.5: bridge window [0xe000-0xefff]
pci 0000:00:1c.5: bridge window [0xfb600000-0xfb6fffff]
pci 0000:00:1c.5: bridge window [0xc0600000-0xc07fffff]
pci 0000:00:1e.0: PCI bridge to [bus 06-06]
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:06
pci 0000:00:1e.0: bridge window [io disabled]
pci 0000:00:1e.0: bridge window [0xfb700000-0xfbffffff]
pci 0000:00:1e.0: bridge window [0xfa000000-0xfaffffff]
alloc irq_desc for 16 on node -1
alloc kstat_irqs on node -1
pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:03.0: setting latency timer to 64
pci 0000:00:05.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:05.0: setting latency timer to 64
pci 0000:00:1c.0: enabling device (0104 -> 0107)
alloc irq_desc for 17 on node -1
alloc kstat_irqs on node -1
pci 0000:00:1c.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
pci 0000:00:1c.0: setting latency timer to 64
pci 0000:00:1c.4: PCI INT A -> GSI 17 (level, low) -> IRQ 17
pci 0000:00:1c.4: setting latency timer to 64
pci 0000:00:1c.5: PCI INT B -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1c.5: setting latency timer to 64
pci 0000:00:1e.0: setting latency timer to 64
pci_bus 0000:00: resource 0 io: [0x00-0xffff]
pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffffffffffff]
pci_bus 0000:03: resource 0 io: [0x2000-0x2fff]
pci_bus 0000:03: resource 1 mem: [0xc0000000-0xc01fffff]
pci_bus 0000:03: resource 2 pref mem [0xc0200000-0xc03fffff]
pci_bus 0000:04: resource 0 io: [0xd000-0xdfff]
pci_bus 0000:04: resource 1 mem: [0xfb500000-0xfb5fffff]
pci_bus 0000:04: resource 2 pref mem [0xc0400000-0xc05fffff]
pci_bus 0000:05: resource 0 io: [0xe000-0xefff]
pci_bus 0000:05: resource 1 mem: [0xfb600000-0xfb6fffff]
pci_bus 0000:05: resource 2 pref mem [0xc0600000-0xc07fffff]
pci_bus 0000:06: resource 1 mem: [0xfb700000-0xfbffffff]
pci_bus 0000:06: resource 2 pref mem [0xfa000000-0xfaffffff]
pci_bus 0000:06: resource 3 io: [0x00-0xffff]
pci_bus 0000:06: resource 4 mem: [0x000000-0xffffffffffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:06:03.0: Boot video device
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 11666k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1319610920.787:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 32768
SELinux: Registering netfilter hooks
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 965331713A640F69
- User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
alloc irq_desc for 29 on node -1
alloc kstat_irqs on node -1
pcieport 0000:00:03.0: irq 29 for MSI/MSI-X
pcieport 0000:00:03.0: setting latency timer to 64
alloc irq_desc for 30 on node -1
alloc kstat_irqs on node -1
pcieport 0000:00:05.0: irq 30 for MSI/MSI-X
pcieport 0000:00:05.0: setting latency timer to 64
alloc irq_desc for 31 on node -1
alloc kstat_irqs on node -1
pcieport 0000:00:1c.0: irq 31 for MSI/MSI-X
pcieport 0000:00:1c.0: setting latency timer to 64
alloc irq_desc for 32 on node -1
alloc kstat_irqs on node -1
pcieport 0000:00:1c.4: irq 32 for MSI/MSI-X
pcieport 0000:00:1c.4: setting latency timer to 64
alloc irq_desc for 33 on node -1
alloc kstat_irqs on node -1
pcieport 0000:00:1c.5: irq 33 for MSI/MSI-X
pcieport 0000:00:1c.5: setting latency timer to 64
aer 0000:00:03.0:pcie02: service driver aer loaded
aer 0000:00:05.0:pcie02: service driver aer loaded
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
Firmware did not grant requested _OSC control
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
ACPI: Power Button [PWRB]
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
ACPI: Power Button [PWRF]
ACPI: SSDT 00000000bf7b0180 00F20 (v01 DpgPmm P001Ist 00000011 INTL 20051117)
ACPI: SSDT 00000000bf7b10a0 004D5 (v01 PmRef P001Cst 00003001 INTL 20051117)
Monitor-Mwait will be used to enter C-1 state
Monitor-Mwait will be used to enter C-2 state
Monitor-Mwait will be used to enter C-3 state
processor LNXCPU:00: registered as cooling_device0
processor LNXCPU:01: registered as cooling_device1
processor LNXCPU:02: registered as cooling_device2
processor LNXCPU:03: registered as cooling_device3
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
serial8250: ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A
00:06: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:07: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:08: ttyS2 at I/O 0x3e8 (irq = 5) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input2
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
alloc irq_desc for 21 on node -1
alloc kstat_irqs on node -1
ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
ehci_hcd 0000:00:1a.0: setting latency timer to 64
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1a.0: irq 21, io mem 0xfb4fc000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-71.29.1.el6.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
alloc irq_desc for 23 on node -1
alloc kstat_irqs on node -1
ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.0: setting latency timer to 64
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported
ehci_hcd 0000:00:1d.0: irq 23, io mem 0xfb4fa000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-71.29.1.el6.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:03: RTC can wake from S4
rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:03: setting system clock to 2011-10-26 06:35:22 UTC (1319610922)
Initalizing network drop monitor service
Freeing unused kernel memory: 1220k freed
Write protecting the kernel read-only data: 7184k
dracut: dracut-004-32.el6
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.17.0-ioctl (2010-03-05) initialised: dm-devel at redhat.com
udev: starting version 147
dracut: Starting plymouth daemon
ata_piix 0000:00:1f.2: version 2.13
alloc irq_desc for 19 on node -1
alloc kstat_irqs on node -1
ata_piix 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
ata_piix 0000:00:1f.2: setting latency timer to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0xcc00 ctl 0xc880 bmdma 0xc400 irq 19
ata2: SATA max UDMA/133 cmd 0xc800 ctl 0xc480 bmdma 0xc408 irq 19
ata_piix 0000:00:1f.5: PCI INT B -> GSI 19 (level, low) -> IRQ 19
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
ata_piix 0000:00:1f.5: setting latency timer to 64
scsi2 : ata_piix
scsi3 : ata_piix
ata3: SATA max UDMA/133 cmd 0xbc00 ctl 0xb880 bmdma 0xb400 irq 19
ata4: SATA max UDMA/133 cmd 0xb800 ctl 0xb480 bmdma 0xb408 irq 19
usb 1-1: new high speed USB device using ehci_hcd and address 2
usb 1-1: New USB device found, idVendor=8087, idProduct=0020
usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1: configuration #1 chosen from 1 choice
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
ata3: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
usb 2-1: new high speed USB device using ehci_hcd and address 2
usb 2-1: New USB device found, idVendor=8087, idProduct=0020
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
usb 1-1.2: new full speed USB device using ehci_hcd and address 3
usb 1-1.2: New USB device found, idVendor=0557, idProduct=2221
usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-1.2: Product: Hermon USB hidmouse Device
usb 1-1.2: Manufacturer: Winbond Electronics Corp
usb 1-1.2: configuration #1 chosen from 1 choice
input: Winbond Electronics Corp Hermon USB hidmouse Device as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/input/input3
generic-usb 0003:0557:2221.0001: input,hidraw0: USB HID v1.00 Mouse [Winbond Electronics Corp Hermon USB hidmouse Device] on usb-0000:00:1a.0-1.2/input0
input: Winbond Electronics Corp Hermon USB hidmouse Device as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.1/input/input4
generic-usb 0003:0557:2221.0002: input,hidraw1: USB HID v1.00 Keyboard [Winbond Electronics Corp Hermon USB hidmouse Device] on usb-0000:00:1a.0-1.2/input1
ata2.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.01: SATA link down (SStatus 0 SControl 300)
ata1.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-8: ST1000NM0011, SN02, max UDMA/133
ata2.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.00: ATA-8: ST1000NM0011, SN02, max UDMA/133
ata1.00: 1953525168 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.01: ATA-8: ST250DM000-1BD141, KC43, max UDMA/133
ata1.01: 488397168 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata2.00: configured for UDMA/133
ata1.00: configured for UDMA/133
ata1.01: configured for UDMA/133
scsi 0:0:0:0: Direct-Access ATA ST1000NM0011 SN02 PQ: 0 ANSI: 5
scsi 0:0:1:0: Direct-Access ATA ST250DM000-1BD14 KC43 PQ: 0 ANSI: 5
scsi 1:0:0:0: Direct-Access ATA ST1000NM0011 SN02 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda:
sd 1:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
sd 1:0:0:0: [sdc] Write Protect is off
sd 1:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdc:
sd 0:0:1:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB)
sd 0:0:1:0: [sdb] 4096-byte physical blocks
sda1 sda2 sda3
sd 0:0:1:0: [sdb] Write Protect is off
sd 0:0:1:0: [sdb] Mode Sense: 00 3a 00 00
sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sdb: sdb1 sdb2
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:1:0: [sdb] Attached SCSI disk
sdc1 sdc2 sdc3
sd 1:0:0:0: [sdc] Attached SCSI disk
dracut: Autoassembling MD Raid
md: md0 stopped.
md: bind<sdc1>
md: bind<sda1>
md: raid1 personality registered for level 1
raid1: raid set md0 active with 2 out of 2 mirrors
md0: detected capacity change from 0 to 41940549632
dracut: mdadm: /dev/md0 has been started with 2 drives.
md0: detected capacity change from 0 to 41940549632
md0: unknown partition table
md: md1 stopped.
md: bind<sdc2>
md: bind<sda2>
raid1: raid set md1 active with 2 out of 2 mirrors
md1: detected capacity change from 0 to 33550827520
dracut: mdadm: /dev/md1 has been started with 2 drives.
md1: detected capacity change from 0 to 33550827520
md1: unknown partition table
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
dracut: Mounted root filesystem /dev/md0
dracut: Loading SELinux policy
SELinux: 2048 avtab hash slots, 199741 rules.
SELinux: 2048 avtab hash slots, 199741 rules.
SELinux: 9 users, 13 roles, 3323 types, 168 bools, 1 sens, 1024 cats
SELinux: 77 classes, 199741 rules
SELinux: Completing initialization.
SELinux: Setting up existing superblocks.
SELinux: initialized (dev md0, type ext3), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev usbfs, type usbfs), uses genfs_contexts
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev devtmpfs, type devtmpfs), uses transition SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
type=1403 audit(1319610924.741:2): policy loaded auid=4294967295 ses=4294967295
dracut:
dracut: Switching root
udev: starting version 147
e1000e: Intel(R) PRO/1000 Network Driver - 1.5.1-NAPI
e1000e: Copyright(c) 1999 - 2011 Intel Corporation.
e1000e 0000:04:00.0: Disabling ASPM L0s
e1000e 0000:04:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
e1000e 0000:04:00.0: setting latency timer to 64
alloc irq_desc for 34 on node -1
alloc kstat_irqs on node -1
e1000e 0000:04:00.0: irq 34 for MSI/MSI-X
alloc irq_desc for 35 on node -1
alloc kstat_irqs on node -1
e1000e 0000:04:00.0: irq 35 for MSI/MSI-X
e1000e 0000:04:00.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:25:90:33:5b:48
e1000e 0000:04:00.0: eth0: Intel(R) PRO/1000 Network Connection
e1000e 0000:04:00.0: eth0: MAC: 4, PHY: 8, PBA No: 0101FF-0FF
e1000e 0000:05:00.0: Disabling ASPM L0s
e1000e 0000:05:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
e1000e 0000:05:00.0: setting latency timer to 64
alloc irq_desc for 36 on node -1
alloc kstat_irqs on node -1
e1000e 0000:05:00.0: irq 36 for MSI/MSI-X
alloc irq_desc for 37 on node -1
alloc kstat_irqs on node -1
e1000e 0000:05:00.0: irq 37 for MSI/MSI-X
e1000e 0000:05:00.0: eth1: (PCI Express:2.5GT/s:Width x1) 00:25:90:33:5b:49
e1000e 0000:05:00.0: eth1: Intel(R) PRO/1000 Network Connection
e1000e 0000:05:00.0: eth1: MAC: 4, PHY: 8, PBA No: 0101FF-0FF
sd 0:0:0:0: Attached scsi generic sg0 type 0
sd 0:0:1:0: Attached scsi generic sg1 type 0
sd 1:0:0:0: Attached scsi generic sg2 type 0
alloc irq_desc for 18 on node -1
alloc kstat_irqs on node -1
i801_smbus 0000:00:1f.3: PCI INT C -> GSI 18 (level, low) -> IRQ 18
iTCO_vendor_support: vendor-support=0
iTCO_wdt: Intel TCO WatchDog Timer Driver v1.05
iTCO_wdt: Found a 3420 TCO device (Version=2, TCOBASE=0x0860)
iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
md: bind<sda3>
md: bind<sdc3>
raid1: raid set md127 active with 2 out of 2 mirrors
md127: detected capacity change from 0 to 924709465088
md127: unknown partition table
EXT3 FS on md0, internal journal
Adding 32764472k swap on /dev/md1. Priority:-1 extents:1 across:32764472k
SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
ADDRCONF(NETDEV_UP): eth0: link is not ready
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
ADDRCONF(NETDEV_UP): eth1: link is not ready
e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
eth0: no IPv6 routers present
eth1: no IPv6 routers present
events: mcg drbd: 2
drbd: initialized. Version: 8.4.0 (api:1/proto:86-100)
drbd: GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root at bussqlha1, 2011-09-30 07:02:41
drbd: registered as block device major 147
d-con r1: Starting worker thread (from drbdsetup [1739])
block drbd0: disk( Diskless -> Attaching )
block drbd0: Method to ensure write ordering: barrier
block drbd0: max BIO size = 130560
block drbd0: drbd_bm_resize called with capacity == 419430400
block drbd0: resync bitmap: bits=52428800 words=819200 pages=1600
block drbd0: size = 200 GB (209715200 KB)
block drbd0: bitmap READ of 1600 pages took 57 jiffies
block drbd0: recounting of set bits took additional 7 jiffies
block drbd0: 96 KB (24 bits) marked out-of-sync by on disk bit-map.
block drbd0: disk( Attaching -> UpToDate )
block drbd0: attached to UUIDs 9E106736E7ED5E4F:7D00401111D97917:D149796E40DBA133:D148796E40DBA133
d-con r1: conn( StandAlone -> Unconnected )
d-con r1: Starting receiver thread (from drbd_w_r1 [1740])
d-con r1: receiver (re)started
d-con r1: conn( Unconnected -> WFConnection )
d-con r1: Handshake successful: Agreed network protocol version 100
d-con r1: Peer authenticated using 20 bytes HMAC
d-con r1: conn( WFConnection -> WFReportParams )
d-con r1: Starting asender thread (from drbd_r_r1 [1757])
block drbd0: drbd_sync_handshake:
block drbd0: self 9E106736E7ED5E4E:7D00401111D97917:D149796E40DBA133:D148796E40DBA133 bits:24 flags:0
block drbd0: peer 09D81D5543CBB23B:7D00401111D97916:D149796E40DBA132:D148796E40DBA133 bits:5980 flags:0
block drbd0: uuid_compare()=100 by rule 90
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
block drbd0: Split-Brain detected but unresolved, dropping connection!
block drbd0: helper command: /sbin/drbdadm split-brain minor-0
block drbd0: helper command: /sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
d-con r1: conn( WFReportParams -> Disconnecting )
d-con r1: error receiving ReportState, e: -5 l: 0!
d-con r1: asender terminated
d-con r1: Terminating asender thread
d-con r1: Connection closed
d-con r1: conn( Disconnecting -> StandAlone )
d-con r1: receiver terminated
d-con r1: Terminating receiver thread
d-con r1: conn( StandAlone -> Unconnected )
d-con r1: Starting receiver thread (from drbd_w_r1 [1740])
d-con r1: receiver (re)started
d-con r1: conn( Unconnected -> WFConnection )
d-con r1: Handshake successful: Agreed network protocol version 100
d-con r1: Peer authenticated using 20 bytes HMAC
d-con r1: conn( WFConnection -> WFReportParams )
d-con r1: Starting asender thread (from drbd_r_r1 [2179])
block drbd0: drbd_sync_handshake:
block drbd0: self 9E106736E7ED5E4E:7D00401111D97917:D149796E40DBA133:D148796E40DBA133 bits:24 flags:0
block drbd0: peer 09D81D5543CBB23B:7D00401111D97916:D149796E40DBA132:D148796E40DBA133 bits:6046 flags:0
block drbd0: uuid_compare()=100 by rule 90
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0
block drbd0: helper command: /sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
block drbd0: Split-Brain detected, manually solved. Sync from peer node
block drbd0: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )
block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 17033(5), total 17033; compression: 99.8%
block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 17043(5), total 17043; compression: 99.8%
block drbd0: conn( WFBitMapT -> WFSyncUUID )
block drbd0: updated sync uuid 7D01401111D97916:0000000000000000:D149796E40DBA133:D148796E40DBA133
block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0
block drbd0: helper command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
block drbd0: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
block drbd0: Began resync as SyncTarget (will sync 24248 KB [6062 bits set]).
block drbd0: Resync done (total 28 sec; paused 0 sec; 864 K/sec)
block drbd0: updated UUIDs 09D81D5543CBB23A:0000000000000000:7D01401111D97916:7D00401111D97916
block drbd0: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0
block drbd0: helper command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
block drbd0: bitmap WRITE of 0 pages took 0 jiffies
block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
block drbd0: peer( Primary -> Secondary )
d-con r1: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )
d-con r1: asender terminated
d-con r1: Terminating asender thread
d-con r1: conn( TearDown -> Disconnecting )
d-con r1: Connection closed
d-con r1: conn( Disconnecting -> StandAlone )
d-con r1: receiver terminated
d-con r1: Terminating receiver thread
block drbd0: disk( UpToDate -> Failed )
block drbd0: disk( Failed -> Diskless )
block drbd0: drbd_bm_resize called with capacity == 0
d-con r1: Terminating worker thread
d-con r1: Starting worker thread (from drbdsetup [4598])
block drbd0: disk( Diskless -> Attaching )
block drbd0: Method to ensure write ordering: barrier
block drbd0: max BIO size = 130560
block drbd0: drbd_bm_resize called with capacity == 419430400
block drbd0: resync bitmap: bits=52428800 words=819200 pages=1600
block drbd0: size = 200 GB (209715200 KB)
block drbd0: bitmap READ of 1600 pages took 101 jiffies
block drbd0: recounting of set bits took additional 7 jiffies
block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
block drbd0: disk( Attaching -> UpToDate )
block drbd0: attached to UUIDs 09D81D5543CBB23A:0000000000000000:7D01401111D97916:7D00401111D97916
d-con r1: conn( StandAlone -> Unconnected )
d-con r1: Starting receiver thread (from drbd_w_r1 [4599])
d-con r1: receiver (re)started
d-con r1: conn( Unconnected -> WFConnection )
block drbd0: role( Secondary -> Primary )
block drbd0: new current UUID 49F8A559F608C6F5:09D81D5543CBB23A:7D01401111D97916:7D00401111D97916
kjournald starting. Commit interval 5 seconds
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3 FS on drbd0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev drbd0, type ext3), uses xattr
d-con r1: Handshake successful: Agreed network protocol version 100
d-con r1: Peer authenticated using 20 bytes HMAC
d-con r1: conn( WFConnection -> WFReportParams )
d-con r1: Starting asender thread (from drbd_r_r1 [4611])
block drbd0: drbd_sync_handshake:
block drbd0: self 49F8A559F608C6F5:09D81D5543CBB23A:7D01401111D97916:7D00401111D97916 bits:1 flags:0
block drbd0: peer 09D81D5543CBB23A:0000000000000000:7D01401111D97916:7D00401111D97916 bits:0 flags:0
block drbd0: uuid_compare()=1 by rule 70
block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%
block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
block drbd0: Began resync as SyncSource (will sync 4 KB [1 bits set]).
block drbd0: updated sync UUID 49F8A559F608C6F5:09D91D5543CBB23A:09D81D5543CBB23A:7D01401111D97916
block drbd0: Resync done (total 1 sec; paused 0 sec; 4 K/sec)
block drbd0: updated UUIDs 49F8A559F608C6F5:0000000000000000:09D91D5543CBB23A:09D81D5543CBB23A
block drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
block drbd0: bitmap WRITE of 0 pages took 0 jiffies
block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
JBD: barrier-based sync failed on drbd0 - disabling barriers
d-con r1: peer( Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )
d-con r1: asender terminated
d-con r1: Terminating asender thread
block drbd0: new current UUID 36C25BAEFD88C481:49F8A559F608C6F5:09D91D5543CBB23A:09D81D5543CBB23A
d-con r1: Connection closed
d-con r1: conn( Disconnecting -> StandAlone )
d-con r1: receiver terminated
d-con r1: Terminating receiver thread
INFO: task drbd_w_r1:4599 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
drbd_w_r1 D ffff880818639900 0 4599 2 0x00000080
ffff88080dd93c70 0000000000000046 ffff88081a9b8af8 ffff8800282569f0
ffff88080dd93bf0 ffffffff81056720 ffff88080dd93c40 000000000000056c
ffff88081a9b8638 ffff88080dd93fd8 0000000000010518 ffff88081a9b8638
Call Trace:
[<ffffffff81056720>] ? __dequeue_entity+0x30/0x50
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02098e5>] bm_page_io_async+0xe5/0x370 [drbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa020b8c2>] bm_rw+0x1a2/0x680 [drbd]
[<ffffffffa0202056>] ? crc32c+0x56/0x7c [libcrc32c]
[<ffffffffa020bdba>] drbd_bm_write_hinted+0x1a/0x20 [drbd]
[<ffffffffa0224602>] _al_write_transaction+0x2c2/0x6a0 [drbd]
[<ffffffffa0224d42>] w_al_write_transaction+0x22/0x50 [drbd]
[<ffffffffa020e85e>] drbd_worker+0x10e/0x480 [drbd]
[<ffffffffa022aa19>] drbd_thread_setup+0xa9/0x160 [drbd]
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffffa022a970>] ? drbd_thread_setup+0x0/0x160 [drbd]
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task kjournald:4785 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D 0000000000000001 0 4785 2 0x00000080
ffff88080fdf9a50 0000000000000046 0000000000000000 ffff88080fdf9a14
ffff88080fdf99c0 ffff88083fc24500 ffff880028256980 0000000102e3c2f3
ffff88081a450638 ffff88080fdf9fd8 0000000000010518 ffff88081a450638
Call Trace:
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0224c14>] drbd_al_begin_io+0x164/0x270 [drbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0221197>] __drbd_make_request+0xb57/0x1230 [drbd]
[<ffffffffa02219f9>] drbd_make_request+0x189/0x300 [drbd]
[<ffffffff81241982>] generic_make_request+0x1b2/0x4f0
[<ffffffff8110e925>] ? mempool_alloc_slab+0x15/0x20
[<ffffffffa006827e>] ? journal_file_buffer+0x4e/0x90 [jbd]
[<ffffffff81241d4f>] submit_bio+0x8f/0x120
[<ffffffff8119d704>] submit_bh+0xf4/0x140
[<ffffffffa0069eb9>] journal_commit_transaction+0x8c9/0x12b0 [jbd]
[<ffffffff8107d73c>] ? lock_timer_base+0x3c/0x70
[<ffffffff8107e1eb>] ? try_to_del_timer_sync+0x7b/0xe0
[<ffffffffa006dad8>] kjournald+0xe8/0x250 [jbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa006d9f0>] ? kjournald+0x0/0x250 [jbd]
[<ffffffff81091b36>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff81091aa0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task flush-147:0:10400 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-147:0 D ffff880776161590 0 10400 2 0x00000080
ffff8807761614b0 0000000000000046 ffff880776161478 ffff880776161474
ffff880000040d80 ffff88083fc24500 ffff880028256980 0000000102e3c232
ffff8807761870e8 ffff880776161fd8 0000000000010518 ffff8807761870e8
Call Trace:
[<ffffffff8105c846>] ? update_curr+0xe6/0x1e0
[<ffffffff814c9ff5>] schedule_timeout+0x225/0x2f0
[<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
[<ffffffff810508e8>] ? resched_task+0x68/0x80
[<ffffffff81067428>] ? check_preempt_wakeup+0x2b8/0x3c0
[<ffffffff814c9c63>] wait_for_common+0x123/0x180
[<ffffffff8105c580>] ? default_wake_function+0x0/0x20
[<ffffffff814cadaa>] ? __up+0x2a/0x40
[<ffffffff814c9d7d>] wait_for_completion+0x1d/0x20
[<ffffffffa0224a8d>] al_write_transaction+0xad/0xd0 [drbd]
[<ffffffffa0224d20>] ? w_al_write_transaction+0x0/0x50 [drbd]
[<ffffffffa0222714>] ? _al_get+0x54/0xc0 [drbd]
[<ffffffffa0224caf>] drbd_al_begin_io+0x1ff/0x270 [drbd]
[<ffffffff811a2206>] ? __bio_clone+0x26/0x70
[<ffffffffa0221197>] __drbd_make_request+0xb57/0x1230 [drbd]
[<ffffffff81155e6c>] ? transfer_objects+0x5c/0x80
[<ffffffffa02219f9>] drbd_make_request+0x189/0x300 [drbd]
[<ffffffff81241982>] generic_make_request+0x1b2/0x4f0
[<ffffffff8110e925>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81241d4f>] submit_bio+0x8f/0x120
[<ffffffff8119d704>] submit_bh+0xf4/0x140
[<ffffffff8119f700>] __block_write_full_page+0x1e0/0x3b0
[<ffffffff8119ed90>] ? end_buffer_async_write+0x0/0x190
[<ffffffff811a3a30>] ? blkdev_get_block+0x0/0x70
[<ffffffff811a3a30>] ? blkdev_get_block+0x0/0x70
[<ffffffff811a0000>] block_write_full_page_endio+0xe0/0x120
[<ffffffff8110be40>] ? find_get_pages_tag+0x40/0x120
[<ffffffff811a0055>] block_write_full_page+0x15/0x20
[<ffffffff811a49c8>] blkdev_writepage+0x18/0x20
[<ffffffff8111f6e7>] __writepage+0x17/0x40
[<ffffffff81120979>] write_cache_pages+0x189/0x430
[<ffffffff8111f6d0>] ? __writepage+0x0/0x40
[<ffffffff81120c44>] generic_writepages+0x24/0x30
[<ffffffff81120c71>] do_writepages+0x21/0x40
[<ffffffff8119694d>] writeback_single_inode+0xdd/0x2c0
[<ffffffff81196d4e>] writeback_sb_inodes+0xce/0x180
[<ffffffff81196ea3>] writeback_inodes_wb+0xa3/0x1a0
[<ffffffff8119723b>] wb_writeback+0x29b/0x3f0
[<ffffffff814c92b6>] ? thread_return+0x4e/0x778
[<ffffffff8107e272>] ? del_timer_sync+0x22/0x30
[<ffffffff81197529>] wb_do_writeback+0x199/0x240
[<ffffffff81197633>] bdi_writeback_task+0x63/0x1b0
[<ffffffff81091d67>] ? bit_waitqueue+0x17/0xd0
[<ffffffff8112f040>] ? bdi_start_fn+0x0/0x100
[<ffffffff8112f0c6>] bdi_start_fn+0x86/0x100
[<ffffffff8112f040>] ? bdi_start_fn+0x0/0x100
[<ffffffff81091b36>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff81091aa0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task mysqld:11428 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mysqld D 0000000000000000 0 11428 11173 0x00000080
ffff88077623da08 0000000000000086 ffff88077623d9a8 ffffffff81266fdc
7300646c7173796d 0000000800656661 0000000000000000 ffff88077623dbc4
ffff88081a44f028 ffff88077623dfd8 0000000000010518 ffff88081a44f028
Call Trace:
[<ffffffff81266fdc>] ? __bitmap_weight+0x8c/0xb0
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00690f5>] do_get_write_access+0x2c5/0x500 [jbd]
[<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00694c1>] journal_get_write_access+0x31/0x50 [jbd]
[<ffffffffa009273d>] __ext3_journal_get_write_access+0x2d/0x60 [ext3]
[<ffffffffa0083d4b>] ext3_reserve_inode_write+0x7b/0xa0 [ext3]
[<ffffffffa0083da6>] ext3_mark_inode_dirty+0x36/0x60 [ext3]
[<ffffffffa0083f51>] ext3_dirty_inode+0x61/0xa0 [ext3]
[<ffffffff8119633b>] __mark_inode_dirty+0x3b/0x160
[<ffffffff81186d82>] file_update_time+0xf2/0x170
[<ffffffff8110e4f0>] __generic_file_aio_write+0x220/0x480
[<ffffffff8105c484>] ? try_to_wake_up+0x284/0x380
[<ffffffff8105c5b0>] ? wake_up_state+0x10/0x20
[<ffffffff8110e7bf>] generic_file_aio_write+0x6f/0xe0
[<ffffffff8116d05a>] do_sync_write+0xfa/0x140
[<ffffffff81056720>] ? __dequeue_entity+0x30/0x50
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8106761c>] ? pick_next_task_fair+0xec/0x120
[<ffffffff8120caab>] ? selinux_file_permission+0xfb/0x150
[<ffffffff811fff16>] ? security_file_permission+0x16/0x20
[<ffffffff8116d358>] vfs_write+0xb8/0x1a0
[<ffffffff810d4372>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff8116de52>] sys_pwrite64+0x82/0xa0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
INFO: task mysqld:15359 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mysqld D 0000000000000000 0 15359 11173 0x00000080
ffff88080ff7da08 0000000000000086 ffff88080ff7d978 ffffffff814c9a72
ffff88080ff7d9e8 ffffffff81155e6c ffff880028033a98 ffff880817742880
ffff880818765a98 ffff88080ff7dfd8 0000000000010518 ffff880818765a98
Call Trace:
[<ffffffff814c9a72>] ? io_schedule+0x92/0xc0
[<ffffffff81155e6c>] ? transfer_objects+0x5c/0x80
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00690f5>] do_get_write_access+0x2c5/0x500 [jbd]
[<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00694c1>] journal_get_write_access+0x31/0x50 [jbd]
[<ffffffffa009273d>] __ext3_journal_get_write_access+0x2d/0x60 [ext3]
[<ffffffffa0083d4b>] ext3_reserve_inode_write+0x7b/0xa0 [ext3]
[<ffffffffa0083da6>] ext3_mark_inode_dirty+0x36/0x60 [ext3]
[<ffffffffa0083f51>] ext3_dirty_inode+0x61/0xa0 [ext3]
[<ffffffff8119633b>] __mark_inode_dirty+0x3b/0x160
[<ffffffff81186d82>] file_update_time+0xf2/0x170
[<ffffffff8110e4f0>] __generic_file_aio_write+0x220/0x480
[<ffffffff8105c5b0>] ? wake_up_state+0x10/0x20
[<ffffffff8110e7bf>] generic_file_aio_write+0x6f/0xe0
[<ffffffff8116d05a>] do_sync_write+0xfa/0x140
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8120caab>] ? selinux_file_permission+0xfb/0x150
[<ffffffff811fff16>] ? security_file_permission+0x16/0x20
[<ffffffff8116d358>] vfs_write+0xb8/0x1a0
[<ffffffff810d4372>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff8116de52>] sys_pwrite64+0x82/0xa0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
INFO: task mysqld:15550 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mysqld D ffff88083fc24500 0 15550 11173 0x00000080
ffff880281bd9c58 0000000000000086 0000000000000000 ffff88083fc10080
ffff88081ca30c00 ffff88083fc224c0 ffff880281bd9c38 0000000102e3e0d3
ffff88074868c678 ffff880281bd9fd8 0000000000010518 ffff88074868c678
Call Trace:
[<ffffffffa00690f5>] do_get_write_access+0x2c5/0x500 [jbd]
[<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00694c1>] journal_get_write_access+0x31/0x50 [jbd]
[<ffffffffa009273d>] __ext3_journal_get_write_access+0x2d/0x60 [ext3]
[<ffffffffa0083d4b>] ext3_reserve_inode_write+0x7b/0xa0 [ext3]
[<ffffffffa0083da6>] ext3_mark_inode_dirty+0x36/0x60 [ext3]
[<ffffffffa0083f51>] ext3_dirty_inode+0x61/0xa0 [ext3]
[<ffffffff8119633b>] __mark_inode_dirty+0x3b/0x160
[<ffffffff81186f2d>] touch_atime+0x12d/0x170
[<ffffffff81180e80>] ? filldir+0x0/0xe0
[<ffffffff81181116>] vfs_readdir+0xd6/0xe0
[<ffffffff81181289>] sys_getdents+0x89/0xf0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
INFO: task drbd_w_r1:4599 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
drbd_w_r1 D ffff880818639900 0 4599 2 0x00000080
ffff88080dd93c70 0000000000000046 ffff88081a9b8af8 ffff8800282569f0
ffff88080dd93bf0 ffffffff81056720 ffff88080dd93c40 000000000000056c
ffff88081a9b8638 ffff88080dd93fd8 0000000000010518 ffff88081a9b8638
Call Trace:
[<ffffffff81056720>] ? __dequeue_entity+0x30/0x50
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa02098e5>] bm_page_io_async+0xe5/0x370 [drbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa020b8c2>] bm_rw+0x1a2/0x680 [drbd]
[<ffffffffa0202056>] ? crc32c+0x56/0x7c [libcrc32c]
[<ffffffffa020bdba>] drbd_bm_write_hinted+0x1a/0x20 [drbd]
[<ffffffffa0224602>] _al_write_transaction+0x2c2/0x6a0 [drbd]
[<ffffffffa0224d42>] w_al_write_transaction+0x22/0x50 [drbd]
[<ffffffffa020e85e>] drbd_worker+0x10e/0x480 [drbd]
[<ffffffffa022aa19>] drbd_thread_setup+0xa9/0x160 [drbd]
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffffa022a970>] ? drbd_thread_setup+0x0/0x160 [drbd]
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task kjournald:4785 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald D 0000000000000001 0 4785 2 0x00000080
ffff88080fdf9a50 0000000000000046 0000000000000000 ffff88080fdf9a14
ffff88080fdf99c0 ffff88083fc24500 ffff880028256980 0000000102e3c2f3
ffff88081a450638 ffff88080fdf9fd8 0000000000010518 ffff88081a450638
Call Trace:
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa0224c14>] drbd_al_begin_io+0x164/0x270 [drbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0221197>] __drbd_make_request+0xb57/0x1230 [drbd]
[<ffffffffa02219f9>] drbd_make_request+0x189/0x300 [drbd]
[<ffffffff81241982>] generic_make_request+0x1b2/0x4f0
[<ffffffff8110e925>] ? mempool_alloc_slab+0x15/0x20
[<ffffffffa006827e>] ? journal_file_buffer+0x4e/0x90 [jbd]
[<ffffffff81241d4f>] submit_bio+0x8f/0x120
[<ffffffff8119d704>] submit_bh+0xf4/0x140
[<ffffffffa0069eb9>] journal_commit_transaction+0x8c9/0x12b0 [jbd]
[<ffffffff8107d73c>] ? lock_timer_base+0x3c/0x70
[<ffffffff8107e1eb>] ? try_to_del_timer_sync+0x7b/0xe0
[<ffffffffa006dad8>] kjournald+0xe8/0x250 [jbd]
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa006d9f0>] ? kjournald+0x0/0x250 [jbd]
[<ffffffff81091b36>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff81091aa0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task flush-147:0:10400 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-147:0 D ffff880776161590 0 10400 2 0x00000080
ffff8807761614b0 0000000000000046 ffff880776161478 ffff880776161474
ffff880000040d80 ffff88083fc24500 ffff880028256980 0000000102e3c232
ffff8807761870e8 ffff880776161fd8 0000000000010518 ffff8807761870e8
Call Trace:
[<ffffffff8105c846>] ? update_curr+0xe6/0x1e0
[<ffffffff814c9ff5>] schedule_timeout+0x225/0x2f0
[<ffffffff8102ea69>] ? native_smp_send_reschedule+0x49/0x60
[<ffffffff810508e8>] ? resched_task+0x68/0x80
[<ffffffff81067428>] ? check_preempt_wakeup+0x2b8/0x3c0
[<ffffffff814c9c63>] wait_for_common+0x123/0x180
[<ffffffff8105c580>] ? default_wake_function+0x0/0x20
[<ffffffff814cadaa>] ? __up+0x2a/0x40
[<ffffffff814c9d7d>] wait_for_completion+0x1d/0x20
[<ffffffffa0224a8d>] al_write_transaction+0xad/0xd0 [drbd]
[<ffffffffa0224d20>] ? w_al_write_transaction+0x0/0x50 [drbd]
[<ffffffffa0222714>] ? _al_get+0x54/0xc0 [drbd]
[<ffffffffa0224caf>] drbd_al_begin_io+0x1ff/0x270 [drbd]
[<ffffffff811a2206>] ? __bio_clone+0x26/0x70
[<ffffffffa0221197>] __drbd_make_request+0xb57/0x1230 [drbd]
[<ffffffff81155e6c>] ? transfer_objects+0x5c/0x80
[<ffffffffa02219f9>] drbd_make_request+0x189/0x300 [drbd]
[<ffffffff81241982>] generic_make_request+0x1b2/0x4f0
[<ffffffff8110e925>] ? mempool_alloc_slab+0x15/0x20
[<ffffffff81241d4f>] submit_bio+0x8f/0x120
[<ffffffff8119d704>] submit_bh+0xf4/0x140
[<ffffffff8119f700>] __block_write_full_page+0x1e0/0x3b0
[<ffffffff8119ed90>] ? end_buffer_async_write+0x0/0x190
[<ffffffff811a3a30>] ? blkdev_get_block+0x0/0x70
[<ffffffff811a3a30>] ? blkdev_get_block+0x0/0x70
[<ffffffff811a0000>] block_write_full_page_endio+0xe0/0x120
[<ffffffff8110be40>] ? find_get_pages_tag+0x40/0x120
[<ffffffff811a0055>] block_write_full_page+0x15/0x20
[<ffffffff811a49c8>] blkdev_writepage+0x18/0x20
[<ffffffff8111f6e7>] __writepage+0x17/0x40
[<ffffffff81120979>] write_cache_pages+0x189/0x430
[<ffffffff8111f6d0>] ? __writepage+0x0/0x40
[<ffffffff81120c44>] generic_writepages+0x24/0x30
[<ffffffff81120c71>] do_writepages+0x21/0x40
[<ffffffff8119694d>] writeback_single_inode+0xdd/0x2c0
[<ffffffff81196d4e>] writeback_sb_inodes+0xce/0x180
[<ffffffff81196ea3>] writeback_inodes_wb+0xa3/0x1a0
[<ffffffff8119723b>] wb_writeback+0x29b/0x3f0
[<ffffffff814c92b6>] ? thread_return+0x4e/0x778
[<ffffffff8107e272>] ? del_timer_sync+0x22/0x30
[<ffffffff81197529>] wb_do_writeback+0x199/0x240
[<ffffffff81197633>] bdi_writeback_task+0x63/0x1b0
[<ffffffff81091d67>] ? bit_waitqueue+0x17/0xd0
[<ffffffff8112f040>] ? bdi_start_fn+0x0/0x100
[<ffffffff8112f0c6>] bdi_start_fn+0x86/0x100
[<ffffffff8112f040>] ? bdi_start_fn+0x0/0x100
[<ffffffff81091b36>] kthread+0x96/0xa0
[<ffffffff810141ca>] child_rip+0xa/0x20
[<ffffffff81091aa0>] ? kthread+0x0/0xa0
[<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task mysqld:11428 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mysqld D 0000000000000000 0 11428 11173 0x00000080
ffff88077623da08 0000000000000086 ffff88077623d9a8 ffffffff81266fdc
7300646c7173796d 0000000800656661 0000000000000000 ffff88077623dbc4
ffff88081a44f028 ffff88077623dfd8 0000000000010518 ffff88081a44f028
Call Trace:
[<ffffffff81266fdc>] ? __bitmap_weight+0x8c/0xb0
[<ffffffff8109218e>] ? prepare_to_wait+0x4e/0x80
[<ffffffffa00690f5>] do_get_write_access+0x2c5/0x500 [jbd]
[<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
[<ffffffffa00694c1>] journal_get_write_access+0x31/0x50 [jbd]
[<ffffffffa009273d>] __ext3_journal_get_write_access+0x2d/0x60 [ext3]
[<ffffffffa0083d4b>] ext3_reserve_inode_write+0x7b/0xa0 [ext3]
[<ffffffffa0083da6>] ext3_mark_inode_dirty+0x36/0x60 [ext3]
[<ffffffffa0083f51>] ext3_dirty_inode+0x61/0xa0 [ext3]
[<ffffffff8119633b>] __mark_inode_dirty+0x3b/0x160
[<ffffffff81186d82>] file_update_time+0xf2/0x170
[<ffffffff8110e4f0>] __generic_file_aio_write+0x220/0x480
[<ffffffff8105c484>] ? try_to_wake_up+0x284/0x380
[<ffffffff8105c5b0>] ? wake_up_state+0x10/0x20
[<ffffffff8110e7bf>] generic_file_aio_write+0x6f/0xe0
[<ffffffff8116d05a>] do_sync_write+0xfa/0x140
[<ffffffff81056720>] ? __dequeue_entity+0x30/0x50
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8106761c>] ? pick_next_task_fair+0xec/0x120
[<ffffffff8120caab>] ? selinux_file_permission+0xfb/0x150
[<ffffffff811fff16>] ? security_file_permission+0x16/0x20
[<ffffffff8116d358>] vfs_write+0xb8/0x1a0
[<ffffffff810d4372>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff8116de52>] sys_pwrite64+0x82/0xa0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b