Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
We have problems with our three node DRBD 9 setup with CentOS 7 and Xen 4. When one of our nodes is rebooted, or becomes unavailable, the other nodes freeze entirely without any information, or give the following message: kernel:NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [drbd_w_db:1469] They then require a reboot, sometimes crashing one of the other nodes again in the process. Not a fun thing in a HA setup... Does anyone have an idea what is going on, and what we can do to prevent this from happening? We are running: DRBD 9.0.4-1 (api:2/proto:86-112) CentOS 3.18.41-20.el7.x86_64 Xen 4.6.3-3.el7 I do not know if it is related, but we are using bonding (mode 1, active-backup) with two network adapters. /var/log/messages: Oct 15 03:54:13 xen1 kernel: drbd db/0 drbd105 xen2: pdsk( UpToDate -> Consistent ) Oct 15 03:54:14 xen1 kernel: drbd db xen3: PingAck did not arrive in time. Oct 15 03:54:14 xen1 kernel: drbd db xen3: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) Oct 15 03:54:14 xen1 kernel: drbd db/0 drbd105: disk( UpToDate -> Consistent ) Oct 15 03:54:14 xen1 kernel: drbd db/0 drbd105 xen3: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) Oct 15 03:54:14 xen1 kernel: drbd db xen3: ack_receiver terminated Oct 15 03:54:14 xen1 kernel: drbd db xen3: Terminating ack_recv thread Oct 15 03:54:14 xen1 kernel: drbd db: Preparing cluster-wide state change 156852247 (2->-1 0/0) Oct 15 03:54:14 xen1 kernel: drbd db xen3: Connection closed Oct 15 03:54:14 xen1 kernel: drbd db xen3: conn( NetworkFailure -> Unconnected ) Oct 15 03:54:14 xen1 kernel: drbd db xen3: Restarting receiver thread Oct 15 03:54:14 xen1 kernel: drbd db xen2: No reconciliation resync even though 'xen3' disappeared. (o=0) Oct 15 03:54:14 xen1 kernel: drbd db: Aborting cluster-wide state change 156852247 (22ms) rv = -19 Oct 15 03:54:27 xen1 kernel: drbd beta/0 drbd102: new current UUID: A3B6968362787AD3 weak: FFFFFFFFFFFFFFF9 Oct 15 03:54:41 xen1 kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [drbd_w_db:1631] Oct 15 03:54:41 xen1 kernel: Modules linked in: tun bridge 8021q garp stp llc bonding drbd_transport_tcp(O) drbd(O) coretemp x86_pkg_temp_thermal crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul sg glue_helper ablk_helper cryptd shpchp mei_me mei iTCO_wdt iTCO_vendor_support joydev pcspkr i2c_i801 lpc_ich wmi 8250_fintek ipmi_si ipmi_msghandler acpi_power_meter xenfs xen_privcmd ip_tables xfs libcrc32c sd_mod crc32c_intel e1000e drm_kms_helper igb ahci hwmon xhci_pci ttm dca xhci_hcd libahci ptp arcmsr(O) pps_core dm_mirror dm_region_hash dm_log dm_mod blktap xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ipv6 autofs4 Oct 15 03:54:41 xen1 kernel: CPU: 4 PID: 1631 Comm: drbd_w_db Tainted: G O 3.18.41-20.el7.x86_64 #1 Oct 15 03:54:41 xen1 kernel: Hardware name: Supermicro Super Server/X10DRW-i, BIOS 2.0a 05/20/2016 Oct 15 03:54:41 xen1 kernel: task: ffff880032e3e0d0 ti: ffff88002822c000 task.ti: ffff88002822c000 Oct 15 03:54:41 xen1 kernel: RIP: e030:[<ffffffff8100bc70>] [<ffffffff8100bc70>] xen_irq_disable_direct+0x0/0x9 Oct 15 03:54:41 xen1 kernel: RSP: e02b:ffff88002822fd50 EFLAGS: 00000202 Oct 15 03:54:41 xen1 kernel: RAX: ffff8800363302b8 RBX: ffff8800363311d8 RCX: ffff8800363311a0 Oct 15 03:54:41 xen1 kernel: RDX: 0000000000000100 RSI: ffff8800363302b8 RDI: ffff8800363311d8 Oct 15 03:54:41 xen1 kernel: RBP: ffff88002822fd58 R08: 0000000000000ac5 R09: 00000000000000a0 Oct 15 03:54:41 xen1 kernel: R10: 00000001003d5791 R11: 0000000000000008 R12: ffff8800363311d8 Oct 15 03:54:41 xen1 kernel: R13: ffff88002822fe10 R14: 0000000000000001 R15: ffff880036331498 Oct 15 03:54:41 xen1 kernel: FS: 0000000000000000(0000) GS:ffff88003f880000(0000) knlGS:0000000000000000 Oct 15 03:54:41 xen1 kernel: CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 15 03:54:41 xen1 kernel: CR2: 00007fd5defb300e CR3: 0000000004cb1000 CR4: 0000000000042660 Oct 15 03:54:41 xen1 kernel: Stack: Oct 15 03:54:41 xen1 kernel: ffffffff8165a29a ffff88002822fd78 ffffffff810b4144 ffff880036331000 Oct 15 03:54:41 xen1 kernel: ffff8800363311d8 ffff88002822fdb8 ffffffffa0496378 ffffffffa04e622c Oct 15 03:54:41 xen1 kernel: 0000000000000100 ffff88002822fdb8 ffff88002822fe10 ffff8800363311a0 Oct 15 03:54:41 xen1 kernel: Call Trace: Oct 15 03:54:41 xen1 kernel: [<ffffffff8165a29a>] ? _raw_spin_lock_irqsave+0x1a/0x50 Oct 15 03:54:41 xen1 kernel: [<ffffffff810b4144>] down_trylock+0x14/0x40 Oct 15 03:54:41 xen1 kernel: [<ffffffffa0496378>] do_unqueued_resource_work+0xb8/0xf0 [drbd] Oct 15 03:54:41 xen1 kernel: [<ffffffffa049cee0>] drbd_worker+0x390/0x530 [drbd] Oct 15 03:54:41 xen1 kernel: [<ffffffff810ad6b0>] ? prepare_to_wait_event+0x110/0x110 Oct 15 03:54:41 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:54:41 xen1 kernel: [<ffffffffa04bbf49>] drbd_thread_setup+0x69/0x130 [drbd] Oct 15 03:54:41 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:54:41 xen1 kernel: [<ffffffff8108c079>] kthread+0xc9/0xe0 Oct 15 03:54:41 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:54:41 xen1 kernel: [<ffffffff8165a898>] ret_from_fork+0x58/0x90 Oct 15 03:54:41 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:54:41 xen1 kernel: Code: ff 85 c0 74 ce 0f 0b 0f 1f 40 00 65 c6 04 25 c1 a0 00 00 00 65 f6 04 25 c0 a0 00 00 ff 74 05 e8 47 00 00 00 c3 66 0f 1f 44 00 00 <65> c6 04 25 c1 a0 00 00 01 c3 66 0f 1f 44 00 00 65 f6 04 25 c1 Oct 15 03:55:09 xen1 kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [drbd_w_db:1631] Oct 15 03:55:09 xen1 kernel: Modules linked in: tun bridge 8021q garp stp llc bonding drbd_transport_tcp(O) drbd(O) coretemp x86_pkg_temp_thermal crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul sg glue_helper ablk_helper cryptd shpchp mei_me mei iTCO_wdt iTCO_vendor_support joydev pcspkr i2c_i801 lpc_ich wmi 8250_fintek ipmi_si ipmi_msghandler acpi_power_meter xenfs xen_privcmd ip_tables xfs libcrc32c sd_mod crc32c_intel e1000e drm_kms_helper igb ahci hwmon xhci_pci ttm dca xhci_hcd libahci ptp arcmsr(O) pps_core dm_mirror dm_region_hash dm_log dm_mod blktap xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ipv6 autofs4 Oct 15 03:55:09 xen1 kernel: CPU: 4 PID: 1631 Comm: drbd_w_db Tainted: G O L 3.18.41-20.el7.x86_64 #1 Oct 15 03:55:09 xen1 kernel: Hardware name: Supermicro Super Server/X10DRW-i, BIOS 2.0a 05/20/2016 Oct 15 03:55:09 xen1 kernel: task: ffff880032e3e0d0 ti: ffff88002822c000 task.ti: ffff88002822c000 Oct 15 03:55:09 xen1 kernel: RIP: e030:[<ffffffffa04962f9>] [<ffffffffa04962f9>] do_unqueued_resource_work+0x39/0xf0 [drbd] Oct 15 03:55:09 xen1 kernel: RSP: e02b:ffff88002822fd88 EFLAGS: 00000246 Oct 15 03:55:09 xen1 kernel: RAX: 0000000000000110 RBX: ffff880036331000 RCX: ffff8800363311a0 Oct 15 03:55:09 xen1 kernel: RDX: 0000000000000110 RSI: 0000000000000010 RDI: ffff880036331000 Oct 15 03:55:09 xen1 kernel: RBP: ffff88002822fdb8 R08: 0000000000000ac5 R09: 00000000000000e0 Oct 15 03:55:09 xen1 kernel: R10: 00000001003dc4f1 R11: 0000000000000008 R12: ffff8800363311a0 Oct 15 03:55:09 xen1 kernel: R13: ffff88002822fe10 R14: 0000000000000001 R15: ffff880036331498 Oct 15 03:55:09 xen1 kernel: FS: 0000000000000000(0000) GS:ffff88003f880000(0000) knlGS:0000000000000000 Oct 15 03:55:09 xen1 kernel: CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 15 03:55:09 xen1 kernel: CR2: 00007fd5defb300e CR3: 0000000004cb1000 CR4: 0000000000042660 Oct 15 03:55:09 xen1 kernel: Stack: Oct 15 03:55:09 xen1 kernel: ffffffffa04e622c ffffffff8165a000 ffff88002822fdb8 ffff88002822fe10 Oct 15 03:55:09 xen1 kernel: ffff8800363311a0 ffff88002822fe10 ffff88002822fe78 ffffffffa049cee0 Oct 15 03:55:09 xen1 kernel: 0000000000000000 ffff88002822fe38 ffff8800363313b8 ffff8800363314a0 Oct 15 03:55:09 xen1 kernel: Call Trace: Oct 15 03:55:09 xen1 kernel: [<ffffffff8165a000>] ? _raw_spin_lock_irq+0x10/0x40 Oct 15 03:55:09 xen1 kernel: [<ffffffffa049cee0>] drbd_worker+0x390/0x530 [drbd] Oct 15 03:55:09 xen1 kernel: [<ffffffff810ad6b0>] ? prepare_to_wait_event+0x110/0x110 Oct 15 03:55:09 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:09 xen1 kernel: [<ffffffffa04bbf49>] drbd_thread_setup+0x69/0x130 [drbd] Oct 15 03:55:09 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:09 xen1 kernel: [<ffffffff8108c079>] kthread+0xc9/0xe0 Oct 15 03:55:09 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:09 xen1 kernel: [<ffffffff8165a898>] ret_from_fork+0x58/0x90 Oct 15 03:55:09 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:09 xen1 kernel: Code: 89 e5 41 55 41 54 53 48 89 fb 48 83 ec 18 0f 1f 40 00 48 8b 93 a0 01 00 00 48 89 d6 48 89 d0 48 81 e6 ff fe ff ff f0 48 0f b1 31 <48> 39 c2 75 e2 81 e2 00 01 00 00 48 89 55 d8 48 8b 45 d8 f6 c4 Oct 15 03:55:16 xen1 kernel: INFO: rcu_sched self-detected stall on CPU { 4} (t=60000 jiffies g=7511 c=7510 q=0) Oct 15 03:55:16 xen1 kernel: Task dump for CPU 4: Oct 15 03:55:16 xen1 kernel: drbd_w_db R running task 0 1631 2 0x00000088 Oct 15 03:55:16 xen1 kernel: ffffffff81c73e40 ffff88003f883bc8 ffffffff81098a68 0000000000000004 Oct 15 03:55:16 xen1 kernel: ffffffff81c73e40 ffff88003f883be8 ffffffff8109bf7d ffff88003f883c28 Oct 15 03:55:16 xen1 kernel: 0000000000000005 ffff88003f883c18 ffffffff810cba60 ffff88003f88d8c0 Oct 15 03:55:16 xen1 kernel: Call Trace: Oct 15 03:55:16 xen1 kernel: <IRQ> [<ffffffff81098a68>] sched_show_task+0xa8/0x110 Oct 15 03:55:16 xen1 kernel: [<ffffffff8109bf7d>] dump_cpu_task+0x3d/0x50 Oct 15 03:55:16 xen1 kernel: [<ffffffff810cba60>] rcu_dump_cpu_stacks+0x90/0xd0 Oct 15 03:55:16 xen1 kernel: [<ffffffff810cf53f>] rcu_check_callbacks+0x3df/0x6f0 Oct 15 03:55:16 xen1 kernel: [<ffffffff8109cfe2>] ? account_process_tick+0x62/0x180 Oct 15 03:55:16 xen1 kernel: [<ffffffff810d3feb>] update_process_times+0x4b/0x80 Oct 15 03:55:16 xen1 kernel: [<ffffffff810e32e5>] tick_sched_handle.isra.17+0x25/0x60 Oct 15 03:55:16 xen1 kernel: [<ffffffff810e3365>] tick_sched_timer+0x45/0x80 Oct 15 03:55:16 xen1 kernel: [<ffffffff810d4d56>] __run_hrtimer+0x86/0x230 Oct 15 03:55:16 xen1 kernel: [<ffffffff810e3320>] ? tick_sched_handle.isra.17+0x60/0x60 Oct 15 03:55:16 xen1 kernel: [<ffffffff810d5197>] hrtimer_interrupt+0xf7/0x240 Oct 15 03:55:16 xen1 kernel: [<ffffffff8100b55f>] xen_timer_interrupt+0x2f/0x150 Oct 15 03:55:16 xen1 kernel: [<ffffffff813e774c>] ? add_interrupt_randomness+0x3c/0x1f0 Oct 15 03:55:16 xen1 kernel: [<ffffffff810c4367>] handle_irq_event_percpu+0x57/0x1e0 Oct 15 03:55:16 xen1 kernel: [<ffffffff810c78fe>] handle_percpu_irq+0x3e/0x60 Oct 15 03:55:16 xen1 kernel: [<ffffffff810c393b>] generic_handle_irq+0x2b/0x40 Oct 15 03:55:16 xen1 kernel: [<ffffffff813a2682>] evtchn_fifo_handle_events+0x162/0x170 Oct 15 03:55:16 xen1 kernel: [<ffffffff8139f510>] __xen_evtchn_do_upcall+0x50/0x90 Oct 15 03:55:16 xen1 kernel: [<ffffffff813a12e7>] xen_evtchn_do_upcall+0x37/0x50 Oct 15 03:55:16 xen1 kernel: [<ffffffff8165c4de>] xen_do_hypervisor_callback+0x1e/0x40 Oct 15 03:55:16 xen1 kernel: <EOI> [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 Oct 15 03:55:16 xen1 kernel: [<ffffffff8100122a>] ? xen_hypercall_xen_version+0xa/0x20 Oct 15 03:55:16 xen1 kernel: [<ffffffff8165c4ec>] ? xen_do_hypervisor_callback+0x2c/0x40 Oct 15 03:55:16 xen1 kernel: [<ffffffff8100b38d>] ? xen_force_evtchn_callback+0xd/0x10 Oct 15 03:55:16 xen1 kernel: [<ffffffff8100bcc2>] ? check_events+0x12/0x20 Oct 15 03:55:16 xen1 kernel: [<ffffffff8100bcaf>] ? xen_restore_fl_direct_reloc+0x4/0x4 Oct 15 03:55:16 xen1 kernel: [<ffffffff81659ef7>] ? _raw_spin_unlock_irqrestore+0x17/0x20 Oct 15 03:55:16 xen1 kernel: [<ffffffff810b415d>] ? down_trylock+0x2d/0x40 Oct 15 03:55:16 xen1 kernel: [<ffffffffa0496378>] ? do_unqueued_resource_work+0xb8/0xf0 [drbd] Oct 15 03:55:16 xen1 kernel: [<ffffffffa049cee0>] ? drbd_worker+0x390/0x530 [drbd] Oct 15 03:55:16 xen1 kernel: [<ffffffff810ad6b0>] ? prepare_to_wait_event+0x110/0x110 Oct 15 03:55:16 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:16 xen1 kernel: [<ffffffffa04bbf49>] ? drbd_thread_setup+0x69/0x130 [drbd] Oct 15 03:55:16 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:16 xen1 kernel: [<ffffffff8108c079>] ? kthread+0xc9/0xe0 Oct 15 03:55:16 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:16 xen1 kernel: [<ffffffff8165a898>] ? ret_from_fork+0x58/0x90 Oct 15 03:55:16 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:41 xen1 kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [drbd_w_db:1631] Oct 15 03:55:41 xen1 kernel: Modules linked in: tun bridge 8021q garp stp llc bonding drbd_transport_tcp(O) drbd(O) coretemp x86_pkg_temp_thermal crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul sg glue_helper ablk_helper cryptd shpchp mei_me mei iTCO_wdt iTCO_vendor_support joydev pcspkr i2c_i801 lpc_ich wmi 8250_fintek ipmi_si ipmi_msghandler acpi_power_meter xenfs xen_privcmd ip_tables xfs libcrc32c sd_mod crc32c_intel e1000e drm_kms_helper igb ahci hwmon xhci_pci ttm dca xhci_hcd libahci ptp arcmsr(O) pps_core dm_mirror dm_region_hash dm_log dm_mod blktap xen_acpi_processor xen_pciback xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ipv6 autofs4 Oct 15 03:55:41 xen1 kernel: CPU: 4 PID: 1631 Comm: drbd_w_db Tainted: G O L 3.18.41-20.el7.x86_64 #1 Oct 15 03:55:41 xen1 kernel: Hardware name: Supermicro Super Server/X10DRW-i, BIOS 2.0a 05/20/2016 Oct 15 03:55:41 xen1 kernel: task: ffff880032e3e0d0 ti: ffff88002822c000 task.ti: ffff88002822c000 Oct 15 03:55:41 xen1 kernel: RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>] xen_hypercall_xen_version+0xa/0x20 Oct 15 03:55:41 xen1 kernel: RSP: e02b:ffff88002822fcd8 EFLAGS: 00000246 Oct 15 03:55:41 xen1 kernel: RAX: 0000000000040006 RBX: ffff8800363311d8 RCX: ffffffff8100122a Oct 15 03:55:41 xen1 kernel: RDX: 00000000cc3ccc3c RSI: 0000000000000000 RDI: 0000000000000000 Oct 15 03:55:41 xen1 kernel: RBP: ffff88002822fcf0 R08: 0000000000000ac5 R09: 0000000000000120 Oct 15 03:55:41 xen1 kernel: R10: 00000001003e41f1 R11: 0000000000000246 R12: 00000000ffffffff Oct 15 03:55:41 xen1 kernel: R13: ffff88002822fe10 R14: 0000000000000001 R15: ffff880036331498 Oct 15 03:55:41 xen1 kernel: FS: 0000000000000000(0000) GS:ffff88003f880000(0000) knlGS:0000000000000000 Oct 15 03:55:41 xen1 kernel: CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 15 03:55:41 xen1 kernel: CR2: 00007fd5defb300e CR3: 0000000004cb1000 CR4: 0000000000042660 Oct 15 03:55:41 xen1 kernel: Stack: Oct 15 03:55:41 xen1 kernel: 0000000000000008 000000000000cc3c ffffffff8100b38d ffff88002822fd58 Oct 15 03:55:41 xen1 kernel: ffffffff8100bcc2 0000000000000008 00000001003e41f1 0000000000000120 Oct 15 03:55:41 xen1 kernel: 0000000000000ac5 ffff8800363302b8 ffff8800363302b8 00000000cc3ccc3c Oct 15 03:55:41 xen1 kernel: Call Trace: Oct 15 03:55:41 xen1 kernel: [<ffffffff8100b38d>] ? xen_force_evtchn_callback+0xd/0x10 Oct 15 03:55:41 xen1 kernel: [<ffffffff8100bcc2>] check_events+0x12/0x20 Oct 15 03:55:41 xen1 kernel: [<ffffffff8100bcaf>] ? xen_restore_fl_direct_reloc+0x4/0x4 Oct 15 03:55:41 xen1 kernel: [<ffffffff81659ef7>] ? _raw_spin_unlock_irqrestore+0x17/0x20 Oct 15 03:55:41 xen1 kernel: [<ffffffff810b415d>] down_trylock+0x2d/0x40 Oct 15 03:55:41 xen1 kernel: [<ffffffffa0496378>] do_unqueued_resource_work+0xb8/0xf0 [drbd] Oct 15 03:55:41 xen1 kernel: [<ffffffffa049cee0>] drbd_worker+0x390/0x530 [drbd] Oct 15 03:55:41 xen1 kernel: [<ffffffff810ad6b0>] ? prepare_to_wait_event+0x110/0x110 Oct 15 03:55:41 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:41 xen1 kernel: [<ffffffffa04bbf49>] drbd_thread_setup+0x69/0x130 [drbd] Oct 15 03:55:41 xen1 kernel: [<ffffffffa04bbee0>] ? w_complete+0x20/0x20 [drbd] Oct 15 03:55:41 xen1 kernel: [<ffffffff8108c079>] kthread+0xc9/0xe0 Oct 15 03:55:41 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:41 xen1 kernel: [<ffffffff8165a898>] ret_from_fork+0x58/0x90 Oct 15 03:55:41 xen1 kernel: [<ffffffff8108bfb0>] ? kthread_create_on_node+0x180/0x180 Oct 15 03:55:41 xen1 kernel: Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc Regards, Maarten Bremer